Monday, March 12, 2007

The state of Java modularity: JSRs 277, 291, and 294

People are pretty perceptive. They have picked up that there's a struggle to work out the right model for Java modularity. However, it's not a simple choice between JSR 277 and JSR 291 as many seem to think. Let's look at what's going on in a bit more depth.

Background

Firstly, a brief recap for the uninitiated.

JSR 277 is trying to create a static module system for Java 7. JSR 291 is referencing and extending the already mature OSGi dynamic component system while maintaining compatibility with JSR 232 which did a similar thing for Java ME. JSR 294, an offshoot of JSR 277, is focused on the VM and language support for modularity.

My personal involvement is that I belong to the JSR 277 Expert Group (but cannot disclose the discussions which are private to the group) and am the spec. lead of JSR 291 (which has an openly readable mailing list). I am also in the JSR 294 Expert Group, which hasn't started work yet.

Secondly, I want to be up front about my potential bias.

I have been involved in the OSGi Core Platform Expert Group for a couple of years during which time I co-authored some of the modularity improvements for OSGi R4 with Richard Hall (the leader of the Apache Felix OSGi project).

I work for IBM which belongs to the OSGi Alliance and which has based some of its products on OSGi. IBM and Sun co-operate on Java, but compete in related areas such as Java EE, not to mention UNIX-style operating systems and hardware. So IBM and Sun don't necessarily agree on everything that happens in Java. In fact, I think it would be pretty unhealthy if that were the case.

I should also re-iterate that my opinions are not necessarily shared by IBM.

That said, I'll try to lay out what's going on as objectively as I can.

Design points

The crucial thing to notice is that the goals of these JSRs are rather different even though the underlying technology is similar. JSRs 277 and 294 are building basic support into the Java SE platform whereas JSR 291 is pure Java built on top of the Java platform.

Looking at it another way, JSRs 277 and 294 are attempting to produce an evolution of java.lang.ClassLoader with a new distribution and packaging format, a new repository, plus new language features supported by the VM. JSR 291 on the other hand is implemented in pure Java on top of java.lang.ClassLoader and uses standard JAR files with standard manifests as a distribution and packaging format.

The design point of OSGi has always been to support a broad set of Java ME environments, such as profiles based on the Connected Device Configuration, and Java SE versions, from Java 1.2 onwards, in a compatible way. It has achieved this by building on commonly available Java concepts, notably class loaders. I think of OSGi as being the pinnacle of custom class loader systems. The way it manages to install, uninstall, start, stop, and update modules without restarting the VM is quite an achievement. Of course, some other systems have provided some of this capability, but I am not aware of any that have done it in such a generally reusable way.

So you can package OSGi with your application, or list it as a prerequisite, and then run on a wide range of Java platforms. JSR 277 on the other hand has the advantage that if you are happy to pre-req. Java 7 or later, then JSR 277 will be built in to the platform.

Not invented here?

So why can't the JCP simply adopt JSR 291 as a standard module system for Java? That's what many of the critics of JSR 277 suggest should happen.

One problem is that the OSGi specification, although mature, is still evolving. A number of new requirements were surfaced by the JSR 291 Expert Group. OSGi is rapidly spreading into new areas of application which are bound to result in additional requirements for OSGi R5 and beyond. To lock a particular Java SE version, such as Java 7, to a particular version of OSGi would be limiting as there wouldn't be an opportunity to upgrade the OSGi support significantly before Java 8. For example, an application that required some new features of OSGi wouldn't necessarily want to limit its deployment to Java 8 environments. So the ability to mix and match Java and OSGi versions as required is essential.

You may be wondering whether JSR 277 has the same problem, but since it is deliberately positioned to be the module system packaged with the Java 7 platform, its subsequent rate of evolution is, by definition, the same as that of Java. So the design point is that it has to be fairly basic in its support and not aim for the level of sophistication of OSGi, particularly in its support for legacy applications, which is one of OSGi's strong points.

The challenge for JSR 277 is to address its requirements in a sufficiently general way that it does not need to evolve quickly. The classic way of achieving this would be to divide the function into a lower level and relatively stable API and a user overridable piece implemented in terms of the lower level API. Like a module version of custom class loaders. JSR 277, as you can see from the Early Draft, is not adopting this approach but is attempting to produce one module system which can cope without being overridden. This will be quite an achievement if the Expert Group can pull it off.

The road ahead

So it seems there is a place for both JSR 277 and JSR 291. Of course, JSR 291 should ideally benefit from the features introduced by JSR 277 and JSR 294. But the immediate requirement is that JSR 277 and JSR 291 interoperate well so that modules written in terms of one JSR are able to make effective use of modules written in terms of the other JSR.

To achieve this, the Expert Groups of JSR 277 and JSR 291 will need to work together. I expect this to take the form of the JSR 277 Expert Group proposing a means of interoperating with other module systems and then the JSR 291 Expert Group reviewing this proposal and providing feedback.

Interoperation is certainly an interesting challenge to keep us occupied, but it's just part of the challenge of working out the right model for modularity in the Java platform.

11 comments:

Anonymous said...

How do you see the comparison with the log4j vs jdk logging decision?
Not going with the existing open source modularity mechanism might sentence us to apache-commons-modularity for life.

Richard Jackson said...

Ok, Lets see if I'm getting this strait. JsR 277 & 294 are trying to solve the current issues with static class path. Example the issues with multiple jar's with the same name on the class path, versioning in jar's ect.. or class path hell. While JSR 291 is the dynamic issues of loading and unloading jar's (bundles) on the fly. If I'm reading this right then this is something I can get behind because the current jar hell is something that needs to be dealt with. It would be really nice to have a system repository where everything is installed then when my application loads the correct version of the lib(jar) I need will be loaded with out having to do class path magic. Something like that would be very help full for the end user and the developer.

Glyn said...

>How do you see the comparison with
>the log4j vs jdk logging decision?

I'm not aware of the detailed background of the logging debate, but logging would seem at least an order of magnitude easier to get 'right', although I dare say there's more to it than I appreciate.

With the benefit of hindsight, how much has log4j evolved since JDK logging was standardised? You see, I think the crucial point here is the rate of evolution of module systems relative to the rate of evolution of the Java platform.

Anyone have a link to a dispassionate analysis of the logging debate? Failing that, anyone prepared to write such an analysis?

Glyn said...

Richard: JSR 291 also solves class path hell and handles versioning, but you're right that JSR 277 is providing a repository for installing modules in.

Anonymous said...

Does JSR 277 take care of class space consistency like OSGi does?
I.e. if class A requires class B, and class C requires both A and B than the modular runtime will resolve the modules of A and C to exacly the same module exporting B. If JSR 277 aims to evolve the class loader it seems to be a crucially important thing to guarantee class space consistency wouldn't you agree?

Even if JSR 277 does gurantee that it's evolved class loader will keep the class space it manages consistent at all times there will be trouble in mapping OSGi bundles to JSR 277 modules right? Because both the OSGi bundle and the JSR 277 module contain many java packages and at the same time the OSGi class loader handles consistency in terms of packages while the JSR 277 class loader would handle this in terms of entire modules only.

Glyn said...

>Does JSR 277 take care of class
>space consistency like OSGi does?
In your example in OSGi, you'd have to encode the dependency from A to B with a "uses" directive on import-package so that OSGi will preserve class space consistency.

JSR 277 does not support this, so you'd probably get a class cast exception or something similar.

Yes, we need to take this into account when JSR 277 steps up to providing interoperation with JSR 291 (OSGi). I suspect there will be some restrictions that are not present when using OSGi on its own.

Glyn said...

Further to my previous comment, I should have mentioned that JSR 277 supports an optional "deep validation" check which detects some types of inconsistent class spaces.

I'm not sure whether deep validation would catch the particular scenario described.

Also I'm not sure whether it is feasible for the JSR 277 resolution process to use deep validation and continue searching until it finds a module wiring which ensures consistent class spaces.

Carlus Henry said...

I have been struggling with understanding what is going on with all of the different JSR's regarding Java Modularity and how it relates to OSGi. After reading this blog posting, it is more clear as to the motivations behind the different JSR's as well as why they are all important.

Thank you so much for taking the time to produce this. I look forward to reading more interesting posts from you in the future.

Carlus

Glyn said...

Glad you found the blog useful, Carlus.

Anonymous said...

These comments have been invaluable to me as is this whole site. I thank you for your comment.

Glyn said...

Glad you've found the blog helpful.

Projects

OSGi (130) Virgo (59) Eclipse (10) Equinox (9) dm Server (8) Felix (4) WebSphere (3) Aries (2) GlassFish (2) JBoss (1) Newton (1) WebLogic (1)

Blog Archive