OSGi could be great for JSR 277

Bryan Atsatt describes a plausible way forward for JSR 277. I agree with his reasoning, although OSGi is not only the de facto standard for Java modularity, it's a proper standard too.

Bryan was one of the key movers in the Expert Group when I was involved. In fact, he proposed a solution to JSR 294 ages ago which is close, if not identical, to the approach Alex Buckley is now considering.

Welcome to the blogosphere, Bryan!

JSR 294 Early Draft

The Early Draft Review of JSR 294 is now complete. Little seems to have changed since when I left the Expert Group. The main weakness from my perspective is that a class file must name its superpackage in order for the class to be a member of the superpackage. This means that you can't easily produce a program which runs on existing versions of Java but which will take advantage of superpackages when it runs on Java 7 and beyond.

Practically speaking, this means that superpackages will only be used in applications that are willing to restrict themselves to Java 7 and beyond. So I suspect that superpackages will not come into popular usage within the next few years.

Of course, this can be gotten round by using a custom class loader with some bytecode rewriting magic, but one of the reasons that JSR 277, and its progeny JSR 294, were introduced was to avoid application developers from having to create custom class loaders.

Leaving IBM

Today I handed in my resignation and will be leaving IBM on November 5. I've had a very enjoyable 26 years here in Hursley working on a variety of projects from workstation graphics algorithms to high-end transaction servers. A year ago I expected to work in IBM until retirement, but I had no idea what an interesting opportunity was about to open up a few miles down the road.

Although I'll still be involved with OSGi, I'll be handing over the job of representing IBM on the JSR 277 and 294 Expert Groups to someone else. I'll be lucky to find anything else quite so contentious to blog about, although I'll still be very interested in the outcome and may find something to say.

Java modularity: where do we go from here?

Things have moved on somewhat since March, so I thought an update would be helpful. Let's recap first.

JSR 277 is trying to create a static module system for Java. Its offshoot, JSR 294, is focused on the VM and language support for modularity. Both these JSRs are targeting Java 7.

JSR 291, which recently passed the Final Approval Ballot, references and extends the OSGi dynamic component system. JSR 291 is compatible with the Java ME JSR 232, which also references OSGi.

OSGi is a mature technology, having been developed in four releases over the last eight years. There are three independent open source implementations (Equinox, Felix, and Knopflerfish), a specification, reference implementation, and TCK.

I am an active member of the JSR 277 and JSR 294 Expert Groups and am the spec. lead of JSR 291.

Design goals

Crucially, the goals of these JSRs are rather different, although the underlying concepts are similar. JSRs 277 and 294 are building basic support into the Java SE platform whereas JSR 291 is pure Java built on top of the Java platform.

Look at it another way. JSRs 277 and 294 are attempting to produce an evolution of java.lang.ClassLoader with a new distribution and packaging format, a new repository, and new language features supported by the VM. JSR 291 on the other hand is implemented in pure Java on top of java.lang.ClassLoader and uses standard JAR files with standard manifests as a distribution and packaging format.

The design point of OSGi has always been to support a broad set of Java ME environments, such as profiles based on the Connected Device Configuration, and Java SE versions, from Java 1.2 onwards, in a compatible way. It has achieved this by building on commonly available Java concepts, notably class loaders.

I think of OSGi as being the pinnacle of custom class loader systems. The way it manages to install, uninstall, start, stop, and update modules without restarting the VM is quite an achievement - not my achievement, I should add, as the dynamic features were mostly complete before I got involved. Of course, other systems have provided some of this capability, but not in such a generally reusable way.

So you can package OSGi with your application, or list it as a prerequisite, and then run on a wide range of Java platforms.

On the other hand, JSRs 277 and 294 have the advantage that if you are happy to pre-req. Java 7 or later, then those JSRs will be built in to the platform.

Why doesn't JSR 277 reuse OSGi?

So why can't the JCP simply adopt JSR 291 as a standard module system for the Java platform? That's what many of the critics of JSR 277 are calling for.

A crucial issue is one of control. I hope the OSGi Alliance would allow OSGi to be included in the Java platform so long as they retained control of the technology. But I suspect Sun would be nervous about the core module system of the Java platform not being under their direct control. Such nervousness would be understandable if it was not in the interests of the OSGi Alliance to maintain the value of Java, but the reality is just opposite: the success of the OSGi Alliance depends to a large extent on the success of Java.

In my earlier blog, I made the point about the OSGi technology still evolving. But I think that problem is soluble. There would need to be a way of overriding the built-in module system with a later, backward compatible version.

Why not scrap JSR 277?

A radical option, favoured by some, of simply scrapping JSR 277 is less than ideal. It would temporarily remove the confusion that competing module systems inevitably create. But before long, the requirements for a module system built in to the Java platform would surface again.

The consumer JRE work has shown there is a real need to modularise the JRE itself and this requires a built-in module system. However, the consumer JRE doesn't seem to be proposing a properly architecture module system. It's a 'quick fix' which can be delivered in Java 6 update releases.

So if JSR 277 was scrapped, then whatever mechanisms evolve out of the consumer JRE work could become the defacto module system for the Java platform. The Java platform would still lack a properly architected built-in module system.

JSR 277 also has a stated objective of interoperating with JSR 291, which means JSR 277 modules will be able to make effective use of OSGi modules and vice versa with minimal restrictions. If this objective is achieved, it will protect the significant investment in OSGi modules in the Java community. Scrapping JSR 277 would jeopardise this investment if a non-interoperating module system grew up in place of JSR 277.

The road ahead

So, given the momentum behind JSR 277 and the emphasis placed on it by Sun at this year's JavaOne, what lies ahead?

Of course, JSR 291 should ideally benefit from the features introduced by JSR 277 and JSR 294. But the immediate requirement is that JSR 277 and JSR 291 interoperate well so that modules written in terms of one JSR are able to make effective use of modules written in terms of the other JSR.

To achieve this, the Expert Groups of JSR 277 and JSR 291 will need to work together. The JSR 277 Expert Group are responsible for producing an interoperation proposal which the JSR 291 Expert Group will review. At the time of writing, the interoperation proposal is still being worked on by the JSR 277 spec. lead.


The dream solution

The dream solution is clear: JSR 277 should adopt JSR 291, underpinned by language and VM support in JSR 294, and add the JSR 277 repository architecture. This would accelerate the progress of JSR 277 by several years and guarantee perfect interoperation between JSR 277 and JSR 291.

But it would take a complete about turn by Sun to turn this dream into reality. Maybe, just maybe, Sun are sufficiently committed to the success of Java that they will eventually decide to do just that. Maybe, just maybe, the Java community will be able to look back later and paraphrase Winston Churchill: "Sun could always be counted on to do the right thing, after having exhausted all the alternatives".

Further information

The JSR 291 Expert Group mailing list and the JSR 294 list have been openly readable since their Expert Groups started. JSR 277 ran in private for many months, but has now opened up its list.

Disclaimer

I'm biassed.

I have been involved in the OSGi Core Platform Expert Group for a couple of years during which time I co-authored some of the modularity improvements for OSGi R4 with Richard Hall (the leader of the Apache Felix OSGi project).

I work for IBM which belongs to the OSGi Alliance and which has based some of its products on OSGi. IBM and Sun co-operate on Java, but compete in related areas such as Java EE, not to mention UNIX-style operating systems and hardware. So IBM and Sun don't necessarily agree on everything that happens in Java. In fact, I think it would be pretty unhealthy if that were the case.

I should also re-iterate that my opinions are not necessarily shared by IBM.

That said, I hope this blog objectively lays out the situation.

JSR 277 and JSR 294 observer lists

You may like to know there is a public observer list for JSR 277 and one for JSR 294 too, both with publicly visible archives.

This is not obvious from the JSR homepages.

Modularity at JavaOne

Modularity had a lot of focus at last week's JavaOne conference. There were multiple references to JSRs 277 and 294 by Sun speakers, specific technical sessions on these JSRs, and a combined JSR 277 and 294 BOF.

There were a number of questions and comments at the JSR 294 session on the strawman's structuring of superpackage membership information in a separate source file rather than distributed across the source files of the member classes. This is something we should discuss on the mailing list (which you can read via the observer list).

Apart from an OSGi Best Practices technical session and the JSR 291 BOF, there were several mentions of OSGi and JSR 291 in other sessions. The goal of interoperating well with JSR 291 was presented in the JSR 277 session and even mentioned in one of Sun's general sessions¹. Stanley Ho will float a proposal on the JSR 277 mailing list in due course, so subscribe to the observer list if you are interested (Expert Group discussions should begin to be mirrored there soon).

Ethan Nicholas gave an update on the browser edition work to minimise the download and initial install size of the JRE. This seems to be making good progress, although it is not based on any of the above JSRs, so there may be some 'shake-out' in due course.

In one of the "meet the team" sessions, I asked Sun's class library team what their plan was for exploiting JSR 277 and JSR 294 in the JRE. It seems they don't yet have one. I expect this to change, particularly after the JSR 277 and JSR 294 prototypes become available in a subproject of OpenJDK.

In contrast, the Apache Harmony session described how OSGi had been used from the start to modularise the class libraries. I gave a demo. of prototype JVM support for modular Harmony in the JSR 291 BOF.

GlassFish v3 is developing a module system known as HK2 ("Hundred Kilobytes Kernel") which is loosely based on JSR 277. It also has a dependency injection framework similar to Guice. (Unlike GlassFish which is mostly licensed under the CDDL, HK2 currently has no license defined, so you may want to avoid looking at the source code until there is a proper license.)

I had face to face meetings with Andreas Sterbenz to discuss aspects of JSR 294 and with Stanley Ho and Michał Cierniak to discuss interoperation of JSR 277 with JSR 291. There's a lot of work ahead and we won't know for quite a while how good the interoperation with JSR 291 will really be, but everyone concerned seems to be well aware of the importance of this capability.

1. See a couple of minutes from Intro. and Technical Deep Dive starting at 18 minutes and 45 seconds from the start when Danny Coward talks about modularity and refers to the goal of interoperation with OSGi.

JSR 291 BOF at JavaOne 2007

After reconsideration, a JSR 291 BOF¹ has been accepted for JavaOne 2007. The plan is for Richard Hall and I to lead the session.

I hope to include some thoughts on JSR 277 interoperation as well as JSR 294 (superpackages) exploitation.

1. BOF-21240 entitled "JSR 291: Dynamic Component Support For The Java SE Platform", Wednesday 9 May, 9:55pm-10:45pm

JSR 294 Expert Group

The JSR 294 Expert Group now has a publicly visible mailing list which is good timing as the Expert Group only kicked off a couple of weeks ago, so there'll be a complete archive of its deliberations.

It's a little disappointing that the newly revamped jcp.org site doesn't appear to be capable of hosting openly readable mailing lists as I would like that to be the default for all JSR Expert Groups.

Background to Java modularity

Should Java have included modularity features from the start or is the kind of modularity really needed in Java rather different from that baked in to some other languages?

In 2003, IBM Research published a paper known internally as "modjava" but named "MJ" externally. The authors' approach was to create a static module system not dissimilar to the strawman for JSR 294 except that MJ was built from class loaders for ease of prototyping and didn't encompass versioning (although a couple of us refactored the internal MJ prototype to enable versioning).

Vernon Green and I then led a working group in IBM's Software Group Architecture Board on Java versioning and modularity which started from the MJ paper. Later we discovered OSGi and how that matched in many respects what the working group was looking for. I was given the task of taking our additional requirements into the OSGi R4 standards activity where I struck up a collaboration with Richard Hall who had a similar perspective based on his research work on the Oscar OSGi framework.

The real world requirements of versioning and dynamic modification of applications and supporting software without restarting the JVM have effectively forced Java to develop the new, and highly successful, OSGi technology.

It's interesting to speculate what would have happened if a static module system had been put into Java much earlier. My guess is that it wouldn't have bothered to address versioning or dynamicity requirements which only become crucial in the context of relatively large systems with continuous operation.

The JSR 294 Expert Group is just starting up, so this background suddenly becomes relevant...

Modularity and the Java platform

Neil questioned the point of putting modularity support into the Java platform.

Of course, OSGi does a pretty good job of modularity support based on java.lang.ClassLoader. So why isn't that sufficient?

At one level, OSGi is quite sufficient. There is already massive adoption of OSGi and this is continuing to spread into new and interesting areas.

At a more theoretical level, there are some things that custom class loader networks will never achieve.

Most importantly is VM enforcement of module boundaries. It is possible to obtain unexported Class instances from a class loader. OSGi bundles are implemented as class loaders. Similarly, JSR 277 modules will boil down to specialised class loaders. So this is where JSR 294 comes in with its objective of enforcing module boundaries in the VM.

This extra level of enforcement is irrelevant if the aim of using modules is to divide a piece of software into manageable units with well-defined interfaces.

However, VM enforcement starts to become interesting if modules need to protect some parts of an application or the system from inappropriate access. There is also some scope for additional, or at least simplified, performance optimisations if certain classes can be guaranteed never to be accessed outside their defining module.

I would dearly like JSR 294 to provide mechanisms that will be equally useful to OSGi as to JSR 277. I think OSGi would benefit from VM support, although it would be an exaggeration to say that OSGi really needs VM support.

The state of Java modularity: JSRs 277, 291, and 294

People are pretty perceptive. They have picked up that there's a struggle to work out the right model for Java modularity. However, it's not a simple choice between JSR 277 and JSR 291 as many seem to think. Let's look at what's going on in a bit more depth.

Background

Firstly, a brief recap for the uninitiated.

JSR 277 is trying to create a static module system for Java 7. JSR 291 is referencing and extending the already mature OSGi dynamic component system while maintaining compatibility with JSR 232 which did a similar thing for Java ME. JSR 294, an offshoot of JSR 277, is focused on the VM and language support for modularity.

My personal involvement is that I belong to the JSR 277 Expert Group (but cannot disclose the discussions which are private to the group) and am the spec. lead of JSR 291 (which has an openly readable mailing list). I am also in the JSR 294 Expert Group, which hasn't started work yet.

Secondly, I want to be up front about my potential bias.

I have been involved in the OSGi Core Platform Expert Group for a couple of years during which time I co-authored some of the modularity improvements for OSGi R4 with Richard Hall (the leader of the Apache Felix OSGi project).

I work for IBM which belongs to the OSGi Alliance and which has based some of its products on OSGi. IBM and Sun co-operate on Java, but compete in related areas such as Java EE, not to mention UNIX-style operating systems and hardware. So IBM and Sun don't necessarily agree on everything that happens in Java. In fact, I think it would be pretty unhealthy if that were the case.

I should also re-iterate that my opinions are not necessarily shared by IBM.

That said, I'll try to lay out what's going on as objectively as I can.

Design points

The crucial thing to notice is that the goals of these JSRs are rather different even though the underlying technology is similar. JSRs 277 and 294 are building basic support into the Java SE platform whereas JSR 291 is pure Java built on top of the Java platform.

Looking at it another way, JSRs 277 and 294 are attempting to produce an evolution of java.lang.ClassLoader with a new distribution and packaging format, a new repository, plus new language features supported by the VM. JSR 291 on the other hand is implemented in pure Java on top of java.lang.ClassLoader and uses standard JAR files with standard manifests as a distribution and packaging format.

The design point of OSGi has always been to support a broad set of Java ME environments, such as profiles based on the Connected Device Configuration, and Java SE versions, from Java 1.2 onwards, in a compatible way. It has achieved this by building on commonly available Java concepts, notably class loaders. I think of OSGi as being the pinnacle of custom class loader systems. The way it manages to install, uninstall, start, stop, and update modules without restarting the VM is quite an achievement. Of course, some other systems have provided some of this capability, but I am not aware of any that have done it in such a generally reusable way.

So you can package OSGi with your application, or list it as a prerequisite, and then run on a wide range of Java platforms. JSR 277 on the other hand has the advantage that if you are happy to pre-req. Java 7 or later, then JSR 277 will be built in to the platform.

Not invented here?

So why can't the JCP simply adopt JSR 291 as a standard module system for Java? That's what many of the critics of JSR 277 suggest should happen.

One problem is that the OSGi specification, although mature, is still evolving. A number of new requirements were surfaced by the JSR 291 Expert Group. OSGi is rapidly spreading into new areas of application which are bound to result in additional requirements for OSGi R5 and beyond. To lock a particular Java SE version, such as Java 7, to a particular version of OSGi would be limiting as there wouldn't be an opportunity to upgrade the OSGi support significantly before Java 8. For example, an application that required some new features of OSGi wouldn't necessarily want to limit its deployment to Java 8 environments. So the ability to mix and match Java and OSGi versions as required is essential.

You may be wondering whether JSR 277 has the same problem, but since it is deliberately positioned to be the module system packaged with the Java 7 platform, its subsequent rate of evolution is, by definition, the same as that of Java. So the design point is that it has to be fairly basic in its support and not aim for the level of sophistication of OSGi, particularly in its support for legacy applications, which is one of OSGi's strong points.

The challenge for JSR 277 is to address its requirements in a sufficiently general way that it does not need to evolve quickly. The classic way of achieving this would be to divide the function into a lower level and relatively stable API and a user overridable piece implemented in terms of the lower level API. Like a module version of custom class loaders. JSR 277, as you can see from the Early Draft, is not adopting this approach but is attempting to produce one module system which can cope without being overridden. This will be quite an achievement if the Expert Group can pull it off.

The road ahead

So it seems there is a place for both JSR 277 and JSR 291. Of course, JSR 291 should ideally benefit from the features introduced by JSR 277 and JSR 294. But the immediate requirement is that JSR 277 and JSR 291 interoperate well so that modules written in terms of one JSR are able to make effective use of modules written in terms of the other JSR.

To achieve this, the Expert Groups of JSR 277 and JSR 291 will need to work together. I expect this to take the form of the JSR 277 Expert Group proposing a means of interoperating with other module systems and then the JSR 291 Expert Group reviewing this proposal and providing feedback.

Interoperation is certainly an interesting challenge to keep us occupied, but it's just part of the challenge of working out the right model for modularity in the Java platform.