Project Jigsaw

Mark Reinhold recently announced Project Jigsaw for modularising the JDK. Could this be a step towards the dream solution? I can hope.

Meanwhile JSR 277 is deferred beyond Java 7. Joy.

OSGi could be great for JSR 277

Bryan Atsatt describes a plausible way forward for JSR 277. I agree with his reasoning, although OSGi is not only the de facto standard for Java modularity, it's a proper standard too.

Bryan was one of the key movers in the Expert Group when I was involved. In fact, he proposed a solution to JSR 294 ages ago which is close, if not identical, to the approach Alex Buckley is now considering.

Welcome to the blogosphere, Bryan!

Popular RFE

My favourite sunbug is now on the list of the Top 25 RFEs (Requests for Enhancement).

The 11th most popular sunbug?

Sunbug 6650394 is coming along nicely - both in terms of votes and supportive comments. At the time of writing, it has 99 votes, so thank you if you took the trouble to vote! If you care and you haven't yet voted or commented, please consider doing so.

I was hoping the bug would make it into Sun's Top 25 Bugs list since it has more votes than most of the bugs in that list, but for some reason it's not yet included. If the list was accurate, which it clearly isn't, then 6650394 would be the 11th most popular sunbug.

Sunbug for JSR 277 module system interoperation

Sun has kindly raised sunbug 6650394 on my behalf to track the module system interoperation issue for JSR 277.

Please consider voting for this bug if you would like JSR 277 to interoperate with JSR 291 (OSGi). Add yourself to the watch list if you want to track the bug.

Leaving IBM

Today I handed in my resignation and will be leaving IBM on November 5. I've had a very enjoyable 26 years here in Hursley working on a variety of projects from workstation graphics algorithms to high-end transaction servers. A year ago I expected to work in IBM until retirement, but I had no idea what an interesting opportunity was about to open up a few miles down the road.

Although I'll still be involved with OSGi, I'll be handing over the job of representing IBM on the JSR 277 and 294 Expert Groups to someone else. I'll be lucky to find anything else quite so contentious to blog about, although I'll still be very interested in the outcome and may find something to say.

Hope for interoperation between JSR 277 and JSR 291

Infoq and Alex Miller picked up on my recent comments about the somewhat closed approach being adopted by JSR 277. Since the whole concept of interoperation between JSR 277 and JSR 291 seems to be doubted, I thought I'd better provide some reassurance.

The saving grace of both these JSRs is that they ultimately boil down to delegation networks between module class loaders. So there is a lingua franca lurking not very far under the surface. Granted the views presented to the module developer are quite different in detail.

Quite a while ago, I wrote a prototype module system interoperation framework, described in more detail here, to start to flesh out how these JSRs could interoperate. I regard it more of an existence proof that a reasonable solution may exist. However, it requires 'cutting' into the module systems at a deeper level than I think is currently being considered by the JSR 277 spec. lead and colleagues.

We discussed some of the options in an informal Expert Group meeting at JavaOne, but all we really concluded was that there was a lot of work ahead to get interoperation working smoothly. I think there is also a consensus among the interested parties that reasonable interoperation is essential to the success of JSR 277. So the sooner we can get more eyes, and brains, on the current proposal, the better.

How to run a JSR Expert Group

The modus operandum of the JSR 277 Expert Group is fascinating.

Strawman proposals are developed in private by the spec. lead, colleagues from Sun, and occasionally one or two others from elsewhere. These are then presented to the Expert Group for comment. After more or less debate, the spec. lead asserts the consensus and the strawman goes forward for inclusion in the draft spec. Typically, there can be a change of position when something small is clearly broken, but significant design changes and smaller tweaks tend to be resisted.

On the whole, the real JSR 277 contributors are working in private and the so-called Expert Group acts as a sounding board without any significant influence on the outcome. Some would call this rubber-stamping.

I'm trying to change this on one crucial front - that of interoperation between JSR 277 and JSR 291. But progress is dreadfully slow.

Apparently, interoperation prototypes are being developed and thinking is going on, but none of this is visible to the Expert Group or the broader community. This is a bit odd as a modules project has been set up on OpenJDK, so I would have expected that kind of prototyping to go on in a branch or subdirectory for early feedback from others.

It's worrying because the OSGi experts, Richard Hall and myself, in the JSR 277 Expert Group and the others in the JSR 291 Expert Group are not currently able to help. By the time we see a strawman, it may be too late to make any significant changes.

Clearly my experience of Apache and Eclipse is no guide to the way things happen in OpenJDK. Similarly, my experience of other standards bodies is also no guide to how a JSR Expert Group is necessarily run.

Java modularity: where do we go from here?

Things have moved on somewhat since March, so I thought an update would be helpful. Let's recap first.

JSR 277 is trying to create a static module system for Java. Its offshoot, JSR 294, is focused on the VM and language support for modularity. Both these JSRs are targeting Java 7.

JSR 291, which recently passed the Final Approval Ballot, references and extends the OSGi dynamic component system. JSR 291 is compatible with the Java ME JSR 232, which also references OSGi.

OSGi is a mature technology, having been developed in four releases over the last eight years. There are three independent open source implementations (Equinox, Felix, and Knopflerfish), a specification, reference implementation, and TCK.

I am an active member of the JSR 277 and JSR 294 Expert Groups and am the spec. lead of JSR 291.

Design goals

Crucially, the goals of these JSRs are rather different, although the underlying concepts are similar. JSRs 277 and 294 are building basic support into the Java SE platform whereas JSR 291 is pure Java built on top of the Java platform.

Look at it another way. JSRs 277 and 294 are attempting to produce an evolution of java.lang.ClassLoader with a new distribution and packaging format, a new repository, and new language features supported by the VM. JSR 291 on the other hand is implemented in pure Java on top of java.lang.ClassLoader and uses standard JAR files with standard manifests as a distribution and packaging format.

The design point of OSGi has always been to support a broad set of Java ME environments, such as profiles based on the Connected Device Configuration, and Java SE versions, from Java 1.2 onwards, in a compatible way. It has achieved this by building on commonly available Java concepts, notably class loaders.

I think of OSGi as being the pinnacle of custom class loader systems. The way it manages to install, uninstall, start, stop, and update modules without restarting the VM is quite an achievement - not my achievement, I should add, as the dynamic features were mostly complete before I got involved. Of course, other systems have provided some of this capability, but not in such a generally reusable way.

So you can package OSGi with your application, or list it as a prerequisite, and then run on a wide range of Java platforms.

On the other hand, JSRs 277 and 294 have the advantage that if you are happy to pre-req. Java 7 or later, then those JSRs will be built in to the platform.

Why doesn't JSR 277 reuse OSGi?

So why can't the JCP simply adopt JSR 291 as a standard module system for the Java platform? That's what many of the critics of JSR 277 are calling for.

A crucial issue is one of control. I hope the OSGi Alliance would allow OSGi to be included in the Java platform so long as they retained control of the technology. But I suspect Sun would be nervous about the core module system of the Java platform not being under their direct control. Such nervousness would be understandable if it was not in the interests of the OSGi Alliance to maintain the value of Java, but the reality is just opposite: the success of the OSGi Alliance depends to a large extent on the success of Java.

In my earlier blog, I made the point about the OSGi technology still evolving. But I think that problem is soluble. There would need to be a way of overriding the built-in module system with a later, backward compatible version.

Why not scrap JSR 277?

A radical option, favoured by some, of simply scrapping JSR 277 is less than ideal. It would temporarily remove the confusion that competing module systems inevitably create. But before long, the requirements for a module system built in to the Java platform would surface again.

The consumer JRE work has shown there is a real need to modularise the JRE itself and this requires a built-in module system. However, the consumer JRE doesn't seem to be proposing a properly architecture module system. It's a 'quick fix' which can be delivered in Java 6 update releases.

So if JSR 277 was scrapped, then whatever mechanisms evolve out of the consumer JRE work could become the defacto module system for the Java platform. The Java platform would still lack a properly architected built-in module system.

JSR 277 also has a stated objective of interoperating with JSR 291, which means JSR 277 modules will be able to make effective use of OSGi modules and vice versa with minimal restrictions. If this objective is achieved, it will protect the significant investment in OSGi modules in the Java community. Scrapping JSR 277 would jeopardise this investment if a non-interoperating module system grew up in place of JSR 277.

The road ahead

So, given the momentum behind JSR 277 and the emphasis placed on it by Sun at this year's JavaOne, what lies ahead?

Of course, JSR 291 should ideally benefit from the features introduced by JSR 277 and JSR 294. But the immediate requirement is that JSR 277 and JSR 291 interoperate well so that modules written in terms of one JSR are able to make effective use of modules written in terms of the other JSR.

To achieve this, the Expert Groups of JSR 277 and JSR 291 will need to work together. The JSR 277 Expert Group are responsible for producing an interoperation proposal which the JSR 291 Expert Group will review. At the time of writing, the interoperation proposal is still being worked on by the JSR 277 spec. lead.


The dream solution

The dream solution is clear: JSR 277 should adopt JSR 291, underpinned by language and VM support in JSR 294, and add the JSR 277 repository architecture. This would accelerate the progress of JSR 277 by several years and guarantee perfect interoperation between JSR 277 and JSR 291.

But it would take a complete about turn by Sun to turn this dream into reality. Maybe, just maybe, Sun are sufficiently committed to the success of Java that they will eventually decide to do just that. Maybe, just maybe, the Java community will be able to look back later and paraphrase Winston Churchill: "Sun could always be counted on to do the right thing, after having exhausted all the alternatives".

Further information

The JSR 291 Expert Group mailing list and the JSR 294 list have been openly readable since their Expert Groups started. JSR 277 ran in private for many months, but has now opened up its list.

Disclaimer

I'm biassed.

I have been involved in the OSGi Core Platform Expert Group for a couple of years during which time I co-authored some of the modularity improvements for OSGi R4 with Richard Hall (the leader of the Apache Felix OSGi project).

I work for IBM which belongs to the OSGi Alliance and which has based some of its products on OSGi. IBM and Sun co-operate on Java, but compete in related areas such as Java EE, not to mention UNIX-style operating systems and hardware. So IBM and Sun don't necessarily agree on everything that happens in Java. In fact, I think it would be pretty unhealthy if that were the case.

I should also re-iterate that my opinions are not necessarily shared by IBM.

That said, I hope this blog objectively lays out the situation.

Eclipse Europa

Coté's coverage of Eclipse Europa expresses a view on OSGi and JSR 277.

Representations of module dependencies

One of the key differences between JSR 291 and JSR 277 is the way in which module dependencies are represented, satisfied, and managed. Both module systems express dependencies using import declarations with version ranges and so, in general, there can be multiple ways of satisfying any particular dependency.

Using an electronics analogy, there can be multiple different ways of wiring a module's dependencies to other modules. Without further constraints, the choice of wiring is essentially non-deterministic, which would make the module system hard to understand and unpredictable in behaviour. The non-determinism needs to be removed.

JSR 291 removes the non-determinism by specifying rules for how dependencies should be satisfied in the presence of a given collection of modules, some of which are already resolved (or 'wired') and others which are unresolved (not yet 'wired').

JSR 277 also specifies some rules to remove the non-determinism, but it allows a module to implement an import policy if the module wants to override these rules. The hope is that, in many simple cases, the import policy will not be needed. But, in general, JSR 277 removes the non-determinism using a combination of rules in the specification and user-written import policies.

Why is this difference important? Well, an import policy is a piece of Java code running inside the module whose dependencies need to be satisfied. So the import policy is in the unusual position of not being able to use the modules upon which its module depends. This is a potential source of errors.

But a more important difference relates to the need to be able predict the behaviour of a collection of modules. This is crucial when managing the dependencies of modules. With JSR 291, an external management system is able to read the dependency declarations in each module and apply the rules in the specification to determine how these modules will be wired up, whether there are any missing dependencies, and, if there are, how such dependencies could be satisfied.

With JSR 277 the position is quite different when import policies are used. The only way to determine the behaviour of an import policy is to execute it. But even then, there is no guarantee that the import policy will give the same result each time it runs. Also, if there are missing dependencies, it is not feasible to examine the import policy to determine how the missing dependencies could be satisfied.

(For further reading, I recommend Peter Kriens' JSR 277 review from last October.)

Module-superpackage relationship

In the comparison of JSR 277 and JSR 291 features, I alluded to the fact that the module-superpackage relationship is a difference, but Andreas pointed out another way in which the relationship differs.

My original observation was that the cardinality of the relationship is different. JSR 277 specifies modules which build directly on superpackages: there is a one-one relationship between deployment modules and superpackages.

JSR 291, on the other hand, makes no explicit reference to superpackages as these are yet to find their way officially into Java. But, if JSR 294 continues in its current direction, it will be possible to include one or more superpackages inside a JSR 291 module. So the relationship between JSR 291 modules and superpackages will have 1:0..n cardinality.

Andreas' point was that JSR 277 is integrated with JSR 294 in that a JSR 277 module's metadata consists of a superpackage and its annotations.

JSR 291, in contrast, is currently agnostic about superpackages and their names, members, exports, and annotations. JSR 291 maintains its own metadata in a separate manifest file and so is not integrated with JSR 294.

Sun VP 'gets' OSGi

I just ran across a video where James Governor interviews Jeet Kaul, the VP of developer products and programmes for Sun Microsystems. If you don't have much time, fast forward to three minutes from the start and listen to the next minute and a half on OSGi.

Some soundbytes plucked slightly out of context for maximum effect:

JG: "I'm looking at OSGi ... and I think it's possibly the most important standard in the industry right now." (What a nice man.)

JK: "... so much has been done already that it would be insane to ignore that ..."

and later

JK: "... in Java SE 7 where we are trying to build the standard which will incorporate supporting OSGi bundles - it's the right approach."

Default export granularity

In the comparison of JSR 277 and JSR 291 features, I caused some confusion by saying the default export granularity was 'class' for JSR 277 and 'package' for JSR 291. I'll try to explain.

The primitive unit of export in JSR 277 is the class. You can use a wildcard to export all of a module's classes in a specified package, but that's just syntactic sugar. The wildcard will be expanded to a list of classes in the binary file representing the module's superpackage. There is no runtime notion of an exported package.

The primitive unit of export in JSR 291, on the other hand, is the package. You can filter the set of classes available to importers, but the runtime deals in exported packages rather than classes.

I hope that's a bit clearer. Also, try not to confuse default export granularity with import semantics. JSR 277 supports only module import whereas JSR 291 supports package and module imports.

Comparison of JSR 277 and JSR 291 features

A number of people have expressed an interest in the similaries and differences between these JSRs, so this post compares the features of the Early Draft Review specification of JSR 277 and the Final Specification of JSR 291, although you'll have to read the specifications to understand the differences in detail.

This a functional comparison and doesn't attempt to draw out advantages and disadvantages of various features or other aspects of the JSRs. For ease of comparison, I've referred to JSR 291 bundles as modules.

Equivalent features:

• Module name
• Module version
• Module attributes
  
• Module import
• Module re-export
  
• Version range
• Optional import
• Executable modules
• One class loader per module
• Platform dependencies
• Modules may contain classes, native libraries, resources, and extensible metadata
  
• Module signing
• JAR distribution format
• Reflective API
  

Differences (order: JSR 277 vs JSR 291):

• Programmatic import policy vs declarative import constraints (details)
  
• Class space consistency: validation vs constraint solving
  
• Default export granularity: class vs package (details)
  
• Module-superpackage relationship (details)
  

Unique to JSR 277:

• Standard repository:
• organised hierarchically with parental delegation
• multiple standard implementations for local or networked deployment
• support for plugging in 3rd party repository implementations
• JAM distribution format
• Import override policy
  
• Part of the Java SE platform (planned for Java 7)
  

Unique to JSR 291:

• Dynamic lifecycle support both for module lifecycle and service lifecycle
• Service support (above and beyond the basic service provider support in Java SE)
  
• Package import
• Dynamic package import
  
• Package attributes
• Mandatory attributes
• Split package support
• Module classpath
  
• Fragment modules
• Extension modules
• Pure Java implemented on top of Java SE
• Compatible with Java ME (JSR 232)
  

JSR 277 and JSR 294 observer lists

You may like to know there is a public observer list for JSR 277 and one for JSR 294 too, both with publicly visible archives.

This is not obvious from the JSR homepages.

Module system interoperation kernel

Interoperation is currently a hot topic on the JSR 277 mailing list -- see the "Relationship to JSR 291" thread in the archive.

One of the approaches to interoperation is a 'kernel'. What follows is an extract from the README of the prototype plus the related class diagrams.

Goal

The goal is to enable JSR 277 to interoperate in a first class way with JSR 291 (OSGi). JSR 277 modules must be able to use JSR 291 modules and vice versa without significant restrictions.

The framework can probably enable other types of module systems to interoperate, but that isn't the primary goal.

Strawman

The code [of the prototype] is only a strawman. It is not integrated with Felix or any other OSGi implementation. Nor is it integrated with JSR 277.

However, it outlines the interfaces that are likely to be needed to enable such module systems to interoperate well and provides several JUnit 4.1 testcases which drive the framework to illustrate the framework's features.

The code is licensed under the Apache License v2.0. See LICENSE.txt.

Module systems and wiring

The lack of a standard method of building delegation networks between class loaders requires each module system to interoperate specifically with each other module system. The framework provides a standard way for module systems to create delegation paths, or 'wires', to modules in other module systems.

We need a module system registry or 'kernel' to keep track of the available ModuleSystems. Something like the following.

public interface Kernel { 
    boolean registerModuleSystem(String moduleSystem, ModuleSystem ms);
    boolean deregisterModuleSystem(String moduleSystem);
    ModuleSystem getModuleSystem(String moduleSystem);
    ModuleSystem[] getModuleSystems();
    ...
}

Module systems need to co-operate to resolve references to modules in other module systems. So each module system has an associated wire factory.

public interface WireFactory extends ModuleSystemRelated {
    Wire resolve(Dependency dep);
    Wire create(ResolutionContext ctx, Dependency dep);
    Wire createLocal(ResolutionContext ctx, Dependency dep);
    void complete(ResolutionContext ctx);
}

The most basic kind of dependency consists of a module name and a module version range. More sophisticated dependencies, such as package dependencies, are conceivable but omitted since they introduce much extra complexity.

A wire is a delegation path to the classloader of another module.

That leaves resolution contexts...

Resolution context

A resolution context is needed so that module systems can co-operate to resolve cyclic dependencies without ending up in an infinite loop. A resolution context also provides somewhere to anchor the state for sharing partially resolve modules and for back-tracking.

Constraint checking and back-tracking

A module system can anchor state off a resolution context so that, when asked more than once to create a wire for a given dependency, it can return each possibility in turn. To avoid interference between similar dependencies, new Dependency instances must be created for each module. If two modules have what appear to be the same dependency, then there need to be two Dependency instances.

Diagrams

The kernel class hierarchy shows the kernel with which module systems are
registered. Module systems and resolution contexts are each related to a
particular kernel (of which there is usually one per VM).
The module system class hierarchy shows wire factories, dependencies, and
wires which are each related to a particular module system.

Designing module system interoperation

The various approaches to interoperation between JSR 277 and JSR 291 (OSGi) raise some interesting design considerations. The crucial distinction between these JSRs is that JSR 277 aims to be part of the Java 7 platform whereas JSR 291 is implemented as pure Java that will run on more or less any Java platform.

The JSR 277 Expert Group, or more accurately the spec. lead, is exploring the option of making some components such as the repository, module definition, and module instance pluggable and then having JSR 291 provide alternative implementations. This requires JSR 291 to surface its modules as JSR 277 modules, which will either require some generalisation of the notion of a JSR 277 module or a force fit. A force fit would probably result in unacceptable restrictions on interoperation and should be avoided.

A more ambitious approach, favoured by several Expert Group members including myself, is to rework JSR 277 to provide a 'custom module system' API analogous in concept to the way custom class loaders can be built on top of the existing class loader API. In this scheme, the JSR 277 module system would simply be the default module system packaged with the Java 7 platform and JSR 291 could implement its module system on top of the API. The challenge in this approach would be to design the API so that multiple module systems implemented to the API would naturally interoperate. A benefit of this design is that it would allow module system implementations to evolve faster than the underlying platform - something that JSR 277 doesn't yet enable its own module system to do.

A third approach, which I prototyped to prove that reasonable interoperation is achievable, involves a module system interoperation 'kernel' with which multiple module systems may register. Each module system provides a class, known as a 'wire factory' , for creating delegation paths ('wires') to its modules' class loaders.

One benefit of both the custom module system API and of the interoperation kernel is that they do not complicate the API that the module developer has to deal with. The 'plugin' approach requires generalisations of the concepts of repository, module, and module definition to be exposed on the module developer API. Although this kind of generalisation may actually improve the API, it could also end up making it more obscure. We'll have to wait and see.

Modularity at JavaOne

Modularity had a lot of focus at last week's JavaOne conference. There were multiple references to JSRs 277 and 294 by Sun speakers, specific technical sessions on these JSRs, and a combined JSR 277 and 294 BOF.

There were a number of questions and comments at the JSR 294 session on the strawman's structuring of superpackage membership information in a separate source file rather than distributed across the source files of the member classes. This is something we should discuss on the mailing list (which you can read via the observer list).

Apart from an OSGi Best Practices technical session and the JSR 291 BOF, there were several mentions of OSGi and JSR 291 in other sessions. The goal of interoperating well with JSR 291 was presented in the JSR 277 session and even mentioned in one of Sun's general sessions¹. Stanley Ho will float a proposal on the JSR 277 mailing list in due course, so subscribe to the observer list if you are interested (Expert Group discussions should begin to be mirrored there soon).

Ethan Nicholas gave an update on the browser edition work to minimise the download and initial install size of the JRE. This seems to be making good progress, although it is not based on any of the above JSRs, so there may be some 'shake-out' in due course.

In one of the "meet the team" sessions, I asked Sun's class library team what their plan was for exploiting JSR 277 and JSR 294 in the JRE. It seems they don't yet have one. I expect this to change, particularly after the JSR 277 and JSR 294 prototypes become available in a subproject of OpenJDK.

In contrast, the Apache Harmony session described how OSGi had been used from the start to modularise the class libraries. I gave a demo. of prototype JVM support for modular Harmony in the JSR 291 BOF.

GlassFish v3 is developing a module system known as HK2 ("Hundred Kilobytes Kernel") which is loosely based on JSR 277. It also has a dependency injection framework similar to Guice. (Unlike GlassFish which is mostly licensed under the CDDL, HK2 currently has no license defined, so you may want to avoid looking at the source code until there is a proper license.)

I had face to face meetings with Andreas Sterbenz to discuss aspects of JSR 294 and with Stanley Ho and Michał Cierniak to discuss interoperation of JSR 277 with JSR 291. There's a lot of work ahead and we won't know for quite a while how good the interoperation with JSR 291 will really be, but everyone concerned seems to be well aware of the importance of this capability.

1. See a couple of minutes from Intro. and Technical Deep Dive starting at 18 minutes and 45 seconds from the start when Danny Coward talks about modularity and refers to the goal of interoperation with OSGi.

OSGi for application modularity

InfoQ published an article on why one company has decided to use OSGi, in conjunction with Spring, for application modularity.

They are not starting from scratch, so it will be interesting to see how pleased they are with the result. On the face of it, OSGi is a good match to their requirements, particularly for versioning and for deploying application updates without restarting the server.