JSR 291 but not as we know it

Google sometimes coughs up unexpected stuff, like this which could almost be talking about JSR 291 and JSR 277. :-)

Eclipse Europa

Coté's coverage of Eclipse Europa expresses a view on OSGi and JSR 277.

Representations of module dependencies

One of the key differences between JSR 291 and JSR 277 is the way in which module dependencies are represented, satisfied, and managed. Both module systems express dependencies using import declarations with version ranges and so, in general, there can be multiple ways of satisfying any particular dependency.

Using an electronics analogy, there can be multiple different ways of wiring a module's dependencies to other modules. Without further constraints, the choice of wiring is essentially non-deterministic, which would make the module system hard to understand and unpredictable in behaviour. The non-determinism needs to be removed.

JSR 291 removes the non-determinism by specifying rules for how dependencies should be satisfied in the presence of a given collection of modules, some of which are already resolved (or 'wired') and others which are unresolved (not yet 'wired').

JSR 277 also specifies some rules to remove the non-determinism, but it allows a module to implement an import policy if the module wants to override these rules. The hope is that, in many simple cases, the import policy will not be needed. But, in general, JSR 277 removes the non-determinism using a combination of rules in the specification and user-written import policies.

Why is this difference important? Well, an import policy is a piece of Java code running inside the module whose dependencies need to be satisfied. So the import policy is in the unusual position of not being able to use the modules upon which its module depends. This is a potential source of errors.

But a more important difference relates to the need to be able predict the behaviour of a collection of modules. This is crucial when managing the dependencies of modules. With JSR 291, an external management system is able to read the dependency declarations in each module and apply the rules in the specification to determine how these modules will be wired up, whether there are any missing dependencies, and, if there are, how such dependencies could be satisfied.

With JSR 277 the position is quite different when import policies are used. The only way to determine the behaviour of an import policy is to execute it. But even then, there is no guarantee that the import policy will give the same result each time it runs. Also, if there are missing dependencies, it is not feasible to examine the import policy to determine how the missing dependencies could be satisfied.

(For further reading, I recommend Peter Kriens' JSR 277 review from last October.)

Module-superpackage relationship

In the comparison of JSR 277 and JSR 291 features, I alluded to the fact that the module-superpackage relationship is a difference, but Andreas pointed out another way in which the relationship differs.

My original observation was that the cardinality of the relationship is different. JSR 277 specifies modules which build directly on superpackages: there is a one-one relationship between deployment modules and superpackages.

JSR 291, on the other hand, makes no explicit reference to superpackages as these are yet to find their way officially into Java. But, if JSR 294 continues in its current direction, it will be possible to include one or more superpackages inside a JSR 291 module. So the relationship between JSR 291 modules and superpackages will have 1:0..n cardinality.

Andreas' point was that JSR 277 is integrated with JSR 294 in that a JSR 277 module's metadata consists of a superpackage and its annotations.

JSR 291, in contrast, is currently agnostic about superpackages and their names, members, exports, and annotations. JSR 291 maintains its own metadata in a separate manifest file and so is not integrated with JSR 294.

Sun VP 'gets' OSGi

I just ran across a video where James Governor interviews Jeet Kaul, the VP of developer products and programmes for Sun Microsystems. If you don't have much time, fast forward to three minutes from the start and listen to the next minute and a half on OSGi.

Some soundbytes plucked slightly out of context for maximum effect:

JG: "I'm looking at OSGi ... and I think it's possibly the most important standard in the industry right now." (What a nice man.)

JK: "... so much has been done already that it would be insane to ignore that ..."

and later

JK: "... in Java SE 7 where we are trying to build the standard which will incorporate supporting OSGi bundles - it's the right approach."

Default export granularity

In the comparison of JSR 277 and JSR 291 features, I caused some confusion by saying the default export granularity was 'class' for JSR 277 and 'package' for JSR 291. I'll try to explain.

The primitive unit of export in JSR 277 is the class. You can use a wildcard to export all of a module's classes in a specified package, but that's just syntactic sugar. The wildcard will be expanded to a list of classes in the binary file representing the module's superpackage. There is no runtime notion of an exported package.

The primitive unit of export in JSR 291, on the other hand, is the package. You can filter the set of classes available to importers, but the runtime deals in exported packages rather than classes.

I hope that's a bit clearer. Also, try not to confuse default export granularity with import semantics. JSR 277 supports only module import whereas JSR 291 supports package and module imports.

Comparison of JSR 277 and JSR 291 features

A number of people have expressed an interest in the similaries and differences between these JSRs, so this post compares the features of the Early Draft Review specification of JSR 277 and the Final Specification of JSR 291, although you'll have to read the specifications to understand the differences in detail.

This a functional comparison and doesn't attempt to draw out advantages and disadvantages of various features or other aspects of the JSRs. For ease of comparison, I've referred to JSR 291 bundles as modules.

Equivalent features:

• Module name
• Module version
• Module attributes
  
• Module import
• Module re-export
  
• Version range
• Optional import
• Executable modules
• One class loader per module
• Platform dependencies
• Modules may contain classes, native libraries, resources, and extensible metadata
  
• Module signing
• JAR distribution format
• Reflective API
  

Differences (order: JSR 277 vs JSR 291):

• Programmatic import policy vs declarative import constraints (details)
  
• Class space consistency: validation vs constraint solving
  
• Default export granularity: class vs package (details)
  
• Module-superpackage relationship (details)
  

Unique to JSR 277:

• Standard repository:
• organised hierarchically with parental delegation
• multiple standard implementations for local or networked deployment
• support for plugging in 3rd party repository implementations
• JAM distribution format
• Import override policy
  
• Part of the Java SE platform (planned for Java 7)
  

Unique to JSR 291:

• Dynamic lifecycle support both for module lifecycle and service lifecycle
• Service support (above and beyond the basic service provider support in Java SE)
  
• Package import
• Dynamic package import
  
• Package attributes
• Mandatory attributes
• Split package support
• Module classpath
  
• Fragment modules
• Extension modules
• Pure Java implemented on top of Java SE
• Compatible with Java ME (JSR 232)
  

One last hurdle for JSR 291?

The PMO acknowledged receipt of the Final Release materials I sent them a couple of days ago, so at least their email system worked this time. However, they said the spec. license and the final business terms on the OSGi site would need another legal review. Nice to know I'm keeping lawyers as well as customs people in jobs!

It's a funny thing since the JCP process (section 3.5) doesn't mention a legal review, although the spec. lead guide does. There's no state in the process for having failed a legal review after the FAB, so I guess the JSR will just stay in limbo until the lawyers are happy.

I pointed out that nothing has changed since the Final Approval Ballot, so let's hope the legal review is just a formality...

JSR 291 Final Release on its way

"But I thought JSR 291 had already gone final?", I hear you say. Nope: it passed the Final Approval Ballot a few weeks ago, but is still to enter Final Release state.

Since then I've been getting hold of an Export Classification Control Number (ECCN), which is really the only additional information needed by the JCP PMO to publish the Final Release. Having obtained an ECCN¹, I emailed the PMO today and await their response.

I should mention that the JCP bureaucracy needn't hold up anyone who wants to use JSR 291, as everything they need is already available.

[1] The ECCN is EAR99 for anyone who's interested, although I can't imagine who would be. EAR99 seems to be code for "this doesn't really need an ECCN". Keeps the customs people in jobs though.

IcedTea fork of OpenJDK

This news has been around for a few days, but I heard it first from the JavaPosse.

Red Hat has forked OpenJDK and plans to fill the encumbered gaps with GPL code in the newly formed IcedTea project (details here).

Eventually they hope to contribute the changes to OpenJDK, but what kind of contribution/governance model will they want? Will Red Hat happily sign the Sun Contributor Agreement? Meanwhile, will IcedTea qualify as an OpenJDK project and gain access to the JCK?

Lot's of fun to be had...

JSR 277 and JSR 294 observer lists

You may like to know there is a public observer list for JSR 277 and one for JSR 294 too, both with publicly visible archives.

This is not obvious from the JSR homepages.

OSGi community event

The OSGi community event in Munich is approaching fast. I can't make it, but I know there will be good speakers including BJ Hargrave, Peter Kriens, and Neil Bartlett. Register here.

Adrian Colyer on Spring & OSGi

Just dipped into one or two segments of Adrian Colyer's InfoQ video interview on Spring & OSGi. There are bookmarks for anyone else who doesn't want to hear the full half hour.

More OSGi tutorials

Neil Bartlett has published another excellent OSGi tutorial in his EclipseZone series, this time introducing Declarative Services.

A must for anyone learning OSGi, and I include myself since I'm not an application programmer.

He'll also be giving a talk at the OSGi Community Event in Munich later this month.

Module system interoperation kernel

Interoperation is currently a hot topic on the JSR 277 mailing list -- see the "Relationship to JSR 291" thread in the archive.

One of the approaches to interoperation is a 'kernel'. What follows is an extract from the README of the prototype plus the related class diagrams.

Goal

The goal is to enable JSR 277 to interoperate in a first class way with JSR 291 (OSGi). JSR 277 modules must be able to use JSR 291 modules and vice versa without significant restrictions.

The framework can probably enable other types of module systems to interoperate, but that isn't the primary goal.

Strawman

The code [of the prototype] is only a strawman. It is not integrated with Felix or any other OSGi implementation. Nor is it integrated with JSR 277.

However, it outlines the interfaces that are likely to be needed to enable such module systems to interoperate well and provides several JUnit 4.1 testcases which drive the framework to illustrate the framework's features.

The code is licensed under the Apache License v2.0. See LICENSE.txt.

Module systems and wiring

The lack of a standard method of building delegation networks between class loaders requires each module system to interoperate specifically with each other module system. The framework provides a standard way for module systems to create delegation paths, or 'wires', to modules in other module systems.

We need a module system registry or 'kernel' to keep track of the available ModuleSystems. Something like the following.

public interface Kernel { 
    boolean registerModuleSystem(String moduleSystem, ModuleSystem ms);
    boolean deregisterModuleSystem(String moduleSystem);
    ModuleSystem getModuleSystem(String moduleSystem);
    ModuleSystem[] getModuleSystems();
    ...
}

Module systems need to co-operate to resolve references to modules in other module systems. So each module system has an associated wire factory.

public interface WireFactory extends ModuleSystemRelated {
    Wire resolve(Dependency dep);
    Wire create(ResolutionContext ctx, Dependency dep);
    Wire createLocal(ResolutionContext ctx, Dependency dep);
    void complete(ResolutionContext ctx);
}

The most basic kind of dependency consists of a module name and a module version range. More sophisticated dependencies, such as package dependencies, are conceivable but omitted since they introduce much extra complexity.

A wire is a delegation path to the classloader of another module.

That leaves resolution contexts...

Resolution context

A resolution context is needed so that module systems can co-operate to resolve cyclic dependencies without ending up in an infinite loop. A resolution context also provides somewhere to anchor the state for sharing partially resolve modules and for back-tracking.

Constraint checking and back-tracking

A module system can anchor state off a resolution context so that, when asked more than once to create a wire for a given dependency, it can return each possibility in turn. To avoid interference between similar dependencies, new Dependency instances must be created for each module. If two modules have what appear to be the same dependency, then there need to be two Dependency instances.

Diagrams

The kernel class hierarchy shows the kernel with which module systems are
registered. Module systems and resolution contexts are each related to a
particular kernel (of which there is usually one per VM).
The module system class hierarchy shows wire factories, dependencies, and
wires which are each related to a particular module system.

Modular Swing Apps

Thanks to Colm Smyth for pointing out Jasper Potts' presentation from JavaOne on modularising Swing applications using a Spring-OSGi derivative known as SPAR. Nice to see OSGi being used by Sun.

OSGi the most important technology of the decade?

It may have been for me, but read the SD Times' view in OSGi Alliance Reaching Universal Middleware Goal.

JSR 291 Final Approval

JSR 291 just passed the Final Approval Ballot with twelve "yes" votes and two "no" votes.

Fourteen out of the sixteen Executive Committee members voted. The majority increased to ten compared to eight at the earlier ballots.

The positive comments praised the open approach of JSR 291, the suitability of the licenses, and the effectiveness of JSR 291 in influencing the other modularity JSRs. One comment urged JSR 277 and JSR 291 to become integrated, which will be achieved if JSR 277 delivers truly first class interoperation with JSR 291.

Voting comments from the detractors predictably concerned the work of the Expert Group rather than the underlying OSGi technology. They don't seem to like the JCP building on the work of other Java standards bodies. The concerns are discussed on the Expert Group mailing list and in an earlier blog.

I would like to express my thanks to many people who have contributed to the success of this JSR. They are too numerous to mention by name but include the JSR 291 Expert Group, the OSGi Alliance's Core Platform Expert Group, the Eclipse Equinox project, and numerous colleagues in IBM and elsewhere.

Designing module system interoperation

The various approaches to interoperation between JSR 277 and JSR 291 (OSGi) raise some interesting design considerations. The crucial distinction between these JSRs is that JSR 277 aims to be part of the Java 7 platform whereas JSR 291 is implemented as pure Java that will run on more or less any Java platform.

The JSR 277 Expert Group, or more accurately the spec. lead, is exploring the option of making some components such as the repository, module definition, and module instance pluggable and then having JSR 291 provide alternative implementations. This requires JSR 291 to surface its modules as JSR 277 modules, which will either require some generalisation of the notion of a JSR 277 module or a force fit. A force fit would probably result in unacceptable restrictions on interoperation and should be avoided.

A more ambitious approach, favoured by several Expert Group members including myself, is to rework JSR 277 to provide a 'custom module system' API analogous in concept to the way custom class loaders can be built on top of the existing class loader API. In this scheme, the JSR 277 module system would simply be the default module system packaged with the Java 7 platform and JSR 291 could implement its module system on top of the API. The challenge in this approach would be to design the API so that multiple module systems implemented to the API would naturally interoperate. A benefit of this design is that it would allow module system implementations to evolve faster than the underlying platform - something that JSR 277 doesn't yet enable its own module system to do.

A third approach, which I prototyped to prove that reasonable interoperation is achievable, involves a module system interoperation 'kernel' with which multiple module systems may register. Each module system provides a class, known as a 'wire factory' , for creating delegation paths ('wires') to its modules' class loaders.

One benefit of both the custom module system API and of the interoperation kernel is that they do not complicate the API that the module developer has to deal with. The 'plugin' approach requires generalisations of the concepts of repository, module, and module definition to be exposed on the module developer API. Although this kind of generalisation may actually improve the API, it could also end up making it more obscure. We'll have to wait and see.

Open Source Java at JavaOne

Much was made of OpenJDK at last week's JavaOne conference. On Tuesday morning, Rich Green delighted in telling the assembled masses "We are now, as regards the open-sourcing of Java, done" and then posted the following:

From: Rich Green  
To: announce@openjdk.java.net
Subject: Open JDK is here!

Today begins the next phase for Java. I am pleased to  announce that  
we have completed the open source release of Open JDK.
It's a great day for innovation - and remember, compatibility matters!
Rich

It turns out there is a bit more to do, like replacing some of the 'encumbered' code with a GPL version. Currently, the 6.5 MLOCs of OpenJDK source code must be built with some binaries which provide the encumbered code. Also OpenJDK doesn't yet build on Windows.

Sun have set up an Interim Governing Board to create a constituion for OpenJDK and to set up a proper Governing Board. The IGB has the following initial membership: Doug Lea, Fabiane Nardon, Dalibor Topić, and Sun's own Mark Reinhold and Simon Phipps.

The Apache Harmony technical session was well attended with more focus on the technology than on licensing issues and the JCK open letter.

Some anomalies still need working out. Sun said at JavaOne that it would release the JCK to the open source community, but later clarified that this meant the OpenJDK community. I guess that might even exclude forks of OpenJDK, but we'll have to wait and see.

Also, there is some strangeness in the way copyright assignment will be handled for eventual non-Sun committers to OpenJDK. The GNU definition of free software describes four freedoms including "The freedom to improve the program, and release your improvements to the public, so that the whole community benefits". So, in normal GPL projects, no-one has the right the make a fork without publishing their changes under the GPL.

Contrast this with the Apache license which allows anyone to create a fork and essentially do what they like with their changes, either publishing them or keeping them private.

OpenJDK is different from both these models. Sun, and only Sun, has the right to fork the code, make changes, and keep the changes private, although it seems unlikely that Sun would ever want to do that.

At this point when the source code is Sun's donation and when Sun is obliged to provide the source code to its licensees under the TLDA license, this is quite reasonable. But I wonder how many potential contributors will be happy to sign the Sun Contributor Agreement and give Sun these extended rights over their contributions.