Project Jigsaw

Mark Reinhold recently announced Project Jigsaw for modularising the JDK. Could this be a step towards the dream solution? I can hope.

Meanwhile JSR 277 is deferred beyond Java 7. Joy.

Refactoring and Structure101

I'm in the middle of a fairly broad refactoring of the internals of the SpringSource dm Server to introduce an interface to some of the subcomponents and ensure that the rest of the code accesses those subcomponents via the interface rather than directly.

The Structure101 tool has proved invaluable in this process. It has excellent support for analysing the dependencies between components and identifying anomalous dependencies such as those from lower layers of the code to higher layers. But it also supports transformations and visibility settings which turn out to be extremely useful for refactoring.

I created transformations to collect together packages which I want to treat as a unit from the point of view of the current refactoring. This would not be possible using standard refactoring techniques as the packages in question cannot actually be renamed, for reasons I won't go into here. The transformations were of the form:

org.foo.* -> myunit.org.foo.*
org.bar.* -> myunit.org.bar.*

The transformations apply only in Structure101's model of the system and do not affect the actual code. The net is that I can then analyse dependencies on the myunit package hierarchy using Structure101's default mechanisms.

Visibility settings come in useful in identifying the pieces of code I need to refactor. By modifying the Structure101 model so that myunit is a private subcomponent of the new interface component, all dependencies on myunit from outside of the interface component show up as anomalies.

I can then use these anomalies as a 'to do' list for my refactoring. Clicking on one of them shows the precise code dependency which I can refactor by extending the interface component, if necessary, and then changing the using code to depend on the interface component. Refreshing the Structure101 model takes a few seconds after which there is one less anomaly showing on the diagram.

I've really appreciated the support from the Structure101 developers who have helped me get the best out of the tool. It sure beats some combination of grep, a cross-referencing tool like OpenGrok, plus a ton of yellow stickies...

100% bloat free

Here's the image from the T-shirts that were given away at JavaOne. Click the image to enlarge it.

SpringSource Application Platform ships

Today the SpringSource Application Platform shipped its first beta release. Version 1.0 is essentially a combination of Spring, OSGi and Tomcat engineered into a stand-alone runtime.

The programming model is Spring with OSGi and the option of a new packaging format: the platform archive or PAR file. A PAR file comprises multiple OSGi bundles. This brings OSGi modularity and versioning to enterprise applications.

The diagram below gives a high level view of the structure of the platform.

So what does the platform deliver over and above a standard OSGi container? Well, apart from the servlet function, integrated Spring support, the modular kernel-based runtime, and the full support for diagnostics and first failure data capture, there are quite a few features that are of interest to an OSGi audience:

• The platform has a repository which automatically provisions dependencies as bundles are installed into OSGi. This repository stores the platform's own bundles as well as application and third party bundles.
• In addition to the PAR mentioned above, there is a new concept of a library which is a named, versioned collection of OSGi bundles installed in the repository typically for use by applications. This is accessed via a new manifest header, import-library, which imports all the packages exported by the bundles of a library.
• Another new manifest header, import-bundle, imports all the packages exported by a single bundle. This avoids the need to wrap components consisting of a single bundle in a library.
• There is also some novel diagnostic help when a bundle cannot be resolved. The platform's resolution failure detective attempts to extract a concise summary of the problem and the most likely root causes.
• Associated with the platform is a carefully constructed repository of commonly used bundles: the SpringSource Enterprise Bundle Repository known informally as the "bundle repository in the sky", or "BRITS". The bundles have validated OSGi manifests and the whole repository has been checked to make sure that its bundles can be used together.
  

Finally, I can't resist saying a bit about the architecture of the platform. In the past, I've been involved in restructuring projects - adding in modularity after the fact. The platform was built in a modular fashion from day 1.

The whole platform consists of subsystems which are similar in concept to libraries but which are managed specially by the kernel. Each subsystem is a named, versioned collection of bundles. The kernel manages the installation and starting of subsystems as the platform initialises. The kernel manages the lifecycle of each subsystem and its bundles, even reacting to unsolicited lifecycle events of the bundles of a subsystem to keep the state of the subsystem consistent.

Subsystems are used to control the function of the platform. Some subsystems are always present, but others are optional and are configured in a profile. This mechanism allows the precise function of the platform to be configured without rebuilding. For instance, the shipped profile.config file (in the config directory) looks like this:

/*
 * SpringSource Application Platform profile manager default
 * configuration file.
 */
{
 "profile": {
   "version" : 1.0,
   "name" : "web",
   "subsystems" : ["com.springsource.platform.servlet",            
                   "com.springsource.platform.web"]
 }
}

which shows the servlet container and web personality subsystems.

My colleague Rob Harrop just blogged about the platform if you want to know more. Why not register, download the binary, have a play, and tell us what you think.

OSGi could be great for JSR 277

Bryan Atsatt describes a plausible way forward for JSR 277. I agree with his reasoning, although OSGi is not only the de facto standard for Java modularity, it's a proper standard too.

Bryan was one of the key movers in the Expert Group when I was involved. In fact, he proposed a solution to JSR 294 ages ago which is close, if not identical, to the approach Alex Buckley is now considering.

Welcome to the blogosphere, Bryan!

Popular RFE

My favourite sunbug is now on the list of the Top 25 RFEs (Requests for Enhancement).

The 11th most popular sunbug?

Sunbug 6650394 is coming along nicely - both in terms of votes and supportive comments. At the time of writing, it has 99 votes, so thank you if you took the trouble to vote! If you care and you haven't yet voted or commented, please consider doing so.

I was hoping the bug would make it into Sun's Top 25 Bugs list since it has more votes than most of the bugs in that list, but for some reason it's not yet included. If the list was accurate, which it clearly isn't, then 6650394 would be the 11th most popular sunbug.

Sunbug for JSR 277 module system interoperation

Sun has kindly raised sunbug 6650394 on my behalf to track the module system interoperation issue for JSR 277.

Please consider voting for this bug if you would like JSR 277 to interoperate with JSR 291 (OSGi). Add yourself to the watch list if you want to track the bug.

JSR 294 Early Draft

The Early Draft Review of JSR 294 is now complete. Little seems to have changed since when I left the Expert Group. The main weakness from my perspective is that a class file must name its superpackage in order for the class to be a member of the superpackage. This means that you can't easily produce a program which runs on existing versions of Java but which will take advantage of superpackages when it runs on Java 7 and beyond.

Practically speaking, this means that superpackages will only be used in applications that are willing to restrict themselves to Java 7 and beyond. So I suspect that superpackages will not come into popular usage within the next few years.

Of course, this can be gotten round by using a custom class loader with some bytecode rewriting magic, but one of the reasons that JSR 277, and its progeny JSR 294, were introduced was to avoid application developers from having to create custom class loaders.

SpringSource

A couple of weeks in the new job and I work for a different company! Well almost - Interface21 has been renamed to SpringSource to strengthen the association with the Spring portfolio of products.
New web site, new email addresses, new logo, new business cards, etc. On the other hand the job is unchanged, as is the green corporate colour scheme...

Life in a small company

I'm gradually noticing what's different about working in a small company. For one thing, there are very few processes, other than for software development where there are one or two.

For instance, I just took this photo. of the room I work in. Damilola (on the left) and Chris (on the right) noticed the idiot lifting his macbook into mid-air, but Nancie (in the distance) apparently didn't. I wonder how many levels of sign-off would have been needed to publish such a photo. in my previous company?

A more telling example of the minimal processes was that it recently took about an hour to obtain licenses for a piece of commercial software we decided to use. This included getting financial sign-off from someone on a business trip on the West Coast, submitting the order, and getting the license keys ready to use.

This is a pleasant change for me. Processes are sometimes essential, but it's so easy to make them too heavy. The emphasis in a small company seems to be talking to other people and trusting them to do the right thing.

Joining Interface21

Today I start work for Interface21 in Southampton. After more than a month of gardening leave, or more accurately decorating leave, I can't wait!

OSGi, type-safety, and the 'uses' directive

Java preserves type-safety when loading classes, even if there are buggy or malicious custom class loaders present. This is important as without type-safety, Java would not be secure.

For example, it would be possible to access an object's private data using a spoofed version of a class with the private data declared public. Vijay Saraswat found just such a type-safety bug in an early version of Java which was fixed by the introduction of 'loading constraints'.

So now that Java is type-safe, why need OSGi be concerned? Well, although OSGi can't break Java's type-safety, its support for versioning increases the risk that exceptions, such as ClassCastException, will be thrown to preserve type-safety. This usually occurs where two loaded versions of a particular class come into 'contact'.

The most common case is where an attempt is made to cast an object of one version of a class to another version of the same class. There isn't necessarily anything malicious going on - it's simply that two legitimate versions of a class loaded in the same JVM have accidentally come into contact.

OSGi increases the risk of such exceptions because it supports versioning of bundles and their contents: two (or more) versions of a given bundle can run in a JVM at the same time. These bundles will typically export similar classes. Since each OSGi bundle has its own class loader and a class is uniquely identified at runtime by the combination of its fully qualified class name and the class loader instance that defined the class, there can be two versions of each exported class present in the JVM at the same time.

So, to minimise these problems, OSGi works hard to wire together bundles so that they will consistently use a particular version of each class. At a basic level, when OSGi is resolving the dependencies of a bundle, it uses resolved bundles to satisfy those dependencies in preference to unresolved bundles. This avoids gratuitously introducing two or more versions of a given bundle, together with its exported classes, into the system.

But sometimes multiple versions of a class can, and indeed must, legitimately coexist. This can occur when assembling a large system from pieces which have been developed independently. The separate pieces can pre-req. distinct versions of common bundles. If these common bundles don't 'leak out' of the pieces of the system that use them, there is no problem.

But if they do leak out, OSGi's 'uses' directive enables the bundles importing from the common bundles to specify the amount of 'leakage'. For example, the following method signature exposes a common type PType:

package org.bar.q;
...
public org.foo.common.p.PType someMethod() {...}

and the following manifest statements record the leakage:

import-package: org.foo.common.p
export-package: org.bar.q,uses:="org.foo.common.p"

OSGi always resolves bundle dependencies so as to respect the 'uses' constraints. There's even support in Peter Kriens' bundle manifest creation tool, bnd, which spots the most straightforward leakages, like the example above, and create the corresponding 'uses' clauses automatically.

More subtle leakages need to be handled by the bundle programmer, but usually it is fairly obvious that something tricky is going on. For example, the following method take a parameter and casts it to the common type PType:

package org.bar.q;
...
public void anotherMethod(Object o) {
... (org.foo.common.p.PType)o ...
}

To avoid class cast exceptions due to o being an instance of the wrong version of PType, the same manifest statements as in the first example above should be specified, although these could not be inferred automatically by bnd.

So with appropriate 'uses' clauses, OSGi is able to manage multiple versions of bundles and minimise type-safety exceptions.

Writing good code

James Governor drew my attention to Laura Thomson's summary of good coding practice - like a memory jogger for Code Complete. Don't worry about the title if, like me, you have little interest in PHP.

Upcoming 'webinar' on Equinox OSGi

Tom Watson and Simon Kaegi are giving an introduction to Eclipse Equinox and OSGi at noon EST on November 27th. Register if you're interested.

I hadn't come across Eclipse live before, but the archive has some OSGi podcasts which may be of interest. I guess Tom and Simon's talk will appear there in due course.

Free Burma!

An estimated 200 people have been killed by the ruling military junta during the recent peaceful demonstrations in Burma. Although the country currently has a high profile in the news, Burma has a history of human rights abuses. More background may be found in the BBC's profile of Burma.

Today, October 4th 2007, thousands of bloggers worldwide are making a single post to raise awareness of the situation and press for change. If you have a blog, why not join in?

Leaving IBM

Today I handed in my resignation and will be leaving IBM on November 5. I've had a very enjoyable 26 years here in Hursley working on a variety of projects from workstation graphics algorithms to high-end transaction servers. A year ago I expected to work in IBM until retirement, but I had no idea what an interesting opportunity was about to open up a few miles down the road.

Although I'll still be involved with OSGi, I'll be handing over the job of representing IBM on the JSR 277 and 294 Expert Groups to someone else. I'll be lucky to find anything else quite so contentious to blog about, although I'll still be very interested in the outcome and may find something to say.

Hope for interoperation between JSR 277 and JSR 291

Infoq and Alex Miller picked up on my recent comments about the somewhat closed approach being adopted by JSR 277. Since the whole concept of interoperation between JSR 277 and JSR 291 seems to be doubted, I thought I'd better provide some reassurance.

The saving grace of both these JSRs is that they ultimately boil down to delegation networks between module class loaders. So there is a lingua franca lurking not very far under the surface. Granted the views presented to the module developer are quite different in detail.

Quite a while ago, I wrote a prototype module system interoperation framework, described in more detail here, to start to flesh out how these JSRs could interoperate. I regard it more of an existence proof that a reasonable solution may exist. However, it requires 'cutting' into the module systems at a deeper level than I think is currently being considered by the JSR 277 spec. lead and colleagues.

We discussed some of the options in an informal Expert Group meeting at JavaOne, but all we really concluded was that there was a lot of work ahead to get interoperation working smoothly. I think there is also a consensus among the interested parties that reasonable interoperation is essential to the success of JSR 277. So the sooner we can get more eyes, and brains, on the current proposal, the better.

FeatherCast

FeatherCast occasionally has interesting short podcasts about goings on in Apache.

Today I noticed a Google advert on the site for humane poultry slaughter tools, presumably targetting open source, free range chicken farmers.

How to run a JSR Expert Group

The modus operandum of the JSR 277 Expert Group is fascinating.

Strawman proposals are developed in private by the spec. lead, colleagues from Sun, and occasionally one or two others from elsewhere. These are then presented to the Expert Group for comment. After more or less debate, the spec. lead asserts the consensus and the strawman goes forward for inclusion in the draft spec. Typically, there can be a change of position when something small is clearly broken, but significant design changes and smaller tweaks tend to be resisted.

On the whole, the real JSR 277 contributors are working in private and the so-called Expert Group acts as a sounding board without any significant influence on the outcome. Some would call this rubber-stamping.

I'm trying to change this on one crucial front - that of interoperation between JSR 277 and JSR 291. But progress is dreadfully slow.

Apparently, interoperation prototypes are being developed and thinking is going on, but none of this is visible to the Expert Group or the broader community. This is a bit odd as a modules project has been set up on OpenJDK, so I would have expected that kind of prototyping to go on in a branch or subdirectory for early feedback from others.

It's worrying because the OSGi experts, Richard Hall and myself, in the JSR 277 Expert Group and the others in the JSR 291 Expert Group are not currently able to help. By the time we see a strawman, it may be too late to make any significant changes.

Clearly my experience of Apache and Eclipse is no guide to the way things happen in OpenJDK. Similarly, my experience of other standards bodies is also no guide to how a JSR Expert Group is necessarily run.