WebSphere OSGi Application Model

As noted previously, WebSphere Application Server's alpha support for OSGi applications is similar to that of SpringSource dm Server. Now IBM's open source server, Geronimo, has taken its first steps to join the party.

Apparently Geronimo 3.0 is now based on OSGi, like all other modern Java app servers. (I wonder what vestigial traces of its GBean/XBean heritage remain?)

Geronimo's OSGi application model is provided by the Aries incubator, so I wonder if that is also the base of the WAS alpha, since Geronimo is essentially WAS CE.

Maybe the answer is that the closed source WAS is also based on Aries, but I didn't dare accept the license agreement on the download to find out.

SpringSource dm Server catching on

The recently announced alpha driver of IBM WebSphere Application Server v7 support for OSGi applications is covered in a brief dzone interview with Kirk Knoernschild of the Burton Group.

As Kirk notes in his blog, SpringSource dm Server has already brought OSGi to the enterprise. dm Server v1.0 shipped in September 2008 and v2.0 will enter RC1 in the next few weeks.

The WAS alpha has some features in common with dm Server, notably repositories and applications along similar lines to plans and PARs. Immitation is the sincerest form of flattery or, putting it another way, IBM has validated dm Server's approach.

I'm still waiting for technical details of WebLogic DM Server, but since it is microkernel based (unlike WAS) and with a name like that, it's likely to validate dm Server further.

Meanwhile GlassFish and JBoss support for OSGi is coming along nicely, so it'll be fascinating to see what kind of OSGi application models these projects end up providing.

Jigsaw/JSR 294 on JavaPosse

I listened to most of the JavaPosse interview with Mark Reinhold and Alex Buckley about Project Jigsaw and JSR 294. It's nice to see some technical details emerging and I can start to see why OpenJDK would prefer a different design for modularity to that of Apache Harmony.

I was moderately surprised to find that the design point is to load multiple Jigsaw modules with "local" dependencies into the same class loader. I wonder how many class loaders the JRE will end up having? There was even talk of putting all the bootstrap classes in a single class loader. So maybe the answer is most of the JRE code will be loaded by in the bootstrap class loader and subdivided using JSR 294 constructs. Time will no doubt tell.

For the record, I hope I've never claimed that Apache Harmony has shipped runtime modularity. The Harmony class libraries are divided into bundles with OSGi metadata. A prototype that enforced those class loader boundaries in the JVM was built, but never shipped. It had a rather different design for bootstrap class loading to that being proposed for OpenJDK, but I suppose Harmony had the benefit of having modularity in mind from the start.

OSGi, Scala, maven, and git

I've noticed a few people are integrating OSGi and Scala. In particular Heiko Seeberger is building OSGi bundles for the Scala language plus various utility modules and DSLs. Then he's using maven for build and git for version control.

I am, of course, a fan of OSGi. I'm rapidly warming to git. But I'm not sure Scala and maven really solve more problems than they introduce. I don't want to dwell on maven as I'm not sure other build systems are really much better, but Scala does raise lots of question in my mind.

Many years ago, I was keen to adopt new technology and some combinations worked really well while others just seemed to multiply the overall complexity. One particular combination was the use of Z specification language embedded using literate programming in a subset of our target programming language with proof rules. It was possible to create proof trees of simple specifications refined to the target language. The main problem was that there were very few people who could acquire the skills necessary to master the combined technology.

Well, perhaps that was an extreme example and maybe it was doomed from the start, but I have a niggling doubt about the size of the Scala language.

So I'd be delighted to understand what features of OSGi and Scala work well together (and what features are simply best not used as part of this combination). In particular, are there any particular problems which are significantly simplified by combining OSGi and Scala which cannot be attacked very well using OSGi (and Java) or Scala alone?

Postscript

This document partially answers my questions. Scala makes it easy to write a DSL for handling OSGi services simply.

This is similar to the kind of function provided by Spring DM (or, equivalently, the OSGi Blueprint service).

The main difference is that there is still some minimal clutter in the Scala program whereas with Spring DM POJOs are configured in XML to provide and use OSGi services. I suppose the Scala approach favours sophisticated use of OSGi services, but this is not the norm in typical enterprise applications.

Coding and hill-walking

I've recently been re-organising the internals of the SpringSource dm Kernel so that it can be extended to provide structured install artefacts known as "plans". The current code has been extended to provide a plan equivalent to the existing PAR files, but other variants of plans, unscoped and non-atomic, cannot be implemented without the necessary data structures being put in place.

From early in the development of dm Server 1.0, the team was using several terms which summarised the behaviour of the kernel, but which were not easily identified in the code. Terms such as "scope" and "deployment pipeline". As we started thinking about 2.0, we identified "plan", "install tree", and "install environment" as basic concepts, even if the latter was pretty fluffy.

Scopes were implemented as first class types quite a while ago now and were crucial to being able to make cloning work. One of the trickiest areas was performing operations on types which could be cloned or non-cloned. Explicit scopes enabled these operations to be coded to a non-cloned interface type and re-dispatched as necessary to the appropriate cloned type.

Recently, I've been turning deployment pipeline, install tree, and install environment into first class types. I really love this kind of work. It's rather like walking through a range of hills where the higher peaks are only visible when some of the lower summits are behind you.

Sometimes however things get much more strenuous. It's like the terrain is getting much steeper and there no obvious path to follow.

This afternoon I was struggling to structure the code to handle the starting of bundles. There's an interesting coordination and state management issue because starting a bundle has a synchronous phase defined by OSGi and often an asynchronous phase defined by Spring DM or the OSGi Blueprint Service. I made some progress, but I'm still not happy with the current code and it certainly isn't easily unit tested which is a good measure of code quality.

At this point, the only option is to take a break and come back to the problem with a clear head. I tried consulting colleagues and that helped quite a bit, but there are hopefully some really good abstractions waiting to be discovered.

I think the root cause of the complexity of this small area is that it is fairly low level and we're lacking some really good ways of thinking about the asynchronous processes involved. Occasionally, modelling processes in CSP and exploring the properties of the model using a model checker such as FDR has helped. But more often than not, the best approach seems to be iteratively cleaning up a bunch of code until a good structure starts to emerge.

In the worst case, I'll make the code as clean as possible and add it to the list of areas where future discoveries will hopefully make things cleaner. But I'm hoping for a much better outcome.

Well, that's enough for now. I put this blog together as I expect other people have similar experiences when working in complex areas of software and it's good to learn from each other. I'd be interested in your perspective and how you approach these kinds of activities.

I'll leave you with a hill-walking illustration of my current position: standing on one peak trying to make out the next one in the mist.

CaseInsensitiveMap<V>

I recently wrote some utility code which we'll be using to process bundle manifests, typically to map header names to typed values. It was surprising that this code was neither available elsewhere nor completely trivial to implement.

So what was it? A map with String keys which is insensitive to the case of the keys and which preserves the case of the keys. Apache Commons has a CaseInsensitiveMap which whacks the keys to lower case on input. I was tempted to use that code, but in our environment, it's important to preserve the case of keys.

For example, the bundlor tool will eventually use this code to create bundle manifests from templates. If the user carefully spells headers in the template with mixed case, e.g. "Export-Package", they would probably be disappointed to see the headers converted to lower case in the generated manifest.

Implementing an (Apache licensed) case-preserving CaseInsensitiveMap was an interesting exercise. I saved some time by extending AbstractMap and AbstractSet (for the key and entry sets). I used an instance of ConcurrentHashMap to maintain the state. I exposed one internal type to be package visible so I could get 100% unit test coverage and made some inner classes static to stop findbugs complaining.

Why blog about this? Simply to increase your chances of finding the class if you need it.

SpringSource dm Server 2.0 M1

As in Rob's blog, we recently shipped the first milestone of dm Server v2.0. Apart from some new function, this marks an opening up of the dm Server development process with publicly accessible subversion repositories and issue tracker.

The 2.0 roadmap is pretty exciting, and includes constructing a Reference Implementation for RFC 66 ("web container for OSGi") from the existing web components of dm Server.

I'm spending most of my time down in the dm Kernel implementing "cloning" which, roughly speaking, enables a bundle to be wired more than one way, which is not normally possible in OSGi. Cloning copies bundles into an application's scope. This enables the OSGi resolver to have another go at wiring the dependencies of the cloned bundles in a way which suits the application. More information on cloning is included in the above blog entries.

Application scopes can be thought of as an alternative to "nested" OSGi frameworks, which are being specified in the OSGi Alliance and implemented in Equinox. The contents of an application scope can "see" all the contents of the global scope which are not "shadowed" in the application scope. The contents of a nested framework can, by default, not see any of the content of the parent framework(s) unless the nested framework is explicitly defined to see some of the parents' contents.

Much of what we're learning about cloning into application scopes, such as how to calculate what needs cloning and how to manage cloned extenders, will also apply to cloning into nested frameworks if and when that becomes necessary.

Is JDK 7 Java?

I wonder if the JCP rules allow JDK 7 to be called "Java"? Sun appears to think so since the JDK 7 page is entitled "jdk7: Java SE 7". Apache Harmony meanwhile will presumably avoid calling itself "Java" until it can obtain and pass the JCK. Does this make Harmony the true guardian of Java? ;-)

Progress on JSR 294

JSR 294 seems to be underway again. BJ Hargrave describes an EG discussion of visibility and access control which I found enlightening.

Project Jigsaw

Mark Reinhold recently announced Project Jigsaw for modularising the JDK. Could this be a step towards the dream solution? I can hope.

Meanwhile JSR 277 is deferred beyond Java 7. Joy.

Refactoring and Structure101

I'm in the middle of a fairly broad refactoring of the internals of the SpringSource dm Server to introduce an interface to some of the subcomponents and ensure that the rest of the code accesses those subcomponents via the interface rather than directly.

The Structure101 tool has proved invaluable in this process. It has excellent support for analysing the dependencies between components and identifying anomalous dependencies such as those from lower layers of the code to higher layers. But it also supports transformations and visibility settings which turn out to be extremely useful for refactoring.

I created transformations to collect together packages which I want to treat as a unit from the point of view of the current refactoring. This would not be possible using standard refactoring techniques as the packages in question cannot actually be renamed, for reasons I won't go into here. The transformations were of the form:

org.foo.* -> myunit.org.foo.*
org.bar.* -> myunit.org.bar.*

The transformations apply only in Structure101's model of the system and do not affect the actual code. The net is that I can then analyse dependencies on the myunit package hierarchy using Structure101's default mechanisms.

Visibility settings come in useful in identifying the pieces of code I need to refactor. By modifying the Structure101 model so that myunit is a private subcomponent of the new interface component, all dependencies on myunit from outside of the interface component show up as anomalies.

I can then use these anomalies as a 'to do' list for my refactoring. Clicking on one of them shows the precise code dependency which I can refactor by extending the interface component, if necessary, and then changing the using code to depend on the interface component. Refreshing the Structure101 model takes a few seconds after which there is one less anomaly showing on the diagram.

I've really appreciated the support from the Structure101 developers who have helped me get the best out of the tool. It sure beats some combination of grep, a cross-referencing tool like OpenGrok, plus a ton of yellow stickies...

100% bloat free

Here's the image from the T-shirts that were given away at JavaOne. Click the image to enlarge it.

SpringSource Application Platform ships

Today the SpringSource Application Platform shipped its first beta release. Version 1.0 is essentially a combination of Spring, OSGi and Tomcat engineered into a stand-alone runtime.

The programming model is Spring with OSGi and the option of a new packaging format: the platform archive or PAR file. A PAR file comprises multiple OSGi bundles. This brings OSGi modularity and versioning to enterprise applications.

The diagram below gives a high level view of the structure of the platform.

So what does the platform deliver over and above a standard OSGi container? Well, apart from the servlet function, integrated Spring support, the modular kernel-based runtime, and the full support for diagnostics and first failure data capture, there are quite a few features that are of interest to an OSGi audience:

• The platform has a repository which automatically provisions dependencies as bundles are installed into OSGi. This repository stores the platform's own bundles as well as application and third party bundles.
• In addition to the PAR mentioned above, there is a new concept of a library which is a named, versioned collection of OSGi bundles installed in the repository typically for use by applications. This is accessed via a new manifest header, import-library, which imports all the packages exported by the bundles of a library.
• Another new manifest header, import-bundle, imports all the packages exported by a single bundle. This avoids the need to wrap components consisting of a single bundle in a library.
• There is also some novel diagnostic help when a bundle cannot be resolved. The platform's resolution failure detective attempts to extract a concise summary of the problem and the most likely root causes.
• Associated with the platform is a carefully constructed repository of commonly used bundles: the SpringSource Enterprise Bundle Repository known informally as the "bundle repository in the sky", or "BRITS". The bundles have validated OSGi manifests and the whole repository has been checked to make sure that its bundles can be used together.
  

Finally, I can't resist saying a bit about the architecture of the platform. In the past, I've been involved in restructuring projects - adding in modularity after the fact. The platform was built in a modular fashion from day 1.

The whole platform consists of subsystems which are similar in concept to libraries but which are managed specially by the kernel. Each subsystem is a named, versioned collection of bundles. The kernel manages the installation and starting of subsystems as the platform initialises. The kernel manages the lifecycle of each subsystem and its bundles, even reacting to unsolicited lifecycle events of the bundles of a subsystem to keep the state of the subsystem consistent.

Subsystems are used to control the function of the platform. Some subsystems are always present, but others are optional and are configured in a profile. This mechanism allows the precise function of the platform to be configured without rebuilding. For instance, the shipped profile.config file (in the config directory) looks like this:

/*
 * SpringSource Application Platform profile manager default
 * configuration file.
 */
{
 "profile": {
   "version" : 1.0,
   "name" : "web",
   "subsystems" : ["com.springsource.platform.servlet",            
                   "com.springsource.platform.web"]
 }
}

which shows the servlet container and web personality subsystems.

My colleague Rob Harrop just blogged about the platform if you want to know more. Why not register, download the binary, have a play, and tell us what you think.

OSGi could be great for JSR 277

Bryan Atsatt describes a plausible way forward for JSR 277. I agree with his reasoning, although OSGi is not only the de facto standard for Java modularity, it's a proper standard too.

Bryan was one of the key movers in the Expert Group when I was involved. In fact, he proposed a solution to JSR 294 ages ago which is close, if not identical, to the approach Alex Buckley is now considering.

Welcome to the blogosphere, Bryan!

Popular RFE

My favourite sunbug is now on the list of the Top 25 RFEs (Requests for Enhancement).

The 11th most popular sunbug?

Sunbug 6650394 is coming along nicely - both in terms of votes and supportive comments. At the time of writing, it has 99 votes, so thank you if you took the trouble to vote! If you care and you haven't yet voted or commented, please consider doing so.

I was hoping the bug would make it into Sun's Top 25 Bugs list since it has more votes than most of the bugs in that list, but for some reason it's not yet included. If the list was accurate, which it clearly isn't, then 6650394 would be the 11th most popular sunbug.

Sunbug for JSR 277 module system interoperation

Sun has kindly raised sunbug 6650394 on my behalf to track the module system interoperation issue for JSR 277.

Please consider voting for this bug if you would like JSR 277 to interoperate with JSR 291 (OSGi). Add yourself to the watch list if you want to track the bug.

JSR 294 Early Draft

The Early Draft Review of JSR 294 is now complete. Little seems to have changed since when I left the Expert Group. The main weakness from my perspective is that a class file must name its superpackage in order for the class to be a member of the superpackage. This means that you can't easily produce a program which runs on existing versions of Java but which will take advantage of superpackages when it runs on Java 7 and beyond.

Practically speaking, this means that superpackages will only be used in applications that are willing to restrict themselves to Java 7 and beyond. So I suspect that superpackages will not come into popular usage within the next few years.

Of course, this can be gotten round by using a custom class loader with some bytecode rewriting magic, but one of the reasons that JSR 277, and its progeny JSR 294, were introduced was to avoid application developers from having to create custom class loaders.

SpringSource

A couple of weeks in the new job and I work for a different company! Well almost - Interface21 has been renamed to SpringSource to strengthen the association with the Spring portfolio of products.
New web site, new email addresses, new logo, new business cards, etc. On the other hand the job is unchanged, as is the green corporate colour scheme...

Life in a small company

I'm gradually noticing what's different about working in a small company. For one thing, there are very few processes, other than for software development where there are one or two.

For instance, I just took this photo. of the room I work in. Damilola (on the left) and Chris (on the right) noticed the idiot lifting his macbook into mid-air, but Nancie (in the distance) apparently didn't. I wonder how many levels of sign-off would have been needed to publish such a photo. in my previous company?

A more telling example of the minimal processes was that it recently took about an hour to obtain licenses for a piece of commercial software we decided to use. This included getting financial sign-off from someone on a business trip on the West Coast, submitting the order, and getting the license keys ready to use.

This is a pleasant change for me. Processes are sometimes essential, but it's so easy to make them too heavy. The emphasis in a small company seems to be talking to other people and trusting them to do the right thing.