Mind the Gap

CME Group and Virgo

2012-01-19T16:57:00.000Z

Today CME Group released an official statement about their use of Virgo. This is posted under the "Virgo Powered" section of the Virgo home page, but the full text is also reproduced below.

I'm delighted both that CME Group are using Virgo and that such a large player in the finance area was happy to go public.

CME Group, formerly known as Chicago Mercantile Exchange Holdings Inc., was founded in 1898 and operates the CME, CBOT, NYMEX, and COMEX regulatory exchanges worldwide. It is the world’s leading derivatives marketplace with 2570 full time employees and a market capitalisation of over $15B.

CME Group kindly released the following information in January 2012.

At CME Group, the goals for our project were to use and create a platform that would be easy to maintain, expand and reuse. Our applications are of various sizes and complexities, but there are certain aspects that are common, which was the reason to find a very modular solution. In particular, our current efforts are in Surveillance tools which allow monitoring the state of the exchange and the market.

We started using Virgo 2.0.1 during a “proof of concept” in November of 2010 and spent 2011 working closely with them to build the application. Today we use version 3.0.1. with 48 service bundles and 20 web bundles. These services range from simple DAO type services, through authorization and authentication, web services, and even distributed caching. All web bundles use snaps and customized apache tiles views.

What Virgo and OSGi provides for us is complete modularity and plug-and-play capability. This enables us to work more intelligently with other teams who need to build bundles independently of us where the strong isolation of services and APIs allows for a clear separation of control. We have created a core set of services that will be a platform for future projects across CME Group and are already actively building two projects on top of that platform. Having Virgo bundled with Apache’s Tomcat also helped in deciding which container to use as our projects are mostly web applications.

Virgo Nano Technology

2012-01-05T16:49:00.001Z

The first milestone of Virgo 3.5.0 is available. It introduces two significant new features: p2 support (covered previously) and a new Virgo distribution known as nano. (Apologies for the pun in the blog title - it was too hard to resist.)

Nano is essentially a cut-down version of Virgo which starts up really fast and has a single (kernel) region. It is intended for simpler scenarios where you don't need to isolate applications from each other or from the kernel.

The current nano distribution includes Gemini Web and so is a fully-functional OSGi web server which starts in about 3 seconds (compared to Virgo Tomcat/Jetty Server which take about 9-10 seconds). However, we plan to factor out Gemini Web so that nano becomes a small/fast subset of the Virgo kernel. We will then provide a separate distribution which adds Gemini Web to nano, as in the first milestone.

As well as introducing nano, the milestone re-bases the kernel and the Tomcat/Jetty servers on nano. Initial provisioning via p2, with full instructions in the user guide, is provided for all distributions in addition to the familiar ZIP file installs (the ZIP contents are now constructed using p2 at build time). All distributions use a regular Equinox directory layout and launcher which will simplify third party tooling integration (e.g. Pax Exam). A short-term downside is that we need to update the Virgo IDE tooling to work with Virgo 3.5.0.

The current nano distribution can also dynamically provision content via p2 which is particularly useful for automated or cloud deployments. (We tried to implement this for the multi-region kernel and Tomcat/Jetty servers, but that was beyond the scope of the current p2 design, so we've put it on hold.)

There is also a nice engineering side-effect of the introduction of nano: we are starting to refactor the kernel — by far the most complex Virgo component — and move the more general, region-agnostic features down into nano.

The following picture summarises the distributions we anticipate in Virgo 3.5.0:

As well as being the fruit of several months of effort, this first milestone kicks off Virgo development in 2012 in a very exciting way.

OSGi testing with Gradle and Pax Exam

2011-12-08T16:13:00.001Z

I've recently been working on a Spring issue which involved writing a Gradle-built test to ensure that the Spring framework 3.1.x bundles successfully resolve in an OSGi container.

The Gradle side of this was a very simple and pleasant experience, as the following extract of the file build.gradle shows:

apply plugin: 'java'
apply plugin: 'eclipse'

repositories {
    mavenCentral()
}

dependencies {
    ...                
}

There were minor usability issues¹ with the Gradle Eclipse plugin, but I gather these will soon be resolved.

I decided to use Pax Exam to launch the testcase inside Equinox, and this require specifying some dependencies in build.gradle:

dependencies {
    paxExamVersion = '2.3.0.M1'
    equinoxVersion = '3.6.0.v20100517'
    
    testCompile "junit:junit:4.+",
                "org.ops4j.pax.exam:pax-exam-junit4:$paxExamVersion",
                "org.eclipse.osgi:org.eclipse.osgi:$equinoxVersion",
                "javax.inject:javax.inject:1"
                
    testRuntime "org.ops4j.pax.exam:pax-exam-container-native:$paxExamVersion",
                "org.ops4j.pax.exam:pax-exam-link-mvn:$paxExamVersion",
                "org.ops4j.pax.url:pax-url-aether:1.3.5"
}

Let's go through the dependencies in turn:

JUnit 4 is the test framework.
@RunWith(JUnit4TestRunner.class) marks the test for launching under Pax Exam.
Equinox provides OSGi types at compile time and an OSGi container at runtime.
A bundle context is injected into the test using @javax.inject.Inject.
The test uses the native Pax Exam container - all the tests run in a single JVM.
The test uses Pax Exam to install Spring bundles directly from Maven repositories.²
The test caches bundles from the SpringSource Enterprise Bundle Repository using the Pax URL Mvn protocol.

The Pax Exam developers kindly helped me sort a basic setup issue which wasn't too obvious from the documentation. Thanks Harald and Toni! I needed to explicitly install a JUnit bundle, by calling junitBundles from the configuration method:

    @Configuration
    public static Option[] configuration() throws Exception {
        return options(//
            provisionSpringBundle("spring-core"), //
            provisionSpringBundle("spring-beans"), //
            // etc.
            junitBundles());
    }

The full source of the project is available on github (under the Apache 2.0 license) for anyone who wants to give it a spin or use it for their own purposes. If you'd prefer to write your tests in Groovy, take a look at Hamlet D'Arcy's blog.

Footnote:
1: The generated .classpath contains absolute paths rather than paths relative to the project and so is not suitable for committing and sharing with others. Also re-running gradle eclipse without first deleting .classpath can result in it containing duplicate paths. Luke Daley tells me that both these problems will soon be fixed, the latter fix appearing in gradle 1.0 milestone 7.
2: If the testcase was intended a a stand-alone test of published versions of Spring, it would cache Spring bundles in the local Maven repository for efficiency. But we plan to integrate the testcase into the Spring build in due course and so we'll want to grab the latest built version each time to cope with re-spins, so cacheing isn't helpful.

Virgo 3.0.2 released

2011-11-29T15:32:00.001Z

Virgo 3.0.2 is available for download. Please see the release notes for details of this bug fix release.

Opening SourceTree from the command line

2011-11-18T15:33:00.001Z

I've recently started using SourceTree to visualise and manage my git repos. It's very impressive.

The one thing I missed from gitx was the ability to launch SourceTree from the command line, in any directory of the repo. One solution is to set a git alias:

git config --global --add alias.sourcetree '!open -a SourceTree .'

and add an alias in the shell profile:

alias st='git sourcetree'

Now I can issue "st" from the command line and the relevant SourceTree window opens up.

(Thanks to Chris Frost for providing part of this solution.)

What did we learn in 100 sprints?

2011-11-14T11:11:00.001Z

This week marks the 100th Sprint on Eclipse Virgo! That includes the precursor project, SpringSource dm Server. What have we learned about scrum? Well, obviously, it's pretty good otherwise we would have ditched it long ago (and believe me I've been tempted a couple of times). But specifically...

Firstly, that one week sprints are by far the easiest to manage, both in terms of planning effort, but also psychologically. Even in a two week sprint, it's easy to put off unpleasant tasks until it's too late.

Secondly, that the main value of scrum is to keep the team focussed on specific tasks during a sprint and enable progress to be shared easily at daily stand-up meetings.

Thirdly, that story points are more effort than they are worth, so it's more effective to go with gut feel.

Fourthly, that you have to take a relaxed attitude towards planning design work as it's almost impossible to predict the effort up front. Spikes are ok for small items, but sometimes you just need plenty of time to get your head round a large area.

Enterprise Bundle Repository Samples

2011-10-27T10:23:00.002+01:00

After opening up the Enterprise Bundle Repository build system, I've added more sample bundlor templates along with Ant/Ivy build files to github.

So while we are trying to come up with a community-based successor to the EBR, if you need to get a new version of one of the bundles in the EBR, you may be able to find a bundlor template to use as a starting point.

Latest draft subsystems spec

2011-10-12T11:18:00.001+01:00

The OSGi Alliance has made an early draft of some RFCs available. This includes the subsystems RFC 152 I blogged about earlier: see pages 72-144 of the PDF.

Eclipse Dance Steps

2011-10-05T16:00:00.001+01:00

Sometime Eclipse fails to work as expected. The following 'dances', in increasing order of desperation, may help. Remember to maintain a fixed grin throughout and try not to tread on your own toes.

Refresh and clean all projects ("Quickstep")

Click on a project in the Package Explorer. Hit Apple-A (or Ctrl-A) to select all projects. Hit F5 to refresh. Select Project->Clean..., select "Clean all projects", and click Ok.

Open and close all projects ("Foxtrot")

Click on a project in the Package Explorer. Hit Apple-A (or Ctrl-A) to select all projects. Right click the projects and select "Close Project". Right click the projects again and select "Open Project".

Open and close all projects restarting Eclipse in between ("Tango")

Click on a project in the Package Explorer. Hit Apple-A (or Ctrl-A) to select all projects. Right click the projects and select "Close Project". Restart Eclipse. Click on a project in the Package Explorer. Hit Apple-A (or Ctrl-A) to select all projects. Right click the projects again and select "Open Project".

Delete and re-import all projects ("Paso Doble")

Click on a project in the Package Explorer. Hit Apple-A (or Ctrl-A) to select all projects. Press the delete key, select "Do not delete contents", and click Ok. Right click in the Package Explorer and select Import..., then choose General and "Existing Projects into Workspace" and click Next. Browse to select the platform checkout directory and click Choose. Select all the projects for import. Click Finish.

Rebuild the search indices ("Slow Waltz")

Exit the workbench and delete the Java search index:

go into \plugins\.metadata\.plugins\org.eclipse.jdt.core
delete 'savedIndexNames.txt'
delete all *.index
Restart

If all else fails

If you're still stuck, try the "Hokey Cokey" by repeating all of the above until you've had enough.

How Infor ION Uses Virgo

2011-10-03T11:10:00.001+01:00

Infor have kindly made available a presentation on how their ION integration product is based on the Virgo kernel. They migrated from Equinox 3.5 to Virgo 2.1, added a lightweight web server (in preference to using the pre-built Virgo Tomcat or Jetty servers) and a service wrapper, added Spring integration, upgraded Spring framework, added a watched repository, and then added various enterprise service bundles. This is a perfect example of a Virgo kernel application, a so-called OSGi stackless stack.

The Infor team also helpfully pointed out some enhancements they'd like to see, some of which are already supported or planned for Virgo 3.5.

According to their website Infor is the world's third largest provider of enterprise applications and services, so it's nice to see another large company adopt Virgo.

Thanks to Ian Skerrett for bringing this user to my attention. See his blog for more.

Extenders: Pattern or Anti-Pattern?

2011-09-30T09:40:00.000+01:00

The extender pattern has become popular in recent years and has even been utilised in OSGi standards such as the Blueprint service and the Web Applications specification. In Virgo, we've been working with extenders from the start, but in spite of their advantages, they have some significant downsides. Since the OSGi Alliance is considering using extenders in other specifications, I agreed to document some of the issues.

The first difficulty is knowing when an extender has finished processing a bundle. For example, a bundle containing a blueprint XML file will transition to ACTIVE state as soon as any bundle activator has been driven. But that's not the whole story. Administrators are interested in when the bundle is ready for use and so the management code in Virgo tracks the progress of the extender and presents an amalgamated state for the install artefact representing the bundle. The install artefact stays in STARTING state until the application context has been published, at which point it transitions to ACTIVE. Without such additional infrastructure, administrator cannot tell when a bundle processed by an extender really is ready for business.

That's the successful case, but there are complications in error cases too. The first complication is that since an extender runs in a separate thread to that which installed the bundle, if the extender throws an exception, this is not propagated to the code which installed the bundle. So the installer needs somehow to check for errors. Therefore Virgo has infrastructure to detect such errors and propagate them back to the thread which initiated deployment of the bundle: the deployment operation fails with a stack trace indicating what went wrong.

The other error complication is where there is a (possibly indefinite) delay in an extender processing a bundle. For this kind of error Virgo tracks the progress of extender processing and issues warnings to the event log (intended for the administrator's eyes) saying which pieces of processing have been delayed and in some common situations, for example when a blueprint is waiting for a dependency, what is causing the delay.

Extenders suffer from needing to be able to see bundle lifecycle events and so for systems that partition the framework, it is necessary to install each extender into multiple partitions. On the flip side it is crucial to prevent multiple instances of an extender from ever seeing the same bundle event otherwise they will both attempt to extend the bundle.

Another issue with extenders is the need to keep them running and healthy as there is little indication that an extender is down or sickly other than bundles not being processed by the extender. Virgo takes care to ensure its extenders are correctly started and its infrastructure for detecting delays helps to diagnose extender crashes or sickness (both of which are extremely rare situations).

There is also an issue in passing parameters to an extender to affect its behaviour. This is typically done by embedding extender configuration in the bundles being processed or by attaching a fragment containing configuration to the extender bundle. But since the extender is not driven by an API, the normal approach of passing parameters on a call is not available. Essentially, an extender model implies that the programming model for deployment is restricted to BundleContext.installBundle.

With considerable investment in additional infrastructure, Virgo has managed to support the Blueprint and Spring DM extenders reasonably well. But in the case of the Web Applications extender, Virgo couldn't make this sufficiently robust and so it drives the underlying web componentry directly from the Virgo deployment pipeline to avoid the above issues.

I understand at least one other server runtime project has encountered similar issues with extenders, so Virgo is not alone. There is a trade-off between loosely coupling the installer from the resource-specific processing, the main strength of the extender pattern (but far from unique to that pattern), and providing a robust programming model and usable management view -- crucial features of a server runtime -- which is far more straightforward without extenders.

OSGi Subsystems in Virgo

2011-09-22T14:30:00.000+01:00

A public draft(*) of the OSGi subsystems RFC (152) should soon emerge from the OSGi Alliance. A subsystem is a multi-bundle application, not dissimilar to a PAR or plan in Virgo. IBM is leading the spec work and a number of other vendors, including SpringSource/VMware, are contributing. Quite a few projects have multi-bundle application constructs, so it makes sense to agree a standard form.

After going through the RFC again with my implementer's hat on, I listed the features necessary to support subsystems in Virgo. Most of the changes are in the Virgo kernel, although I hope to structure the support into non-subsystem specific generalisations of the kernel plus subsystem-specific code running in the user region.

Currently, it is not possible to deploy a plan which contain artefacts that are already deployed. This will need generalising so that we can support subsystems using a common data structure of deployed artefacts: a directed acyclic graph (DAG) rather than the current collection of trees.

The switch to a DAG has interesting implications for lifecycle management of shared subgraphs. With today's tree, when a node is stopped, started, or uninstalled, any subtrees are also stopped, started, or uninstalled, respectively. With a DAG, shared subgraphs need to be sensitive to all their parents. This boils down to keeping each shared subgraph at the maximum state required by any of its parents. States are ordered: ACTIVE > RESOLVED > UNINSTALLED.

Lifecycle management will also get interesting when a shared subgraph belongs to one or more atomic subgraphs as then lifecycle changes in the common subgraph will propagate to all the containing atomic subgraphs. I think that will just "work", but users might need to be careful in their use of atomic plans if they want to avoid management operations on one application affecting other applications.

I'm also considering using garbage collection as a means of uninstalling artefacts which are no longer needed. Given the number of types of dependency that are possible, this is likely to be more reliable than the alternative of maintaining reference counts.

I'm a little concerned about a possible race between garbage collection detecting an artefact as dead and a new dependency being created on the artefact just before garbage collection goes ahead and uninstalls the artefact, but there would be a similar concern for reference counting. The basic issue in this race is that a dead artefact may be found by a live bundle and a new dependency created before the dead artefact can be uninstalled. For instance, a dead bundle may be found by using the OSGi API to list all bundles. It may be possible to use some technique such as a special region in the region digraph to isolate dead bundles, although this issue is probably something to discuss among those working on the RFC.

Anyway, there's plenty of work to be getting on with. I haven't done detailed estimates of the features identified so far, but I guess there's a person year or so of effort needed, so I'm initially targeting Virgo 4.0. If you feel like lending a hand, please get in touch on virgo-dev.

* - RFC 152 has changed quite a bit since the version in the Enterprise spec early access draft dated 16 May 2011. A new draft is being prepared.

New Releases: Virgo 3.0 and Gemini Web 2.0

2011-08-25T10:46:00.000+01:00

Virgo 3.0 and Gemini Web 2.0, collectively known as the Maya release, are available for download and download, respectively.

The team of committers has expanded considerably and, although the team is now distributed geographically, we have a great working relationship. I'm delighted that these projects are now community collaborations rather than the work of a single vendor.

The theme of Virgo 3.0 is better integration with EclipseRT technologies. To that end, we have created a Jetty variant of the Virgo web server and have switched from Felix to Equinox implementations of some OSGi services. Integration with p2 is proving a tough nut to crack and is deferred to the Bondi (3.5) release, although we expect to issue a milestone soon to get some feedback from the user community.

Gemini Web and the Tomcat variant of the Virgo web server have been upgraded to Tomcat 7 and Servlet 3.0.

Gemini Web 2.0 passes the OSGi Web Applications compliance tests and will replace the SpringSource OSGi Web Container as the OSGi Web Applications reference implementation.

The snaps framework for modular web applications is also released as part of Maya. This started life as the SpringSource slices prototype.

Virgo kernel 3.0 has a new shell based on Apache Felix Gogo for both the kernel and user region and accessible via telnet or ssh.

The 3.0 kernel also uses a new implementation of regions, the region digraph, based on OSGi standard framework hooks. The digraph should position Virgo well for implementing the OSGi Subsystems specification in a future release. (The specification is still under development, but we expect a public draft from the OSGi Alliance before long.) The digraph has moved from Virgo to Equinox to enable it to be used by other projects, especially those that are aiming to implement the Subsystems spec.

Highlights:

New Virgo Jetty Server
Virgo Tomcat Server and Gemini Web upgraded to Tomcat 7 and Servlet 3.0
New Snaps framework for modular web applications with new guide
New Gogo shell
New user guide for Gemini Web
Updated GreenPages sample with new guide

See the release notes for details.

OSGi Running in the Cloud: Virgo in CloudFoundry

2011-08-03T15:44:00.000+01:00

I'm just been tryout out Hristo Iliev's support for Virgo in CloudFoundry, described on his blog. The VM took a few hours to download and a few minutes to unzip, but then starting CloudFoundry and deploying an application to a Virgo instance worked first time.

CloudFoundry takes a little while to get started:

But then it's trivial to deploy an application to a Virgo instance:

after which the application, in this case the Virgo web admin console, runs as expected at the application URL http://virgo.vcap.me:

and jconsole shows where the Virgo instance has been created:

The cloud nature of this setup becomes more apparent when another application is deployed. Since I'm lazy, I deployed the same application under a different name:

The application is then available at http://virgo2.vcap.me and jconsole shows there are now two Virgo instances running:

I'm looking forward to Hristo's pull requests become merged into the CloudFoundry base and maybe eventually Virgo showing up in some clouds near you.

Virgo 3.0 (Maya) Release Candidate 1

2011-07-26T10:09:00.001+01:00

The first Release Candidate of Virgo 3.0 (Maya) is available for download. See the release notes for details.

RC1 finally brings the snaps framework for modular web applications out of prototype status. The separate snaps download includes a completely new Virgo Snaps User Guide and a comprehensive sample application. The User Guide describes how to install snaps on top of Virgo Tomcat Server and uses the sample application to verify the installation and step through all the major features of snaps.

RC1 of Virgo Tomcat Server embeds Gemini Web 2.0 RC1 which now comes with new documentation in the form of a User Guide plus a minimal Programmer Guide.

We encourage users to take RC1 for a spin and provide feedback via bugzilla or the Virgo forum. Unless critical bugs are found, we'll be focussing on the documentation until 3.0 is released, probably towards the end of August.

Exploiting p2 in EclipseRT projects

2011-07-14T15:13:00.000+01:00

Virgo committer Borislav Kapukaranov has written an interesting blog entitled "RT meets p2". Here he gives some of the background behind the work to integrate p2 into Virgo (bug 343543) which is targetting the Azure (3.5) release of Virgo.

Virgo Maya (3.0) feature freeze at milestone 6

2011-07-05T16:01:00.000+01:00

Milestone 6 of the Virgo Maya (3.0) release is available for download. This milestone marks the feature freeze of Maya, so we'll be working on bugs and documentation from now on. We plan to ship the release in August.

The milestone includes the new shell based on Apache Felix GoGo, accessible via telnet or ssh as described in the release notes. There are a few bugs in this support which need fixing, but apart from that the new function is in good shape for trying out.

This milestone also includes snaps for the first time. Snaps is a framework supporting modular OSGi web applications. We are bringing snaps out of prototype status in Maya, primarily by documenting its function and limitations in the next few weeks. Several groups are using snaps, so this should give them a more stable base. Including snaps in the release process will ensure it stays in step with Virgo Tomcat Server as the two evolve.

Finally, this milestone moves Virgo Jetty Server out of prototype status as the User and Programmer Guides now include appropriate documentation.

First milestone of Virgo tooling

2011-06-27T15:35:00.001+01:00

The first milestone of the Virgo IDE tooling is available. See the wiki for instructions. It should provide approximately the same level of function as the Virgo tooling that was provided as part of SpringSource Tool Suite (STS). It works nicely on Eclipse Indigo and Helios SR2, for example.

The basic functions allow you to develop OSGi bundles, including Web Application Bundles, and PARs and deploy them to an instance of Virgo Tomcat Server (or Virgo Web Server if you prefer to run v2.1 rather than a 3.0 milestone) running under the control of the tooling. It is also possible to develop plans, but this needs documenting better as it's not immediately obvious how to do it. See an earlier blog post if you need to run the Virgo kernel instead of Virgo Tomcat Server.

Please direct discussion to the Virgo forum and raise bugs on bugzilla (thanks Daniël van 't Ooster for the bugs you raised while I was writing this blog!). The source is available in Eclipse git for anyone who wants to make changes and/or send in patches. We are keen to encourage new contributors to Virgo in general and the Virgo tooling in particular.

What next? Well, the most urgent feature, bug 341910 which is a pre-req to the first release, is to remove the dependency on Spring IDE.

Then, over time, we expect to align the Virgo tooling with the Eclipse Libra project. Libra provides tooling for OSGi standards, so we aim to move the OSGi standard features of the Virgo tooling to Libra and end up with the Virgo tooling being a Virgo-specific extension of Libra.

Congratulations to Martin Lippert who started with the STS code and single-handedly achieved this milestone! You can see him in last week's OSGi Development Tooling Panel.

Jigsaw déjà vu

2011-06-27T14:47:00.000+01:00

I recently questioned one of the Jigsaw requirements on the jigsaw-dev mailing list. I agree with the general direction for interoperability with OSGi, but I wanted to suggest an improvement:

Hi Mark

I'm pleased to see the explicit acknowledgement of some basic OSGi interoperation requirements in the requirements document ([1]).

I agree with David Bosschaert ([2]), that it would make sense for OSGi to support the Java SE 8 module format and, for modules which can serve equally well as OSGi bundles, I'd like to avoid dual-maintenance of module metadata and OSGi manifest. I'd like to be able to "decorate" the standard metadata.

However, the requirement "The syntax must place all extended metadata after all standard metadata, with a clear delineation between them." precludes inline decorations. The result would be duplication and clunkiness.

I propose that this requirement be changed so that standard metadata could be decorated inline (the decorations would be ignored by the Java SE module system).

What do you think?

Regards,
Glyn
[1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12
[2] http://osgithoughts.blogspot.com/2011/05/java-se-8-modularity-requirements.html

The response: zilch!

After a follow-up, I got a single supportive response, but that was from OSGi aficionado Peter Kriens.

I would have preferred some defence of the current requirement rather than silence. This is awfully like the days of JSRs 277 and 294. Plus ça change, plus c'est la même chose...

Virgo 3.0 plan update

2011-06-14T14:43:00.000+01:00

The feature to provision Virgo using p2 is taking a while to finalise. Also, it changes the directory layout of Virgo and so will need some changes to the tooling. So we've decided to defer the feature to a later release, possibly 3.5 to indicate the relatively big jump from 3.0.

This means the 3.0 release content is pretty much complete, so we just have to fix a few remaining bugs and polish the documentation before releasing, probably in July or August 2011.

See the plan on the Virgo wiki for details.

Eclipse Virgo 3.0.0 Milestone 5

2011-06-01T16:59:00.001+01:00

Milestone 5 of Virgo 3.0.0 is available for download. This is an important milestone which adds significant functional enhancements, upgrades several dependencies to their latest levels including Spring 3.0.5, Tomcat 7.0.12, and Servlet 3.0, and fixes a number of bugs. Full details are available in the release notes.

Equinox digraph ready for use

2011-05-09T16:12:00.000+01:00

Previously I described how Virgo used OSGi framework hooks to implement a directed graph, or digraph, of isolated groups of bundles known as regions. The region digraph has now moved to Equinox and I'm modifying Virgo to use the Equinox region digraph in place of Virgo's own (see bug 343364 for details).

I always prefer making such changes in an incremental way, so the first step was to add the Equinox region digraph bundle to the Virgo kernel without exploiting it. The Virgo kernel runs fine with the Equinox region digraph in situ. This is because the Equinox digraph adds all bundles to its kernel region and so provides no isolation between those bundles. Later, Virgo's region digraph partitions the bundles in Virgo kernel and Virgo user regions with appropriate isolation and things continue to work as normal.

This is interesting, but not particularly useful. The whole point of the region digraph abstraction was to make it easy for multiple parties wishing to provide isolation via the OSGi framework hooks to be able to collaborate together by sharing and manipulating a single region digraph. It was never intended that multiple digraphs should co-exist.

But there is a useful point here. If you have an Equinox-based system, of which there are many, you can introduce the Equinox digraph with zero functional impact. All your bundles, including those installed before and after the Equinox region bundle, go in the Equinox kernel region and there is no isolation between those bundles.

When you want to start exploiting the digraph, you can use the RegionDigraph service to create a new region, attach it with appropriate filters, if any, to and from the kernel region, and then install a bundle into the new region using the Region.installBundle operation. After starting this bundle, you will then have a bundle context that can be used to install further bundles into the new region using the OSGi standard BundleContext API.

I hope this will inspire you to have a go at using the Equinox region digraph if you are an Equinox user and need to isolate groups of bundles.

Meanwhile, the standardisation of multi-bundle applications, including isolation, continues in the OSGi Alliance and a public draft of this spec should be available in the not too distant future. I hope the Equinox region digraph will be used more widely and this will help to provide feedback on the spec work.

Opening up the Enterprise Bundle Repository

2011-05-09T12:13:00.000+01:00

We'd like to enable the SpringSource Enterprise Bundle Repository to turn into a shared community resource.

As a first, small step in that direction, I've made available the EBR build system (a few Ant/Ivy scripts) and some sample bundle projects on github. If you want to try it out, clone the repository and refer to doc/README.txt.

The idea is that the community will be able to add bundles to this repository and then the content can be hosted in some cloud storage similar to the way the SpringSource EBR is hosted in Amazon S3.

Virgo survey results

2011-05-03T11:39:00.004+01:00

Thanks to everyone who responded to the recent Eclipse Virgo and SpringSource dm Server survey. The results make interesting reading.

Close to one quarter of respondents report they are "in production".

The transfer to Eclipse.org is a success. It is 6 months since the first Virgo release and 84% of respondents are now using Virgo exclusively with only 13% using dm Server exclusively.

Virgo is popular with early adopters. Of the 84% using Virgo exclusively, 59% are using 2.1.x or 3.0 milestones and 25% are on 3.0 milestones only (some of these already in production!). Please note that Virgo has versions 2.1.0 and greater whereas dm Server covers versions 1.x and 2.0.x.

The Virgo kernel is seeing significant usage. The kernel underpins the Virgo Web Server and provides Virgo's advanced deployment, provisioning, isolation, scoping, and diagnostic capabilities. The kernel has no web-specific code and so can be used to build messaging and other non-HTTP servers and to act as a container for non-server applications. 16% of respondents are using the kernel exclusively; 5% are in production.

Documentation and blogs were the most useful information sources closely followed by the Virgo forum, Virgo wiki, and Virgo development mailing list (1= not at all useful, 5 = very useful).

The survey was advertised on Twitter, the Virgo forum, the Virgo development mailing list, at www.springsource.org, and in this blog. 166 people responded. If you are interested, the raw results of the survey are available online or a summary can be downloaded.

Finally, I'd like to thank Chris Frost for coming up with the idea for the survey and then delivering the goods.

OSGi comes of age?

2011-04-05T20:38:00.000+01:00

Early in my career at SpringSource, my colleagues presented me a T-shirt with "CICS does that!" printed on the front. However, nobody anticipated what that statement might eventually include.

IBM has just announced that the latest version of the CICS transaction monitor includes support for applications packaged as OSGi bundles. From the announcement letter:

The JVM server infrastructure with integrated OSGi support and tooling provides a new industry standard Java runtime environment in CICS.

[...]

The JVM server environment supports the development and deployment of Java applications through the use of industry standard OSGi bundles. This will require Java applications to be packaged into an OSGi bundle, and then deployed as a CICS BUNDLE resource with reference to the OSGi bundle and its version, removing the need to load Java applications from a statically defined class path. This function will significantly improve the system management of Java applications in CICS through the provision of application isolation and versioning, cross-package prerequisite checking, and simplified package redeployment.

This is heart-warming news to me, both as an OSGi proponent and as a former member of the CICS development team. I'm guessing CICS is using Equinox as its OSGi framework, but I'm looking forward to reading the documentation for the details.

Java update breaks jps, jconsole, etc

2011-03-31T16:37:00.002+01:00

The latest update of Java broke jps and jconsole when Java is launched specifying "-Djava.io.tmpdir" with a value of anything other than $TMPDIR. This affects Virgo and Tomcat.

The problem was introduced in Java 6 update 21 and is fixed by bug sunbug 7009828 in Java 6 update 25.

A workaround for Virgo is as follows. On Mac and Linux to delete the following line in dmk.sh:

-Djava.io.tmpdir=$TMP_DIR \

and on Windows to delete the following line in dmk.bat:

set KERNEL_JAVA_PARMS=%KERNEL_JAVA_PARMS% -Djava.io.tmpdir="%TMP_DIR%"

Eclipse Virgo diagnostics for "uses" conflicts

2011-03-31T11:54:00.002+01:00

Congratulations to Felix on producing some really nice diagnostics for uses conflicts, and thanks to Neil Bartlett for his blog on the subject.

I thought I'd try a similar example in Virgo for comparison. Here are Virgo's diagnostics:

Unable to satisfy dependencies of bundle 'org.example.foo.xydata' at version '0.0.0': Cannot resolve: org.example.foo.xydata
    Resolver report:
        Uses violation: ⟨Import-Package: org.example.foo.bar.datastore; version="0.0.0"⟩ in bundle ⟨org.example.foo.xydata_0.0.0[1301563814803]⟩
            Found conflicts:
                package        'org.example.foo.util.verify_1.0.0' in bundle 'org.example.foo.platform.util_0.0.0[1301563814806]' used by 'org.example.foo.bar.datastore_0.0.0' in bundle 'org.example.foo.bar.datastore_0.0.0[1301563814805]'
                conflicts with 'org.example.foo.util.verify_2.0.0' in bundle 'org.example.foo.framework_0.0.0[1301563814804]' imported by bundle 'org.example.foo.xydata_0.0.0[1301563814803]'

(The numbers in square brackets are bundle ids - they are enormous because Virgo is attempting resolution in a "side state". Bundles with enormous ids have either been deployed directly or auto-installed from the Virgo repository and are in the process of being resolved. They are committed to the OSGi framework only if the directly deployed bundle(s) can be resolved successfully.)

The Virgo formatting isn't quite as pretty as Felix's, but it looks reasonable on a wide terminal window. However, there are some subtle differences:

Felix dumps out the dependency chains whereas Virgo does not attempt that. We had in mind examples where the dependency chains are unmanageably long, i.e. those where good diagnostics really are essential. But having seen Felix's diagnostics, we would like to dump out the chains in simple cases. On the other hand, that area of code is pretty complex, so this is not something we'll be rushing to do.
Felix depicts wirings from import to export which could be especially useful for newcomers to OSGi.
The Felix diagnostics are built in to the framework whereas non-Virgo users of Equinox do not get the Virgo diagnostics.
The Virgo diagnostics highlight the fact that there is a uses violation whereas Felix leaves the user to infer that from the rest of the message. It would be nice if Felix was more explicit because there are other reasons a bundle cannot be resolved.
Felix highlights the duplicated package whereas Virgo highlights the import that encountered the uses constraint. Virgo is designed to cope well when there are multiple duplicated packages. It's not clear what the Felix diagnostics would look like in such cases.

Note: on a resolution failure Virgo dumps the "side state" to disk for offline analysis. With this example, the offline analysis produced poorer diagnostics than were printed to the console, so I raised bug 341460 to investigate. The beauty of offline analysis is it lets you explore the details of the bundles. For example, the following screenshots shows the details of the duplicated package, including the import version ranges which result in the attempt to use the distinct versions of the duplicated package:

Anyway, I'm really pleased that we are starting to see some stiff competition on diagnostics. This will help OSGi users enormously.

Postscript
Bug 341460 is fixed in the Virgo 3.0 line. The admin console now displays the same diagnostics that were printed to the console.

Enterprise OSGi Runtimes and Tooling

2011-03-30T12:10:00.000+01:00

It was great to see several enterprise OSGi runtimes in action at EclipseCon.

Readers of this blog will be familiar with Eclipse Virgo, so suffice to say that tutorials covered developing applications using Eclipse Gemini components (reference implementations of several OSGi Enterprise standards) in Virgo and configuring and managing web applications in Virgo.

Several of the tutorials developed Web Application Bundles, or WABs, which are part of the OSGi Enterprise standard. A Web Application Bundle is like a WAR file which specifies the web context path in its manifest and which obtains its dependencies using OSGi instead of having to include them all in WEB-INF/lib. The tutorials also implemented persistence using the OSGi Enterprise standard mapping of Java Persistence API (JPA).

Apache Aries provides a collection of components including implementations of several Enterprise OSGi standards and an application model similar to Virgo's. The EclipseCon tutorial had us creating a blogging application using the free IBM® Rational® tooling. The application consisted of an Aries application containing a plain OSGi bundle, a bundle utilising JPA persistence, and a WAB. The JPA bundle used Aries extensions to the standard blueprint service. The WAB used JNDI to look up the service published by the persistence bundle - the same approach that was taken in the Gemini and Virgo tutorial. We then deployed the application to a target platform which included Aries bundles.

Apache Karaf is similar in concept to the Virgo kernel. Karaf has several functions that Virgo lacks, some of which are good candidates to add to Virgo. Apache Geronimo is based on Karaf.

GlassFish now has good support for OSGi. The EclipseCon tutorial started with us developing a basic client bundle calling a service published by a back-end bundle. We then replaced the client bundle with a WAB and the back-end bundle by a JPA persistence bundle which published a service in the form of an EJB. The WAB used custom annotations to inject the service into the WAB's servlet. The EJB was used to demarcate transactions and security. The M2Eclipse "core" and "extras" plugins provided a basic Eclipse development environment, although the bundles were deployed via the command line. Kudos to the GlassFish team, including Arun Gupta and Sanjeeb Sahoo who led the tutorial, for possibly the most polished hands-on tutorial I've experienced.

I was also very pleased to see the Eclipse Libra project making progress. The version destined for inclusion in this year's Eclipse release train can be used to develop WABs. Runtime launching support is in the pipeline. We also hope to migrate standards based features from the Virgo tooling into Libra.

git tip: listing or editing the commits on a branch

2011-03-30T10:20:00.001+01:00

Sometimes I need to list all the commits on a branch. I found the following command does the trick, even after I have merged from master into the branch one or more times:

git rev-list my-branch ^master

This lists all the commits reachable from my-branch but not reachable from master.

You can also change the committer or author info of commits on a branch by specifying my-branch ^master as the on git filter-branch (see git help filter-branch).

Quick survey of Virgo and SpringSource dm Server usage

2011-03-29T17:26:00.000+01:00

I would be grateful if users of Virgo and SpringSource dm Server would take a short, anonymous survey to help us in our planning. Should take no more than 2 minutes to complete.

Correction: Oracle do not contribute to Virgo

2011-03-29T10:18:00.000+01:00

A recent Oracle blog stated:

Also presented were Eclipse projects, that Oracle engineers are contributing to: Gemini, Virgo, Sapphire and Web Tool Platform.

I submitted a comment on the blog pointing out that Oracle engineers contribute to Gemini but not Virgo. Unfortunately, the comment was not published and the blog was not corrected so perhaps this blog will help to set the record straight.

That said, Virgo would welcome contributions from Oracle (and others). We've enjoyed collaborating with Oracle in the Gemini project and we really appreciate having Virgo committers from three different companies and a few more would be even better.

Incremental OSGi with Virgo, Gemini, and Libra

2011-03-21T12:07:00.001Z

As I anticipate the start of EclipseCon 2011 later today, I'm reflecting on the current state of the Eclipse Virgo, Gemini, and Libra projects.

As I'll be discussing on Tuesday, Virgo evolved out of a relatively mature project whose aim was to make it easy for enterprise applications and enterprise application developers to adopt OSGi. The goal was to make the transition to OSGi incremental. Essentially you can start with a standard WAR file and deploy it to Virgo as-is. Then you can incrementally refactor the WAR file into an application comprising multiple OSGi bundles while continuing to use familiar technologies such as Spring, Hibernate, JPA, etc. We based Virgo on Tomcat suitably embedded into OSGi, again to provide familiarity for enterprise developers as well as systems administrators.

In parallel with the evolution of Virgo, the OSGi Alliance Enterprise Expert Group, being discussed on Tuesday, produced a series of enterprise specifications again with the goal of enabling enterprise applications to be migrate straightforwardly to OSGi. The reference implementations of these specifications became today's Eclipse Gemini project:

Gemini Blueprint - an OSGi standard dependency injection container which also supports the popular Spring and Spring DM namespaces
Gemini Web - support for servlets in the form of OSGi standard Web Application Bundles (WABs) or standard WAR files
Gemini JPA - support for JPA persistence in OSGi
Gemini DBAccess - modularised JDBC drivers for use in OSGi
Gemini Management - JMX management for OSGi
Gemini Naming - support for JNDI naming in OSGi

Until recently, integrating the Gemini projects into Virgo has been a fairly specialised skill. As we'll see in a workshop on Thursday, things are now really starting to come together and Gemini Blueprint, Gemini Web, and various other Gemini components can now be deployed and used in Virgo. We'll also be touching on how to manage and configure WABs in Virgo as part of a workshop later this morning.

So much for the past. What about the future? The primary goal of Virgo 3.0 is better integration with other EclipseRT technologies such as Equinox, Jetty, and p2. This will provide richer function and greater choice for enterprise applications as they migrate to OSGi. We'll be hearing about how Virgo and EclipseRT technologies work together on Wednesday. The recently released third milestone of this release shows the progress towards the goal of better integration with EclipseRT:

Virgo now uses Equinox Configuration Admin and Event Admin services instead of their Apache equivalents.
Virgo has moved to the new framework hooks model for isolation in Equinox which is part of the OSGi 4.3 standard soon to be issued for public review.
Virgo now provides two web servers: Virgo Tomcat Server, which continues to be based on Gemini Web, and Virgo Jetty Server. Some exciting technology for modularising the view portion of web applications is also emerging into the Virgo mainstream as we'll hear on Wednesday.
p2 support in Virgo is also making good progress although this is currently in prototype form.

Development tooling is also entering an exciting new phase which will greatly help applications migrate to enterprise OSGi:

The Libra project, which will be presented on Wednesday, is providing development tooling for building WABs and will ship in the Eclipse Indigo release train later this year.
The Virgo development tooling has entered the Eclipse IP process and will soon emerge in the Virgo project. Over time, the standards-based features of the Virgo tooling will be factored out and migrated into Libra.
OSGi runtime launchers are also being contributed to Libra. I would hope to see a Virgo launcher for Libra emerge so that Libra can start to take over from the Virgo tooling for many aspects of enterprise OSGi application development.

If you're attending EclipesCon, come and hear the talks, join in the workshops and BoFs and help to shape incremental OSGi in the Virgo, Gemini, and Libra projects.

Importing packages into a 3rd party bundle

2011-03-18T16:30:00.002Z

I regularly come across the situation where someone needs to import additional packages into a 3rd party OSGi bundle. Sometimes this is because the bundle manifest is incorrect, but more often it's because the bundle uses Class.forName or similar to load classes that are somehow configured at runtime.

Unless the bundle is signed, it's possible to edit the manifest to add the import, but this is a really bad idea. If the original bundle needs upgrading, you have to remember to re-edit the manifest. Also, it's easy to mess something up in the process. Finally, it's bad practice to change the contents of a bundle without modifying the bundle version.

Thankfully, there is a simple way to add imports to a bundle without resorting to such invasive surgery: use a fragment bundle. Fragment bundles can extend the manifest of their host in various ways. In particular, they can add package imports. The fragment bundle need only contain a manifest, for example:

Manifest-Version: 1.0

Bundle-ManifestVersion: 2

Bundle-SymbolicName: my.fragment

Bundle-Version: 1

Bundle-Name: My Fragment

Fragment-Host: host.symbolic.name;bundle-version="[2.4,2.5)"

Import-Package: extra.package;version="[1.1,1.2)"

More than one fragment bundle can attach to a given host bundle, so this technique can be used when unrelated groups of packages need to be imported: a separate fragment can be used for each group.

Each fragment bundle needs to be installed before its host is resolved at which point the fragment's manifest is merged into the host's and the resultant Bundle instance imports the packages added by the fragment. On Virgo, this can be achieved by placing the fragment bundles in repository/usr. When the host is deployed, the fragments are automatically installed along with the host.

Note: make sure you use the "bundle-version" attribute of Fragment-Host. "version" is not a standard attribute of Fragment-Host and if you use it accidentally in place of "bundle-version", chances are your fragment will not attach to its host.

Virgo 3.0.0.M03 and EclipseCon

2011-03-17T17:04:00.000Z

Milestone 3 of Virgo 3.0 is available for download. Apart from several bug fixes, this milestone includes a contribution from Tom Watson who is collaborating with the Virgo team to prepare the region digraph for its move to Equinox so that it is available to all Equinox users rather than just Virgo users.

We will be using milestone3 in tutorials at next week's EclipseCon. If you're attending, watch out for Virgo committers delivering the following sessions:

Building web applications with EclipseRT
Donating a mature project to Eclipse
Snaps, modular & dynamic web applications with Virgo
Virgo and RT playing together
Hands on with Gemini and Virgo--The new Stars of the Eclipse RT project!
Virgo BoF -- to be arranged, keep a look out!

Hope to see you there!

Thread Context Class Loading in Virgo

2011-02-28T17:12:00.000Z

A number of popular open source libraries, particularly persistence providers, use thread context class loading to load application types. This is an issue in OSGi once an application is divided into more than one bundle. Each bundle has its own class loader and no particular one of these bundle class loaders is necessarily suitable for loading all the application classes that libraries need to load.

Virgo overcomes this problem with its notion of a scoped application. A scoped application consists of either a PAR archive file containing the application artefacts (bundles, property files, plans, etc.) or a scoped plan referring to the application artefacts. The scope limits the visibility of application packages and services so that scoped applications do not interfere with each another. But Virgo also creates an additional synthetic context bundle for each scope and it's this bundle's class loader which is used for thread context class loading when the application calls out to libraries.

The synthetic context bundle is very simple. It's a bundle which uses Virgo's import-bundle manifest header to import each of the bundles in the scope. Thus all the packages exported by the application are available for thread context class loading by libraries. There is a minor restriction implied by this approach: no two bundles in a scoped application may export the same package, which would be pretty unusual.

The following example should help make things clear:

An Example Scoped Application

Here a scoped application contains two bundles A and B. A calls B which calls out to a library bundle L. If L uses the thread context class loader to load application types, it will have access to all the packages exported by bundles A and B. The synthetic context bundle, which imports bundles A and B, provides the thread context class loader for the scoped application.

This simple solution enables a variety of existing open source libraries, once converted to OSGi bundles, to function correctly in their use of thread context class loading.

Stumbling towards a better design

2011-02-10T14:11:00.015Z

Some programs have a clear design and coding new features is quick and easy. Other programs are a patchwork quilt of barely comprehensible fragments, bug fixes, and glue. If you have to code new features for such programs, you're often better off rewriting them.

However there is a middle ground that I suspect is pretty common in these days of clean code and automated test suites: you have a good program with a clear design, but as you start to implement a new feature, you realise it's a force fit and you're not sure why. What to do?

I've recently being implementing a feature in Eclipse Virgo which raised this question and I hope sheds some light on how to proceed. Let's take a look.

The New Feature
I've been changing the way the Virgo kernel isolates itself from applications. Previously, Equinox supported nested OSGi frameworks and it was easy to isolate the kernel from applications by putting the applications in a nested framework (a "user region") and sharing selected packages and services with the kernel. However, the nested framework support is being withdrawn in favour of an OSGi standard set of framework hooks. These hooks let you control, or at least limit, the visibility of bundles, packages, and services -- all in a single framework.

So I set about re-basing Virgo on the framework hooks. The future looked good: eventually the hooks could be used to implement multiple user regions and even to rework the way application scoping is implemented in Virgo.

An Initial Implementation
One bundle in the Virgo kernel is responsible for region support, so I set about reworking it to use the framework hooks. After a couple of weeks the kernel and all its tests were running ok. However, the vision of using the hooks to implement multiple user regions and redo application scoping had receded into the distance given the rather basic way I had written the framework hooks. I had the option of ignoring this and calling "YAGNI!" (You Ain't Gonna Need It!). But I was certain that once I merged my branch into master, the necessary generalisation would drop down the list of priorities. Also, if I ever did prioritise the generalisation work, I would have forgotten much of what was then buzzing around my head.

Stepping Back
So the first step was to come up with a suitable abstract model. I had some ideas when we were discussing nested frameworks in the OSGi Alliance a couple of years ago: to partition the framework into groups of bundles and then to connect these groups together with one-way connections which would allow certain packages and services to be visible from one group to another.

Using Virgo terminology, I set about defining how I could partition the framework into regions and then connect the regions together using package, service, and bundle filters. At first it was tempting to avoid cycles in the graph, but it soon became clear that cycles are harmless and indeed necessary for modelling Virgo's existing kernel and user region, which need to be connected to each other with appropriate filters.

A Clean Abstraction
Soon I had a reasonably good idea of the kind of graph with filters that was necessary, so it was tempting to get coding and then refactor the thing into a reasonable shape. But I had very little idea of how the filtering of bundles and packages would interact. In the past I've found that refactoring from such a starting point can waste a lot of time, especially when tests have been written and need reworking. Code has inertia to being changed, so its often better to defer coding until I get a better understanding.

To get a clean abstraction and a clear understanding, while avoiding "analysis paralysis", I wrote a formal specification of these connected regions. This is essentially a mathematical model of the state of the graph and the operations on it. This kind of model enables properties of the system to be discovered before it is implemented in code. My friend and colleague, Steve Powell, was keen to review the spec and suggest several simplifications and before long, we had a formal spec with some rather nice algebraic properties for filtering and combining regions.

To give you a feel for how these properties look, take this example which says that "combining" two regions (used when describing the combined appearance of two regions) and then filtering is equivalent to filtering the two regions first and then combining the result:

Being a visual thinker and to make the formal spec more useful to non-mathematicians, I also drew plenty of pictures along the way. Here's an example graph of regions:

A New Implementation
I defined a RegionDigraph ("digraph" is short for "directed graph") interface, implemented it, and defined a suit of unit tests to give good code coverage. I then implemented a fresh collection of framework hooks in terms of the region digraph and then ripped out the old framework hooks and code supporting what in retrospect was a poorly formed notion of region membership and replaced this with the new framework hooks underpinned by the region digraph.

I Really Did Need It (IRDNI?)
It took a while to get all the kernel integration tests running again, mainly because the user region needs to be configured so that packages from the system bundle (which belongs in the kernel region) are imported along with some new services such as the region digraph service.

As problems occurred, I could step back and think in terms of the underlying graph. By writing appropriate toString methods on Region and RegionDigraph implementation classes, the model became easier to visualise in the debugger. This gives me hope that if and when other issues arise, I will have a better chance of debugging them because I can understand the underlying model.

A couple of significant issues turned up along the way, both related to the use of "side states" when Virgo deploys applications.

The first is the need to temporarily add bundle descriptions to the user region.

The second is the need to respect the region digraph when diagnosing resolver errors. This is relatively straightforward when deploying and diagnosing failures. It is less straightforward when dumping resolution failure states for offline analysis: the region digraph also needs to be dumped so it can also be used in the offline analysis.

These issues would have been much harder to address in the initial framework hooks implementation. The first issue would have involved some fairly arbitrary code to record and remove bundle descriptions from the user region. The second would have been much trickier as there was a poorly defined and overly static notion of region membership which wouldn't have lent itself to including in a state dump without looking like a gross hack. But with the region digraph it was easy to create a temporary "coregion" to contain the temporary bundle descriptions and it should be straightforward to capture the digraph alongside the state dump.

Ok, so I'm convinced that the region digraph is pulling its weight and isn't a bunch of YAGNI. But someone challenged me the other day by asking "Why do the framework hooks have to be so complex?".

Unnecessary Complexity?
Well, firstly the region digraph ensures consistent behaviour across the five framework hooks (bundle find, bundle event, service find, service event, and resolver hooks), especially regarding filtering behaviour, treatment of the system bundle, and transitive dependencies (i.e. across more than one region connection). This consistency should lead to fewer bugs, more consistent documentation, and ease of understanding for users.

Secondly, the region digraph is much more flexible than hooks based on a static notion of region membership: bundles may be added to the kernel after the user region has been created, application scoping should be relatively straightforward to rework in terms of regions thus giving scoping and regions consistent semantics (fewer bugs, better documentation etc), and multiple user regions should be relatively tractable to implement.

Thirdly, the region digraph should be an excellent basis for implementing the notion of a multi-bundle application. In the OSGi Alliance, we are currently discussing how to standardise the multi-bundle application constructs in Virgo, Apache Aries, the Paremus Service Fabric, and elsewhere. Indeed I regard it as a proof of concept that the framework hooks can be used to implement certain basic kinds of multi-bundle application. As a nice spin-off, the development of the region digraph has resulted in several Equinox bugs being fixed and some clarifications being made to the framework hooks specification.

Next Steps
I am writing this while the region digraph is "rippling" through the Virgo repositories on its way into the 3.0 line. But this item is starting to have a broader impact. Last week I gave a presentation on the region digraph to the OSGi Alliance's Enterprise Expert Group. There was a lot of interest and subsequently there has even been discussion of whether the feature should be implemented in Equinox so that it can be reused by other projects outside Virgo.

Postscript (30 March 2010)
The region digraph is working out well in Virgo. We had to rework the function underlying the admin console because there is no longer a "surrogate" bundle representing the kernel packages and services in the user region. To better represent the connections from the user region to the kernel, the runtime artefact model inside Virgo needs to be upgraded to understand regions directly. This is work in progress in the 3.0 line.

Meanwhile, Tom Watson, an Equinox committer, is working with me to move the region digraph code to Equinox. The rationale is to ensure that multiple users of the framework hooks can co-exist (by using the region digraph API instead of directly using the framework hooks).

Tom contributed several significant changes to the digraph code in Virgo including persistence support. When Virgo dumps a resolution failure state, it also dumps the region digraph. The dumped digraph is read back in later and used to provide a resolution hook for analysing the dumped state, which ensures consistency between the live resolver and the dumped state analysis.

How to develop with the Virgo Kernel in Eclipse

2010-11-25T11:50:00.000Z

A number of groups are discovering that the Virgo Kernel is a really useful OSGi runtime. For instance the Rich Ajax Platform project have documented how to run RAP on the kernel.

Others have steered clear of the kernel because the Eclipse-based Virgo tooling does not support the kernel directly (see bug 331101). However, it's really easy to get the tooling working with the kernel. Here's how...

First, set up the Virgo tooling in Eclipse as described in the Tooling chapter of the Virgo Programmer Guide.

However, when you create a new Virgo Web Server server runtime environment and point at a Virgo kernel installation, you'll see the following error:

Edit lib/.version in the kernel installation to contain the key 'virgo.server.version' instead of 'virgo.kernel.version'. Then the above error goes away and you can continue to set up the runtime environment and server as per the instructions in the Programmer Guide.

You can then use the Virgo tooling to do all the usual things: start and stop the kernel, create and deploy bundles, PARs and plans, examine bundles, and explore package and service wirings.

OSGi Adoption Experience

2010-11-11T20:02:00.001Z

The Mule team have blogged about their decision not to adopt OSGi. The decision seems reasonable: OSGi isn't a panacea and I guess the benefits didn't outweigh the costs in their case.

One of their number, Ross Mason, correctly identifies some of the costs of adopting OSGi. He would like OSGi to be almost completely invisible to the developer, leaving him or her free to define an application in terms of modules without dealing directly with manifests, bundles, and build systems. The somewhat messy details of developing and building modular applications with OSGi are partly what led SpringSource to donate the dm Server project to Eclipse as Virgo. The aim is to grow the user community and smooth out the wrinkles in the developer experience.

Adoption is going pretty well. Virgo recently shipped its first release. SAP are adopting Virgo in a strategic way. VMware products in development are using dm Server/Virgo to obtain the benefits of modular application code. Others are using dm Server/Virgo to host substantial applications in sectors including investment banking, networking, healthcare, online gaming, and academic research.

As we prepare to donate the Virgo tooling to Eclipse and collaborate in the OSGi Enterprise Tools project inside Eclipse with the likes of SAP, EclipseSource, and Eteration, I'm optimistic about improving the developer experience.

Dissenting voices such as those of the Mule team are a good reality check. Efforts, such as Mule, to try to find intermediate modularity solutions with some of the benefits of OSGi but with smaller costs should also help by providing data points for comparison with full blown OSGi adoption.

I'm looking forward to more experience reports from application development groups that have adopted OSGi or technologies like Mule so others can make a more informed decision about the technology to adopt. Certainly OSGi users are starting to publish their experience. Perhaps users of Mule will soon do the same.

Eclipse Virgo 2.1.0 Release

2010-11-01T12:23:00.000Z

Jump to Hyperspace, courtesy of Steve Jurvetson

Eclipse Virgo has shipped its first release after several months of work to contribute it to Eclipse.

The majority of the effort was to make the contribution acceptable from an Eclipse legal perspective. There were some close shaves, like when a file (mime.types) in the Spring Framework turned out to have a restrictive license. The file would have been virtually impossible to recreate from scratch. I am very grateful to the original author, Ian Graham, for solving this problem by making the content available under an Apache license. I would also like to thank Barb Cochrane and Sharon Corbett of the Eclipse legal team who cheerfully and efficiently handled over 200 "contribution questionnaires" covering the code donated to Virgo and its dependencies.

Apart from that, we have repackaged the code in the org.eclipse namespace, speeded up the startup considerably, upgraded several dependencies, fixed a few bugs, and made several enhancements.

The response from the community was particularly encouraging with thousands of downloads of the milestones and numerous forum posts and mailing list discussions. We are particularly grateful to the code contributors, listed in the release notes.

The main objective of contributing Virgo was to grow the community and thereby make the technology more easily consumed by application developers. The project seems to be meeting the community objective and we are now preparing to donate the Virgo tooling to Eclipse which will enable the community to help us make the developer experience smoother, especially for newcomers to modularity and OSGi.

Finally, I'd like to thank the other Virgo committers, Chris Frost and Steve Powell, for their persistence and thoroughness. I'm glad we can now move on to more interesting items for the next release. Thanks too to everyone else in SpringSource who supported the Virgo project in various ways. We couldn't have done it without you.

The future looks promising, starting with a Virgo face to face meeting at the end of this month. It's not too late to express your interest (on virgo-dev) if you would like to attend.

Eclipse Virgo: the ideal OSGi server runtime

2010-10-12T18:49:00.007+01:00

Since the Eclipse Virgo web site is seeing more new visitors than ever, it seemed like a good time to summarise the benefits of Virgo.

Essentially Virgo is the culmination of 3 years of work in developing an application server based entirely on OSGi and exposing OSGi for use by applications. Virgo started inside SpringSource as the dm Server project which was licensed under the GNU Public License and shipped two major versions. About a year ago we started donating the codebase to Eclipse as the Virgo project licensed under the much more liberal Eclipse Public License. The transfer of the runtime code to Eclipse is now complete, we have satisfied the rather stringent IP checks required of all Eclipse projects, and Virgo is nearing its first formal release by the end of October.

Virgo is an ideal OSGi server runtime, possibly the ideal OSGi server runtime, from several perspectives. Firstly, it was designed from the ground up as OSGi bundles so it's extremely modular and extensible. For example, the core of the runtime is the Virgo kernel and the Virgo web server is constructed by configuring a web layer, an admin console, and some other utilities on top of the kernel. The kernel protects itself from interference from such additions, and applications, by running them in a nested framework. This nested framework approach also enables applications to use new versions of the Spring framework (regardless of the fact the kernel depends on Spring 3.0.0). All the major Java EE application servers are now built on top of OSGi, but only Virgo was designed for OSGi and didn't need OSGi to be retrofitted.

Secondly, Virgo is an ideal OSGi server runtime because it exposes OSGi in a usable way to applications. In addition to all the standard features of OSGi — encapsulated modules or bundles, a powerful service registry, versioning, class space isolation, sharing of dependencies, dynamic refreshing and updating, and much more — Virgo provides a multi-bundle application model to simplify the deployment and management of non-trivial applications. Virgo also provides a repository which can store dependencies such as OSGi bundles which are "faulted in" when needed. This results in cleaner definitions of applications and a small footprint compared to traditional Java EE servers which pre-load many features just in case an application needs them. The major Java EE application servers are beginning to follow suit in exposing an application model, so SpringSource is working with other vendors in the OSGi Alliance to produce a standard multi-bundle application construct.

Thirdly, Virgo is an ideal OSGi server runtime because it enables existing Java libraries to run successfully in an OSGi environment. The Virgo kernel supports thread context class loading, load time weaving, classpath scanning, and a number of other features which are commonly used by persistence providers and other common Java utilities. Essentially, we observed the commonly-occurring problems when people attempted to migrate to OSGi and implemented general solutions to those problems early on.

Fourthly, Virgo is an ideal OSGi server runtime because it integrates OSGi with Spring and Tomcat. Virgo uses Spring DM to wire application contexts to the OSGi service registry. Spring beans can be published as OSGi services and can consume OSGi services, both with minimal effort. The embedded form of Tomcat is used as a servlet engine in Virgo's web support and is configured and managed just like standard Tomcat.

Fifthly, Virgo is an ideal OSGi server runtime because of its excellent diagnostics:

An admin console lets you examine the state of all artifacts deployed in Virgo as well as explore the state of bundles in the OSGi framework.
Virgo provides multiple types of diagnostics: event logging aimed at administrators, diagnostics logging aimed at developers, as well as various types of diagnostic dumps.
Virgo builds on Logback to support highly configurable and efficient logging.
When an application is deployed, Virgo first resolves the application in a "side state" and, if resolution fails, no changes are committed to the OSGi framework and the side state is dumped to disk so that it can be analysed using the OSGi state inspector in the admin console.
If a resolution failure occurs, Virgo also analyses the resolver error, including the root causes of any uses constraint violation, and extracts a meaningful message to include in an exception which is then thrown.
Virgo adds advanced diagnostics to Spring DM to track the starting of bundles and their application contexts and issue error messages on failure and warnings when dependencies are delayed.
Virgo automatically detects deadlocks and generates a thread stack dump.

Finally, Virgo is an ideal OSGi server runtime because it has advanced tooling support in the SpringSource Tool Suite or in standard Eclipse via an update site. This enables a Virgo server to run under the control of the tooling, applications to be deployed, debugged, and updated by the tooling, and package and service dependencies between bundles to be analysed. The Virgo tooling will also be donated to Eclipse in due course. Oh, I almost forgot to mention the obvious: Virgo is an open source project with a liberal license and active participation from multiple vendors, which positions it ideally for the future.

Footnote: The bulk of this article was originally posted to springsource.org. Reproducing here for the benefit of Planet Eclipse and others.

I Love OSGi, Now Please Change It!

2010-09-30T12:39:00.001+01:00

I'm sitting in the OSGi DevCon where a panel are discussing what needs to change in OSGi. Several points are coming up that are relevant to OSGi standards and also to what we have been doing in Virgo.

One excellent question was posed by a panel member, IBM Distinguished Engineer Ian Robinson, who asked whether OSGi needs to distinguish better between APIs intended for applications and APIs intended for middleware implementers and how this squares with the OSGi tendency for all bundles to be created equal.

I believe the answer here is to use composite bundles or nested frameworks to provide a unit of modularity larger than an individual bundle. I don't think OSGi needs to prescribe which APIs fall into each category, but it should provide mechanisms for hiding APIs in a given environment which are not intended for application use.

Virgo uses an early prototype of nested frameworks to achieve this kind of isolation between the Virgo kernel and the user region, which is a nested framework in which applications run. Users are free to deploy bundles in the user region and can even install additional bundles into the kernel and configure the packages and services shared between the kernel and the user region.

Fortunately, the notion of composite bundles and nested frameworks is being actively developed in the OSGi standards work and so their mechanisms should become more broadly available in a standard form in due course.

There is also OSGi standards work going on to expose these mechanisms in a usable way to applications. Virgo provides plans and plan archives as a way of grouping together artefacts for deployment and management. Apache Aries has also been implementing some similar grouping mechanisms with their application model. Paremus and others have similar constructs. So the standardisation activity has plenty of implementation experience and exposure to users to inform it. Virgo has also introduced a scoping mechanism as a way of isolating groups of application bundles from other groups of application bundles. This will also feed into the standards work.

Another great question from another panel member, the OSGi consultant Neil Bartlett, was how much OSGi should accommodate and support legacy code and how much it should require legacy code to be restructured to be more modular.

The direction of the Enterprise Expert Group in recent years has been to accommodate porting Java EE applications to an OSGi environment and then modularising them.

Virgo has had support for legacy code as one of the main design points from its inception. The Virgo kernel implements solutions to several knotty problems which crop up regularly when using non-OSGi libraries in an OSGi environment. These are assumptions about thread context class loading, load time weaving, class path scanning, and implicit package use.

So using implementations of OSGi Enterprise specifications is one part of the answer as is providing the kind of support for legacy libraries like that available in Virgo.

Virgo and Ocean Observatories Initiative

2010-09-26T09:48:00.001+01:00

Virgo seems to be finding its way into the scientific community. A research hospital and a networking research group already use Virgo heavily. It seems that an oceanography group are also considering Virgo for their "ioncore-java" runtime.

They are at the stage of comparing various application servers. The front-runners, at least in terms of their feature comparison table, are Virgo and WSO2. Fortunately, I believe the architecture and design of these products are quite different, so this should make the decision easier.

Virgo was built from the ground up as OSGi bundles and offers first class support for OSGi applications. The Virgo kernel is independent of any particular communications protocol. Tomcat is added as part of the Virgo web server which sits on top of the kernel. The whole of Virgo runs inside an OSGi container. Web services can be supported on Virgo, but it's not something Virgo has majored on as this is just one of the features available in the Spring portfolio running on Virgo.

WSO2 on the other hand is focussed on - and the clue's in the name here - web services. It is constructed as an interoperating network of web services components. WSO2 does support OSGi, but it hosts an OSGi container inside Tomcat.

Let's see what the Ocean Observatories Initiative regard as the key requirements for their component model. If it's OSGi components with primarily local communication (i.e. Java method call), then I reckon Virgo would be ideal. If on the other hand they want a primarily distributed component model (and it's a big if) with web services as the communication protocol, then WSO2 may be a good choice.

One last word. I felt a bit sorry for Glassfish as it didn't get a tick in the J2EE box even though it's the J2EE reference implementation. Also Tomcat is the servlet engine used by Virgo, so it deserves just as much of a J2EE tick in the box as Virgo. Similarly JBoss should get the J2EE tick.

Grails on Virgo

2010-07-12T08:36:00.000+01:00

Wolfgang Schell recently released an OSGi plugin for Grails. This runs on Virgo but apparently uses Spring DM's web extender rather than Virgo's web support. This makes me wonder if it would be better run on the Virgo kernel to avoid loading Virgo's web support. Perhaps he'll go into more detail when he has experimented more with Virgo.

RAP on Virgo

2010-07-11T10:57:00.001+01:00

Owen Ou blogged about deploying RAP (Eclipse Rich Ajax Platform) on an OSGi based server like Virgo. He says it's extremely easy, which is good to hear, but I'd love to see a bit more detail. For instance, assuming he's actually doing it on Virgo, is the RAP infrastructure launched as a plan in the initial list of artifacts for the user region? How are RAP applications then deployed? Fascinating stuff...

[The link to Owen's blog broke. I guess he deleted the blog. Anyway, Florian Waibel has also blogged about RAP on Virgo.]

Virgo one of the best and brightest Eclipse projects

2010-06-29T08:09:00.000+01:00

from eWeek.

Virgo kernel checked in and ready for use

2010-05-11T14:58:00.003+01:00

The Virgo kernel was checked in to Eclipse git earlier today and is ready for you to take it for a spin.

No, this isn't the Virgo kernel - it's a 5 Mb hard disk from 1956 which weighed over a ton. The Virgo kernel zip file would have needed a couple of tons of hard disk for storage. Thank goodness times have moved on.

IntelliJ IDEA Support for dm Server

2010-05-05T08:29:00.000+01:00

IntelliJ just announced support for dm Server. Although I haven't tried using the support, it appears to be at a reasonable level of function and only a little behind the SpringSource Tools Suite (as far as dm Server support is concerned) with plans to fill the gap. IntelliJ haven't mentioned Virgo yet, but I'm hopeful that the Eclipse (RT) branding won't put them off.

This, in my opinion, is a sign of a healthy runtime: multiple vendors competing on tooling. I'd like to see the same on management tooling which could be implemented relatively easily layer to dm Server's JMX interface.

First Virgo components available for development

2010-04-23T19:32:00.002+01:00

See this if you are interested in doing some Virgo development.

Special thanks go to the Eclipse IP team, especially Barb and Sharon, for making this possible at this stage.

Also thanks to Chris Frost for his help in raising CQs.

Virgo contribution underway

2010-04-01T16:46:00.000+01:00

I just finished entering the main CQs (Contribution Questionnaires) for Virgo. The corresponding tracking bugs, which are visible to everyone, are at the bottom of the list here.

There are now two gating factors before we can commit the code: the workload of the small but committed Eclipse legal team and my team's ability to field any issues they raise...

Once we've gotten the code committed, it will be possible to others to clone the git repositories, build, test, develop improvements, and submit patches. Essentially we'll be in business as far as contributors and build-it-yourself users are concerned.

Later, when we've also gotten the 3rd party dependencies through the legal process, we'll be able to start doing releases out of eclipse.org.

EclipseCon 2010 Virgo Presentation

2010-03-26T19:31:00.000Z

Eclipse Virgo

View more presentations from glynnormington.

Reflections on EclipseCon

2010-03-26T05:24:00.000Z

EclipseCon has been a great conference - on a par with the Spring conferences for enthusiasm and quality of talks etc. There was an enthusiastic welcome to the Virgo project which made my attendance really worthwhile. Several people discussed possible contributions to the Virgo project and there were a number of useful chats with potential consumers of Virgo as well as committers on other EclipseRT components who are keen to see Virgo fully integrate with the rest of RT.

For me, the highlight was the Virgo BoF with a room full of people really interested in learning more about Virgo. I was able to give a bit more detail about the internals of Virgo, clarify the separation of the various components, and start to explain the concepts underpinning the kernel.

I'd like to congratulate the organisers for putting on such a great conference and I look forward with anticipation to future EclipseCons.

Eclipse Virgo home page

2010-03-20T09:30:00.000Z

Eclipse Virgo. What more can I say, other than "Thanks Chris"?

EclipseCon Sessions

2010-03-17T10:54:00.000Z

EclipseCon starts in next week and there are plenty of sessions relevant to Virgo.

I'm doing the main Virgo talk and there's a Virgo launch BoF session (and a proposed BoF on Virgo tooling). I'm on panels on Gemini and the future of application servers and co-leading a BoF on application models for OSGi. I'm also giving a brief update on Virgo at the Eclipse Membership Meeting.

Then there's the main Gemini talk, an EclipseRT BoF, and other sessions on OSGi, Apache Aries, Maven and OSGi, and JPA and OSGi.

"Strong" optionality

2010-02-03T18:57:00.000Z

Richard Hall tells me Felix interprets optional imports as "strong", to use my earlier term. I believe there is enough latitude in the OSGi spec to permit this. Equinox takes the opposite interpretation and can discard optional imports of available packages in order to overcome uses constraints which would otherwise prevent resolution.

Regardless of that, I think that strong optionality is more likely to be what the user wants.

Without strong optionality, if the user provides a bundle which can satisfy some optional imports, they are not likely to be pleased if the resolver discards corresponding optional imports as the functions provided by the bundle will then be unavailable. Better to fail fast with diagnostics that allow the user to sort out the uses constraints.

Even worse, without strong optionality resolution may fail after a protracted attempt to discard combinations of optional imports in order to get a valid wiring. All the optional imports could be discarded before embarking upon a protracted search to ensure that the search is not in vain. Perhaps Equinox already does this, but I'm doubtful it as it could be criticised as optimising for the failure case.

OSGi resolution is NP-Complete. So what?

2010-02-03T06:31:00.010Z

At last we have a proof, thanks to Robert Dunne, that the OSGi R4 resolution problem is NP-Complete. He showed that it is possible to represent a solution to the NP-Complete boolean satisfaction problem 3-SAT as a wiring of a carefully constructed set of OSGi bundles. After the idea arose of using a SAT solver to construct an OSGi resolver, some of us have been discussing the possibility of a SAT-based proof. The hard step was how to represent a boolean OR in terms of resolution, which Robert has now provided.

NP-Completeness tallies with the experience of those who have written an OSGi R4 resolver: it takes about a fortnight of intensive hacking, combined with sleepless nights, to crack the problem followed by a much longer period of bug fixing and optimisation. Now we know why: NP-Complete problems are hard to solve efficiently.

I'm partly to blame as I led the spec work on OSGi RFC 79 which added considerable function, and with it complexity, to the resolution algorithm. I presented some of the background at the 2004 OSGi Congress. The RFC 79 spec was one of the more interesting that the OSGi Core Platform Expert Group has worked on. There was a tension between addressing the basic use cases and creating something that was too hard to implement.

Fortunately, we developed a prototype resolver in the Equinox incubator. This provided useful feedback to the spec work, particularly when features were dropped from the spec which made the implementation more tractable. The team which created the prototype included Simon Burns, Steve Poole, and Tom Watson. Simon wrote the initial R4 resolver. Steve created a rigorous model-based test based on a slow, but functionally corrrect, resolver of his own. Tom provided much advice on the R3 resolver and eventually integrated the new resolver into Equinox and has been improving it ever since.

So what features emerged from this process that made the resolver NP-Complete? Well, crucial to the proof are the uses constraint and, its evil twin, optional imports. Steve Powell observes that the proof does not depend on the transitivity of the uses constraint. (Other, arguably much more evil, features in R4, require-bundle and fragments, also did not contribute to the resolver's intrinsic complexity.)

This suggests some practical ways round the problem. Examples occasionally crop up when the time to resolve a set of bundles becomes unacceptable. How can we work around such issues if we are willing to put in the effort?

Firstly, uses constraints can be avoided by preventing types from one package leaking out through the interface of another package. This is an application of the Law of Demeter.

Secondly, in some cases it may be possible to remove, or at least cut down, optional imports by using mandatory imports instead and splitting the function which has those dependencies across bundles so that the user's dependencies determine which bundles are needed.

These approaches are feasible only for new code. Bundles generated from popular Java libraries tend to exhibit both uses constraints and optional imports.

Another possibility we've been considering is a "strong" form of optionality in which the optional import cannot be discarded if it is locally satisfiable, i.e. before uses constraints are applied. Certainly the current definition of optional import is critical to Robert's proof. The challenge is to convince ourselves that R4 resolution with strong optionality in place of the current definition is no longer NP-Complete or, conversely, to prove the opposite.

I'm undecided. Strong optionality drastically reduces the search space in some cases. On the other hand, I feel there may be a proof based on 3-SAT in which multiple bundles import, but do not export, packages.

OSGi standards and adoption

2010-01-27T09:01:00.006Z

The TSS thread on Virgo and whether or not OSGi is ready for mainstream enterprise application development contains some misconceptions.

Firstly, that SpringSource is somehow not involved in the OSGi standards process.

That couldn't be further from the truth. Recently we've led the standardisation of Spring DM into the blueprint service specification and reference implementation while IBM provided the compliance tests. We are developing the reference implementation of the OSGi web container while IBM wrote the specification and are providing the compliance tests. Then, as we look to the future, we are leading two, and involved in other, new specifications targeted at R4.3.

Secondly, it is easy to get the impression that other vendors, such as IBM and Oracle, are about to solve the adoption problem.

I certainly hope they do, but, at least in the case of IBM's WebSphere Application Server OSGi alpha, their programming models seem very similar to that of dm Server and their development and build tools, where the real adoption issues seem to be, do not seem to be particularly advanced.

Perhaps IBM's large customer base and marketing machine will encourage developers to jump the necessary hurdles. But the tricky thing about a product like WAS is that, after the alpha code has been rolled into the main product, it will be very hard to measure real usage. The only real evidence will be customer testimonials and APAR (i.e. external defect) rates against the OSGi features, which IBM typically do not publish.

Let's see how Virgo gets on. At least the adoption should be easier to measure as users will almost certainly be using it for its OSGi features.

Building OSGi applications

2010-01-15T09:35:00.003Z

The recent SD Times article on Virgo paints an overly bleak picture of the problems of building an OSGi application.

Build certainly complicates adopting OSGi, but many users are willing to overcome that problem in order to get the benefits of modularity and the service registry.

Also various tools are available to help. We developed the bundlor tool for automatically generating bundle manifests from the code in a bundle. Peter Kriens's bnd tool provides similar function.

People have successfully developed Maven and Ant builds for OSGi projects and various helper plugins. for instance PAX Construct, are starting to emerge.

Also, SD Times fails to mention the Enterprise Bundle Repository which already makes available (directly or via other repository front ends) about a thousand of the most popular open source Java components packaged as OSGi bundles with carefully constructed bundle manifests.

Eclipse Virgo project

2010-01-12T11:12:00.002Z

The news just broke. SpringSource are donating the dm Server code base to eclipse.org as part of the EclipseRT umbrella project. This should be just the server runtime that EclipseRT has been looking for. The Virgo project proposal is available and more details are on the SpringSource blog.

WebSphere OSGi Application Model

2009-12-08T09:29:00.003Z

As noted previously, WebSphere Application Server's alpha support for OSGi applications is similar to that of SpringSource dm Server. Now IBM's open source server, Geronimo, has taken its first steps to join the party.

Apparently Geronimo 3.0 is now based on OSGi, like all other modern Java app servers. (I wonder what vestigial traces of its GBean/XBean heritage remain?)

Geronimo's OSGi application model is provided by the Aries incubator, so I wonder if that is also the base of the WAS alpha, since Geronimo is essentially WAS CE.

Maybe the answer is that the closed source WAS is also based on Aries, but I didn't dare accept the license agreement on the download to find out.

SpringSource dm Server catching on

2009-12-07T05:27:00.004Z

The recently announced alpha driver of IBM WebSphere Application Server v7 support for OSGi applications is covered in a brief dzone interview with Kirk Knoernschild of the Burton Group.

As Kirk notes in his blog, SpringSource dm Server has already brought OSGi to the enterprise. dm Server v1.0 shipped in September 2008 and v2.0 will enter RC1 in the next few weeks.

The WAS alpha has some features in common with dm Server, notably repositories and applications along similar lines to plans and PARs. Immitation is the sincerest form of flattery or, putting it another way, IBM has validated dm Server's approach.

I'm still waiting for technical details of WebLogic DM Server, but since it is microkernel based (unlike WAS) and with a name like that, it's likely to validate dm Server further.

Meanwhile GlassFish and JBoss support for OSGi is coming along nicely, so it'll be fascinating to see what kind of OSGi application models these projects end up providing.

Jigsaw/JSR 294 on JavaPosse

2009-06-16T20:17:00.002+01:00

I listened to most of the JavaPosse interview with Mark Reinhold and Alex Buckley about Project Jigsaw and JSR 294. It's nice to see some technical details emerging and I can start to see why OpenJDK would prefer a different design for modularity to that of Apache Harmony.

I was moderately surprised to find that the design point is to load multiple Jigsaw modules with "local" dependencies into the same class loader. I wonder how many class loaders the JRE will end up having? There was even talk of putting all the bootstrap classes in a single class loader. So maybe the answer is most of the JRE code will be loaded by in the bootstrap class loader and subdivided using JSR 294 constructs. Time will no doubt tell.

For the record, I hope I've never claimed that Apache Harmony has shipped runtime modularity. The Harmony class libraries are divided into bundles with OSGi metadata. A prototype that enforced those class loader boundaries in the JVM was built, but never shipped. It had a rather different design for bootstrap class loading to that being proposed for OpenJDK, but I suppose Harmony had the benefit of having modularity in mind from the start.

OSGi, Scala, maven, and git

2009-06-13T05:41:00.005+01:00

I've noticed a few people are integrating OSGi and Scala. In particular Heiko Seeberger is building OSGi bundles for the Scala language plus various utility modules and DSLs. Then he's using maven for build and git for version control.

I am, of course, a fan of OSGi. I'm rapidly warming to git. But I'm not sure Scala and maven really solve more problems than they introduce. I don't want to dwell on maven as I'm not sure other build systems are really much better, but Scala does raise lots of question in my mind.

Many years ago, I was keen to adopt new technology and some combinations worked really well while others just seemed to multiply the overall complexity. One particular combination was the use of Z specification language embedded using literate programming in a subset of our target programming language with proof rules. It was possible to create proof trees of simple specifications refined to the target language. The main problem was that there were very few people who could acquire the skills necessary to master the combined technology.

Well, perhaps that was an extreme example and maybe it was doomed from the start, but I have a niggling doubt about the size of the Scala language.

So I'd be delighted to understand what features of OSGi and Scala work well together (and what features are simply best not used as part of this combination). In particular, are there any particular problems which are significantly simplified by combining OSGi and Scala which cannot be attacked very well using OSGi (and Java) or Scala alone?

Postscript

This document partially answers my questions. Scala makes it easy to write a DSL for handling OSGi services simply.

This is similar to the kind of function provided by Spring DM (or, equivalently, the OSGi Blueprint service).

The main difference is that there is still some minimal clutter in the Scala program whereas with Spring DM POJOs are configured in XML to provide and use OSGi services. I suppose the Scala approach favours sophisticated use of OSGi services, but this is not the norm in typical enterprise applications.

Coding and hill-walking

2009-06-12T18:32:00.008+01:00

I've recently been re-organising the internals of the SpringSource dm Kernel so that it can be extended to provide structured install artefacts known as "plans". The current code has been extended to provide a plan equivalent to the existing PAR files, but other variants of plans, unscoped and non-atomic, cannot be implemented without the necessary data structures being put in place.

From early in the development of dm Server 1.0, the team was using several terms which summarised the behaviour of the kernel, but which were not easily identified in the code. Terms such as "scope" and "deployment pipeline". As we started thinking about 2.0, we identified "plan", "install tree", and "install environment" as basic concepts, even if the latter was pretty fluffy.

Scopes were implemented as first class types quite a while ago now and were crucial to being able to make cloning work. One of the trickiest areas was performing operations on types which could be cloned or non-cloned. Explicit scopes enabled these operations to be coded to a non-cloned interface type and re-dispatched as necessary to the appropriate cloned type.

Recently, I've been turning deployment pipeline, install tree, and install environment into first class types. I really love this kind of work. It's rather like walking through a range of hills where the higher peaks are only visible when some of the lower summits are behind you.

Sometimes however things get much more strenuous. It's like the terrain is getting much steeper and there no obvious path to follow.

This afternoon I was struggling to structure the code to handle the starting of bundles. There's an interesting coordination and state management issue because starting a bundle has a synchronous phase defined by OSGi and often an asynchronous phase defined by Spring DM or the OSGi Blueprint Service. I made some progress, but I'm still not happy with the current code and it certainly isn't easily unit tested which is a good measure of code quality.

At this point, the only option is to take a break and come back to the problem with a clear head. I tried consulting colleagues and that helped quite a bit, but there are hopefully some really good abstractions waiting to be discovered.

I think the root cause of the complexity of this small area is that it is fairly low level and we're lacking some really good ways of thinking about the asynchronous processes involved. Occasionally, modelling processes in CSP and exploring the properties of the model using a model checker such as FDR has helped. But more often than not, the best approach seems to be iteratively cleaning up a bunch of code until a good structure starts to emerge.

In the worst case, I'll make the code as clean as possible and add it to the list of areas where future discoveries will hopefully make things cleaner. But I'm hoping for a much better outcome.

Well, that's enough for now. I put this blog together as I expect other people have similar experiences when working in complex areas of software and it's good to learn from each other. I'd be interested in your perspective and how you approach these kinds of activities.

I'll leave you with a hill-walking illustration of my current position: standing on one peak trying to make out the next one in the mist.

CaseInsensitiveMap

2009-04-28T14:17:00.004+01:00

I recently wrote some utility code which we'll be using to process bundle manifests, typically to map header names to typed values. It was surprising that this code was neither available elsewhere nor completely trivial to implement.

So what was it? A map with String keys which is insensitive to the case of the keys and which preserves the case of the keys. Apache Commons has a CaseInsensitiveMap which whacks the keys to lower case on input. I was tempted to use that code, but in our environment, it's important to preserve the case of keys.

For example, the bundlor tool will eventually use this code to create bundle manifests from templates. If the user carefully spells headers in the template with mixed case, e.g. "Export-Package", they would probably be disappointed to see the headers converted to lower case in the generated manifest.

Implementing an (Apache licensed) case-preserving CaseInsensitiveMap was an interesting exercise. I saved some time by extending AbstractMap and AbstractSet (for the key and entry sets). I used an instance of ConcurrentHashMap to maintain the state. I exposed one internal type to be package visible so I could get 100% unit test coverage and made some inner classes static to stop findbugs complaining.

Why blog about this? Simply to increase your chances of finding the class if you need it.

SpringSource dm Server 2.0 M1

2009-04-09T09:28:00.003+01:00

As in Rob's blog, we recently shipped the first milestone of dm Server v2.0. Apart from some new function, this marks an opening up of the dm Server development process with publicly accessible subversion repositories and issue tracker.

The 2.0 roadmap is pretty exciting, and includes constructing a Reference Implementation for RFC 66 ("web container for OSGi") from the existing web components of dm Server.

I'm spending most of my time down in the dm Kernel implementing "cloning" which, roughly speaking, enables a bundle to be wired more than one way, which is not normally possible in OSGi. Cloning copies bundles into an application's scope. This enables the OSGi resolver to have another go at wiring the dependencies of the cloned bundles in a way which suits the application. More information on cloning is included in the above blog entries.

Application scopes can be thought of as an alternative to "nested" OSGi frameworks, which are being specified in the OSGi Alliance and implemented in Equinox. The contents of an application scope can "see" all the contents of the global scope which are not "shadowed" in the application scope. The contents of a nested framework can, by default, not see any of the content of the parent framework(s) unless the nested framework is explicitly defined to see some of the parents' contents.

Much of what we're learning about cloning into application scopes, such as how to calculate what needs cloning and how to manage cloned extenders, will also apply to cloning into nested frameworks if and when that becomes necessary.

Is JDK 7 Java?

2009-04-08T14:27:00.003+01:00

I wonder if the JCP rules allow JDK 7 to be called "Java"? Sun appears to think so since the JDK 7 page is entitled "jdk7: Java SE 7". Apache Harmony meanwhile will presumably avoid calling itself "Java" until it can obtain and pass the JCK. Does this make Harmony the true guardian of Java? ;-)

Progress on JSR 294

2009-03-26T08:36:00.003Z

JSR 294 seems to be underway again. BJ Hargrave describes an EG discussion of visibility and access control which I found enlightening.

Project Jigsaw

2008-12-09T15:05:00.003Z

Mark Reinhold recently announced Project Jigsaw for modularising the JDK. Could this be a step towards the dream solution? I can hope.

Meanwhile JSR 277 is deferred beyond Java 7. Joy.

Refactoring and Structure101

2008-11-06T09:05:00.003Z

I'm in the middle of a fairly broad refactoring of the internals of the SpringSource dm Server to introduce an interface to some of the subcomponents and ensure that the rest of the code accesses those subcomponents via the interface rather than directly.

The Structure101 tool has proved invaluable in this process. It has excellent support for analysing the dependencies between components and identifying anomalous dependencies such as those from lower layers of the code to higher layers. But it also supports transformations and visibility settings which turn out to be extremely useful for refactoring.

I created transformations to collect together packages which I want to treat as a unit from the point of view of the current refactoring. This would not be possible using standard refactoring techniques as the packages in question cannot actually be renamed, for reasons I won't go into here. The transformations were of the form:

org.foo.* -> myunit.org.foo.*
org.bar.* -> myunit.org.bar.*

The transformations apply only in Structure101's model of the system and do not affect the actual code. The net is that I can then analyse dependencies on the myunit package hierarchy using Structure101's default mechanisms.

Visibility settings come in useful in identifying the pieces of code I need to refactor. By modifying the Structure101 model so that myunit is a private subcomponent of the new interface component, all dependencies on myunit from outside of the interface component show up as anomalies.

I can then use these anomalies as a 'to do' list for my refactoring. Clicking on one of them shows the precise code dependency which I can refactor by extending the interface component, if necessary, and then changing the using code to depend on the interface component. Refreshing the Structure101 model takes a few seconds after which there is one less anomaly showing on the diagram.

I've really appreciated the support from the Structure101 developers who have helped me get the best out of the tool. It sure beats some combination of grep, a cross-referencing tool like OpenGrok, plus a ton of yellow stickies...

100% bloat free

2008-06-09T17:23:00.004+01:00

Here's the image from the T-shirts that were given away at JavaOne. Click the image to enlarge it.

SpringSource Application Platform ships

2008-04-30T04:38:00.013+01:00

Today the SpringSource Application Platform shipped its first beta release. Version 1.0 is essentially a combination of Spring, OSGi and Tomcat engineered into a stand-alone runtime.

The programming model is Spring with OSGi and the option of a new packaging format: the platform archive or PAR file. A PAR file comprises multiple OSGi bundles. This brings OSGi modularity and versioning to enterprise applications.

The diagram below gives a high level view of the structure of the platform.

So what does the platform deliver over and above a standard OSGi container? Well, apart from the servlet function, integrated Spring support, the modular kernel-based runtime, and the full support for diagnostics and first failure data capture, there are quite a few features that are of interest to an OSGi audience:

The platform has a repository which automatically provisions dependencies as bundles are installed into OSGi. This repository stores the platform's own bundles as well as application and third party bundles.
In addition to the PAR mentioned above, there is a new concept of a library which is a named, versioned collection of OSGi bundles installed in the repository typically for use by applications. This is accessed via a new manifest header, import-library, which imports all the packages exported by the bundles of a library.
Another new manifest header, import-bundle, imports all the packages exported by a single bundle. This avoids the need to wrap components consisting of a single bundle in a library.
There is also some novel diagnostic help when a bundle cannot be resolved. The platform's resolution failure detective attempts to extract a concise summary of the problem and the most likely root causes.
Associated with the platform is a carefully constructed repository of commonly used bundles: the SpringSource Enterprise Bundle Repository known informally as the "bundle repository in the sky", or "BRITS". The bundles have validated OSGi manifests and the whole repository has been checked to make sure that its bundles can be used together.

Finally, I can't resist saying a bit about the architecture of the platform. In the past, I've been involved in restructuring projects - adding in modularity after the fact. The platform was built in a modular fashion from day 1.

The whole platform consists of subsystems which are similar in concept to libraries but which are managed specially by the kernel. Each subsystem is a named, versioned collection of bundles. The kernel manages the installation and starting of subsystems as the platform initialises. The kernel manages the lifecycle of each subsystem and its bundles, even reacting to unsolicited lifecycle events of the bundles of a subsystem to keep the state of the subsystem consistent.

Subsystems are used to control the function of the platform. Some subsystems are always present, but others are optional and are configured in a profile. This mechanism allows the precise function of the platform to be configured without rebuilding. For instance, the shipped profile.config file (in the config directory) looks like this:


/*
 * SpringSource Application Platform profile manager default
 * configuration file.
 */
{
 "profile": {
   "version" : 1.0,
   "name" : "web",
   "subsystems" : ["com.springsource.platform.servlet",            
                   "com.springsource.platform.web"]
 }
}

which shows the servlet container and web personality subsystems.

My colleague Rob Harrop just blogged about the platform if you want to know more. Why not register, download the binary, have a play, and tell us what you think.

OSGi could be great for JSR 277

2008-04-15T08:36:00.006+01:00

Bryan Atsatt describes a plausible way forward for JSR 277. I agree with his reasoning, although OSGi is not only the de facto standard for Java modularity, it's a proper standard too.

Bryan was one of the key movers in the Expert Group when I was involved. In fact, he proposed a solution to JSR 294 ages ago which is close, if not identical, to the approach Alex Buckley is now considering.

Welcome to the blogosphere, Bryan!

Popular RFE

2008-03-01T09:19:00.003Z

My favourite sunbug is now on the list of the Top 25 RFEs (Requests for Enhancement).

The 11th most popular sunbug?

2008-02-13T22:58:00.006Z

Sunbug 6650394 is coming along nicely - both in terms of votes and supportive comments. At the time of writing, it has 99 votes, so thank you if you took the trouble to vote! If you care and you haven't yet voted or commented, please consider doing so.

I was hoping the bug would make it into Sun's Top 25 Bugs list since it has more votes than most of the bugs in that list, but for some reason it's not yet included. If the list was accurate, which it clearly isn't, then 6650394 would be the 11th most popular sunbug.

Sunbug for JSR 277 module system interoperation

2008-01-31T09:07:00.000Z

Sun has kindly raised sunbug 6650394 on my behalf to track the module system interoperation issue for JSR 277.

Please consider voting for this bug if you would like JSR 277 to interoperate with JSR 291 (OSGi). Add yourself to the watch list if you want to track the bug.

JSR 294 Early Draft

2008-01-01T12:16:00.000Z

The Early Draft Review of JSR 294 is now complete. Little seems to have changed since when I left the Expert Group. The main weakness from my perspective is that a class file must name its superpackage in order for the class to be a member of the superpackage. This means that you can't easily produce a program which runs on existing versions of Java but which will take advantage of superpackages when it runs on Java 7 and beyond.

Practically speaking, this means that superpackages will only be used in applications that are willing to restrict themselves to Java 7 and beyond. So I suspect that superpackages will not come into popular usage within the next few years.

Of course, this can be gotten round by using a custom class loader with some bytecode rewriting magic, but one of the reasons that JSR 277, and its progeny JSR 294, were introduced was to avoid application developers from having to create custom class loaders.

SpringSource

2007-11-21T01:54:00.000Z

A couple of weeks in the new job and I work for a different company! Well almost - Interface21 has been renamed to SpringSource to strengthen the association with the Spring portfolio of products.
New web site, new email addresses, new logo, new business cards, etc. On the other hand the job is unchanged, as is the green corporate colour scheme...

Life in a small company

2007-11-14T10:44:00.000Z

I'm gradually noticing what's different about working in a small company. For one thing, there are very few processes, other than for software development where there are one or two.

For instance, I just took this photo. of the room I work in. Damilola (on the left) and Chris (on the right) noticed the idiot lifting his macbook into mid-air, but Nancie (in the distance) apparently didn't. I wonder how many levels of sign-off would have been needed to publish such a photo. in my previous company?

A more telling example of the minimal processes was that it recently took about an hour to obtain licenses for a piece of commercial software we decided to use. This included getting financial sign-off from someone on a business trip on the West Coast, submitting the order, and getting the license keys ready to use.

This is a pleasant change for me. Processes are sometimes essential, but it's so easy to make them too heavy. The emphasis in a small company seems to be talking to other people and trusting them to do the right thing.

Joining Interface21

2007-11-06T07:27:00.000Z

Today I start work for Interface21 in Southampton. After more than a month of gardening leave, or more accurately decorating leave, I can't wait!

OSGi, type-safety, and the 'uses' directive

2007-10-17T13:51:00.000+01:00

Java preserves type-safety when loading classes, even if there are buggy or malicious custom class loaders present. This is important as without type-safety, Java would not be secure.

For example, it would be possible to access an object's private data using a spoofed version of a class with the private data declared public. Vijay Saraswat found just such a type-safety bug in an early version of Java which was fixed by the introduction of 'loading constraints'.

So now that Java is type-safe, why need OSGi be concerned? Well, although OSGi can't break Java's type-safety, its support for versioning increases the risk that exceptions, such as ClassCastException, will be thrown to preserve type-safety. This usually occurs where two loaded versions of a particular class come into 'contact'.

The most common case is where an attempt is made to cast an object of one version of a class to another version of the same class. There isn't necessarily anything malicious going on - it's simply that two legitimate versions of a class loaded in the same JVM have accidentally come into contact.

OSGi increases the risk of such exceptions because it supports versioning of bundles and their contents: two (or more) versions of a given bundle can run in a JVM at the same time. These bundles will typically export similar classes. Since each OSGi bundle has its own class loader and a class is uniquely identified at runtime by the combination of its fully qualified class name and the class loader instance that defined the class, there can be two versions of each exported class present in the JVM at the same time.

So, to minimise these problems, OSGi works hard to wire together bundles so that they will consistently use a particular version of each class. At a basic level, when OSGi is resolving the dependencies of a bundle, it uses resolved bundles to satisfy those dependencies in preference to unresolved bundles. This avoids gratuitously introducing two or more versions of a given bundle, together with its exported classes, into the system.

But sometimes multiple versions of a class can, and indeed must, legitimately coexist. This can occur when assembling a large system from pieces which have been developed independently. The separate pieces can pre-req. distinct versions of common bundles. If these common bundles don't 'leak out' of the pieces of the system that use them, there is no problem.

But if they do leak out, OSGi's 'uses' directive enables the bundles importing from the common bundles to specify the amount of 'leakage'. For example, the following method signature exposes a common type PType:


package org.bar.q;
...
public org.foo.common.p.PType someMethod() {...}

and the following manifest statements record the leakage:


import-package: org.foo.common.p
export-package: org.bar.q,uses:="org.foo.common.p"

OSGi always resolves bundle dependencies so as to respect the 'uses' constraints. There's even support in Peter Kriens' bundle manifest creation tool, bnd, which spots the most straightforward leakages, like the example above, and create the corresponding 'uses' clauses automatically.

More subtle leakages need to be handled by the bundle programmer, but usually it is fairly obvious that something tricky is going on. For example, the following method take a parameter and casts it to the common type PType:


package org.bar.q;
...
public void anotherMethod(Object o) {
... (org.foo.common.p.PType)o ...
}

To avoid class cast exceptions due to o being an instance of the wrong version of PType, the same manifest statements as in the first example above should be specified, although these could not be inferred automatically by bnd.

So with appropriate 'uses' clauses, OSGi is able to manage multiple versions of bundles and minimise type-safety exceptions.

Writing good code

2007-10-13T09:17:00.000+01:00

James Governor drew my attention to Laura Thomson's summary of good coding practice - like a memory jogger for Code Complete. Don't worry about the title if, like me, you have little interest in PHP.

Upcoming 'webinar' on Equinox OSGi

2007-10-05T17:28:00.001+01:00

Tom Watson and Simon Kaegi are giving an introduction to Eclipse Equinox and OSGi at noon EST on November 27th. Register if you're interested.

I hadn't come across Eclipse live before, but the archive has some OSGi podcasts which may be of interest. I guess Tom and Simon's talk will appear there in due course.

Free Burma!

2007-10-04T06:59:00.000+01:00

An estimated 200 people have been killed by the ruling military junta during the recent peaceful demonstrations in Burma. Although the country currently has a high profile in the news, Burma has a history of human rights abuses. More background may be found in the BBC's profile of Burma.

Today, October 4th 2007, thousands of bloggers worldwide are making a single post to raise awareness of the situation and press for change. If you have a blog, why not join in?

Leaving IBM

2007-09-26T16:59:00.000+01:00

Today I handed in my resignation and will be leaving IBM on November 5. I've had a very enjoyable 26 years here in Hursley working on a variety of projects from workstation graphics algorithms to high-end transaction servers. A year ago I expected to work in IBM until retirement, but I had no idea what an interesting opportunity was about to open up a few miles down the road.

Although I'll still be involved with OSGi, I'll be handing over the job of representing IBM on the JSR 277 and 294 Expert Groups to someone else. I'll be lucky to find anything else quite so contentious to blog about, although I'll still be very interested in the outcome and may find something to say.

Hope for interoperation between JSR 277 and JSR 291

2007-09-14T13:18:00.000+01:00

Infoq and Alex Miller picked up on my recent comments about the somewhat closed approach being adopted by JSR 277. Since the whole concept of interoperation between JSR 277 and JSR 291 seems to be doubted, I thought I'd better provide some reassurance.

The saving grace of both these JSRs is that they ultimately boil down to delegation networks between module class loaders. So there is a lingua franca lurking not very far under the surface. Granted the views presented to the module developer are quite different in detail.

Quite a while ago, I wrote a prototype module system interoperation framework, described in more detail here, to start to flesh out how these JSRs could interoperate. I regard it more of an existence proof that a reasonable solution may exist. However, it requires 'cutting' into the module systems at a deeper level than I think is currently being considered by the JSR 277 spec. lead and colleagues.

We discussed some of the options in an informal Expert Group meeting at JavaOne, but all we really concluded was that there was a lot of work ahead to get interoperation working smoothly. I think there is also a consensus among the interested parties that reasonable interoperation is essential to the success of JSR 277. So the sooner we can get more eyes, and brains, on the current proposal, the better.

FeatherCast

2007-09-11T15:40:00.000+01:00

FeatherCast occasionally has interesting short podcasts about goings on in Apache.

Today I noticed a Google advert on the site for humane poultry slaughter tools, presumably targetting open source, free range chicken farmers.

How to run a JSR Expert Group

2007-09-07T09:16:00.000+01:00

The modus operandum of the JSR 277 Expert Group is fascinating.

Strawman proposals are developed in private by the spec. lead, colleagues from Sun, and occasionally one or two others from elsewhere. These are then presented to the Expert Group for comment. After more or less debate, the spec. lead asserts the consensus and the strawman goes forward for inclusion in the draft spec. Typically, there can be a change of position when something small is clearly broken, but significant design changes and smaller tweaks tend to be resisted.

On the whole, the real JSR 277 contributors are working in private and the so-called Expert Group acts as a sounding board without any significant influence on the outcome. Some would call this rubber-stamping.

I'm trying to change this on one crucial front - that of interoperation between JSR 277 and JSR 291. But progress is dreadfully slow.

Apparently, interoperation prototypes are being developed and thinking is going on, but none of this is visible to the Expert Group or the broader community. This is a bit odd as a modules project has been set up on OpenJDK, so I would have expected that kind of prototyping to go on in a branch or subdirectory for early feedback from others.

It's worrying because the OSGi experts, Richard Hall and myself, in the JSR 277 Expert Group and the others in the JSR 291 Expert Group are not currently able to help. By the time we see a strawman, it may be too late to make any significant changes.

Clearly my experience of Apache and Eclipse is no guide to the way things happen in OpenJDK. Similarly, my experience of other standards bodies is also no guide to how a JSR Expert Group is necessarily run.

Decimal syntax

2007-08-30T13:02:00.000+01:00

On this topic, Danny Coward recently referred to a limited form of operator overloading specifically for BigDecimal. Good to know this is still in Sun's thinking.

I'm continuing to look for users and companies who are willing to back the requirement for decimal syntax in Java, regardless of whether this is via limited operator overloading or decimal-specific support, so if anyone is interested, I'm all ears.

Is enjoyable standards work a contradiction in terms?

2007-08-29T15:11:00.000+01:00

Eric Newcomer says he's enjoyed working in the OSGi Enterprise Expert Group. From my experience of the OSGi Core Platform Expert Group, which has a significant overlap with the EEG, I'd certainly agree.

The defining characteristic of the OSGi Alliance seems to be that the members are there with a fairly clear common purpose. Other standards groups I've been involved in - and I'm not just talking about the JCP - have tended to suffer from divergent objectives among the members.

Maybe the key is that OSGi provides a dynamic component model which is broadly applicable across small devices, PCs, and large servers and that it has strong conceptual integrity making it easy to learn and apply.

I also found Eric's recent InfoQ interview on OSGi futures interesting - take a look.

JSR 291 goes Final Release

2007-08-16T11:01:00.000+01:00

The last administrative hurdle for JSR 291 has been cleared and the spec., RI, and TCK are now available, either via the JCP site or directly and, somewhat more simply, from the OSGi site.

The Expert Group however lives on...

JCP Awards

2007-07-23T09:09:00.000+01:00

The JCP awards brightened up a wet Monday morning.

The Apache Software Foundation got the "Member of the Year" (!) award for its contributions and active participation in 2007.

JSR 308 was hailed as the most innovative JSR for Java SE/EE, although the other two contenders haven't issued a draft spec. yet either. A sign of things to come?

Not sure which JSR beat 291 in the "most contentious" category, but it could have been the abortive JSR 313.

But seriously, congratulations to "Participant of the Year" Wayne Carr who was a welcome voice of reason in the early days of JSR 277.

Java modularity: where do we go from here?

2007-07-16T10:05:00.000+01:00

Things have moved on somewhat since March, so I thought an update would be helpful. Let's recap first.

JSR 277 is trying to create a static module system for Java. Its offshoot, JSR 294, is focused on the VM and language support for modularity. Both these JSRs are targeting Java 7.

JSR 291, which recently passed the Final Approval Ballot, references and extends the OSGi dynamic component system. JSR 291 is compatible with the Java ME JSR 232, which also references OSGi.

OSGi is a mature technology, having been developed in four releases over the last eight years. There are three independent open source implementations (Equinox, Felix, and Knopflerfish), a specification, reference implementation, and TCK.

I am an active member of the JSR 277 and JSR 294 Expert Groups and am the spec. lead of JSR 291.

Design goals

Crucially, the goals of these JSRs are rather different, although the underlying concepts are similar. JSRs 277 and 294 are building basic support into the Java SE platform whereas JSR 291 is pure Java built on top of the Java platform.

Look at it another way. JSRs 277 and 294 are attempting to produce an evolution of java.lang.ClassLoader with a new distribution and packaging format, a new repository, and new language features supported by the VM. JSR 291 on the other hand is implemented in pure Java on top of java.lang.ClassLoader and uses standard JAR files with standard manifests as a distribution and packaging format.

The design point of OSGi has always been to support a broad set of Java ME environments, such as profiles based on the Connected Device Configuration, and Java SE versions, from Java 1.2 onwards, in a compatible way. It has achieved this by building on commonly available Java concepts, notably class loaders.

I think of OSGi as being the pinnacle of custom class loader systems. The way it manages to install, uninstall, start, stop, and update modules without restarting the VM is quite an achievement - not my achievement, I should add, as the dynamic features were mostly complete before I got involved. Of course, other systems have provided some of this capability, but not in such a generally reusable way.

So you can package OSGi with your application, or list it as a prerequisite, and then run on a wide range of Java platforms.

On the other hand, JSRs 277 and 294 have the advantage that if you are happy to pre-req. Java 7 or later, then those JSRs will be built in to the platform.

Why doesn't JSR 277 reuse OSGi?

So why can't the JCP simply adopt JSR 291 as a standard module system for the Java platform? That's what many of the critics of JSR 277 are calling for.

A crucial issue is one of control. I hope the OSGi Alliance would allow OSGi to be included in the Java platform so long as they retained control of the technology. But I suspect Sun would be nervous about the core module system of the Java platform not being under their direct control. Such nervousness would be understandable if it was not in the interests of the OSGi Alliance to maintain the value of Java, but the reality is just opposite: the success of the OSGi Alliance depends to a large extent on the success of Java.

In my earlier blog, I made the point about the OSGi technology still evolving. But I think that problem is soluble. There would need to be a way of overriding the built-in module system with a later, backward compatible version.

Why not scrap JSR 277?

A radical option, favoured by some, of simply scrapping JSR 277 is less than ideal. It would temporarily remove the confusion that competing module systems inevitably create. But before long, the requirements for a module system built in to the Java platform would surface again.

The consumer JRE work has shown there is a real need to modularise the JRE itself and this requires a built-in module system. However, the consumer JRE doesn't seem to be proposing a properly architecture module system. It's a 'quick fix' which can be delivered in Java 6 update releases.

So if JSR 277 was scrapped, then whatever mechanisms evolve out of the consumer JRE work could become the defacto module system for the Java platform. The Java platform would still lack a properly architected built-in module system.

JSR 277 also has a stated objective of interoperating with JSR 291, which means JSR 277 modules will be able to make effective use of OSGi modules and vice versa with minimal restrictions. If this objective is achieved, it will protect the significant investment in OSGi modules in the Java community. Scrapping JSR 277 would jeopardise this investment if a non-interoperating module system grew up in place of JSR 277.

The road ahead

So, given the momentum behind JSR 277 and the emphasis placed on it by Sun at this year's JavaOne, what lies ahead?

Of course, JSR 291 should ideally benefit from the features introduced by JSR 277 and JSR 294. But the immediate requirement is that JSR 277 and JSR 291 interoperate well so that modules written in terms of one JSR are able to make effective use of modules written in terms of the other JSR.

To achieve this, the Expert Groups of JSR 277 and JSR 291 will need to work together. The JSR 277 Expert Group are responsible for producing an interoperation proposal which the JSR 291 Expert Group will review. At the time of writing, the interoperation proposal is still being worked on by the JSR 277 spec. lead.

The dream solution

The dream solution is clear: JSR 277 should adopt JSR 291, underpinned by language and VM support in JSR 294, and add the JSR 277 repository architecture. This would accelerate the progress of JSR 277 by several years and guarantee perfect interoperation between JSR 277 and JSR 291.

But it would take a complete about turn by Sun to turn this dream into reality. Maybe, just maybe, Sun are sufficiently committed to the success of Java that they will eventually decide to do just that. Maybe, just maybe, the Java community will be able to look back later and paraphrase Winston Churchill: "Sun could always be counted on to do the right thing, after having exhausted all the alternatives".

Further information

The JSR 291 Expert Group mailing list and the JSR 294 list have been openly readable since their Expert Groups started. JSR 277 ran in private for many months, but has now opened up its list.

Disclaimer

I'm biassed.

I have been involved in the OSGi Core Platform Expert Group for a couple of years during which time I co-authored some of the modularity improvements for OSGi R4 with Richard Hall (the leader of the Apache Felix OSGi project).

I work for IBM which belongs to the OSGi Alliance and which has based some of its products on OSGi. IBM and Sun co-operate on Java, but compete in related areas such as Java EE, not to mention UNIX-style operating systems and hardware. So IBM and Sun don't necessarily agree on everything that happens in Java. In fact, I think it would be pretty unhealthy if that were the case.

I should also re-iterate that my opinions are not necessarily shared by IBM.

That said, I hope this blog objectively lays out the situation.

New development models for IBM

2007-07-02T11:40:00.000+01:00

IBM launched Project Zero on Friday. The aim is to provide simpler ways to build web applications using an approach termed Community-Driven Commercial Development.

The whole idea is to engage with the community of potential users to provide something that they want to use: similar to an open source project, but with a commercial license. There is a big emphasis on 'consumability' these days in IBM and this is an example of IBM changing its ways to produce more usable software.

Another new initiative, Jazz, is in contrast only accessible to IBM Rational's customers and partners plus invited others. I heard about it on a DeveloperWorks interview with Erich Gamma, of design patterns and Eclipse fame, but haven't had time to watch the videos. Anything Erich is endorsing is bound to be pretty interesting. If anyone has a link to a white paper or some such, please let me know.

JSR 291 but not as we know it

2007-06-29T16:43:00.000+01:00

Google sometimes coughs up unexpected stuff, like this which could almost be talking about JSR 291 and JSR 277. :-)

Eclipse Europa

2007-06-28T15:45:00.000+01:00

Coté's coverage of Eclipse Europa expresses a view on OSGi and JSR 277.

Representations of module dependencies

2007-06-22T17:03:00.001+01:00

One of the key differences between JSR 291 and JSR 277 is the way in which module dependencies are represented, satisfied, and managed. Both module systems express dependencies using import declarations with version ranges and so, in general, there can be multiple ways of satisfying any particular dependency.

Using an electronics analogy, there can be multiple different ways of wiring a module's dependencies to other modules. Without further constraints, the choice of wiring is essentially non-deterministic, which would make the module system hard to understand and unpredictable in behaviour. The non-determinism needs to be removed.

JSR 291 removes the non-determinism by specifying rules for how dependencies should be satisfied in the presence of a given collection of modules, some of which are already resolved (or 'wired') and others which are unresolved (not yet 'wired').

JSR 277 also specifies some rules to remove the non-determinism, but it allows a module to implement an import policy if the module wants to override these rules. The hope is that, in many simple cases, the import policy will not be needed. But, in general, JSR 277 removes the non-determinism using a combination of rules in the specification and user-written import policies.

Why is this difference important? Well, an import policy is a piece of Java code running inside the module whose dependencies need to be satisfied. So the import policy is in the unusual position of not being able to use the modules upon which its module depends. This is a potential source of errors.

But a more important difference relates to the need to be able predict the behaviour of a collection of modules. This is crucial when managing the dependencies of modules. With JSR 291, an external management system is able to read the dependency declarations in each module and apply the rules in the specification to determine how these modules will be wired up, whether there are any missing dependencies, and, if there are, how such dependencies could be satisfied.

With JSR 277 the position is quite different when import policies are used. The only way to determine the behaviour of an import policy is to execute it. But even then, there is no guarantee that the import policy will give the same result each time it runs. Also, if there are missing dependencies, it is not feasible to examine the import policy to determine how the missing dependencies could be satisfied.

(For further reading, I recommend Peter Kriens' JSR 277 review from last October.)

Module-superpackage relationship

2007-06-22T16:29:00.001+01:00

In the comparison of JSR 277 and JSR 291 features, I alluded to the fact that the module-superpackage relationship is a difference, but Andreas pointed out another way in which the relationship differs.

My original observation was that the cardinality of the relationship is different. JSR 277 specifies modules which build directly on superpackages: there is a one-one relationship between deployment modules and superpackages.

JSR 291, on the other hand, makes no explicit reference to superpackages as these are yet to find their way officially into Java. But, if JSR 294 continues in its current direction, it will be possible to include one or more superpackages inside a JSR 291 module. So the relationship between JSR 291 modules and superpackages will have 1:0..n cardinality.

Andreas' point was that JSR 277 is integrated with JSR 294 in that a JSR 277 module's metadata consists of a superpackage and its annotations.

JSR 291, in contrast, is currently agnostic about superpackages and their names, members, exports, and annotations. JSR 291 maintains its own metadata in a separate manifest file and so is not integrated with JSR 294.

Sun VP 'gets' OSGi

2007-06-22T10:21:00.000+01:00

I just ran across a video where James Governor interviews Jeet Kaul, the VP of developer products and programmes for Sun Microsystems. If you don't have much time, fast forward to three minutes from the start and listen to the next minute and a half on OSGi.

Some soundbytes plucked slightly out of context for maximum effect:

JG: "I'm looking at OSGi ... and I think it's possibly the most important standard in the industry right now." (What a nice man.)

JK: "... so much has been done already that it would be insane to ignore that ..."

and later

JK: "... in Java SE 7 where we are trying to build the standard which will incorporate supporting OSGi bundles - it's the right approach."

Default export granularity

2007-06-20T13:27:00.000+01:00

In the comparison of JSR 277 and JSR 291 features, I caused some confusion by saying the default export granularity was 'class' for JSR 277 and 'package' for JSR 291. I'll try to explain.

The primitive unit of export in JSR 277 is the class. You can use a wildcard to export all of a module's classes in a specified package, but that's just syntactic sugar. The wildcard will be expanded to a list of classes in the binary file representing the module's superpackage. There is no runtime notion of an exported package.

The primitive unit of export in JSR 291, on the other hand, is the package. You can filter the set of classes available to importers, but the runtime deals in exported packages rather than classes.

I hope that's a bit clearer. Also, try not to confuse default export granularity with import semantics. JSR 277 supports only module import whereas JSR 291 supports package and module imports.

Comparison of JSR 277 and JSR 291 features

2007-06-18T10:10:00.000+01:00

A number of people have expressed an interest in the similaries and differences between these JSRs, so this post compares the features of the Early Draft Review specification of JSR 277 and the Final Specification of JSR 291, although you'll have to read the specifications to understand the differences in detail.

This a functional comparison and doesn't attempt to draw out advantages and disadvantages of various features or other aspects of the JSRs. For ease of comparison, I've referred to JSR 291 bundles as modules.

Equivalent features:

Module name
Module version
Module attributes
Module import
Module re-export
Version range
Optional import
Executable modules
One class loader per module
Platform dependencies
Modules may contain classes, native libraries, resources, and extensible metadata
Module signing
JAR distribution format
Reflective API

Differences (order: JSR 277 vs JSR 291):

Programmatic import policy vs declarative import constraints (details)
Class space consistency: validation vs constraint solving
Default export granularity: class vs package (details)
Module-superpackage relationship (details)

Unique to JSR 277:

Standard repository:

organised hierarchically with parental delegation
multiple standard implementations for local or networked deployment
support for plugging in 3rd party repository implementations

JAM distribution format
Import override policy
Part of the Java SE platform (planned for Java 7)

Unique to JSR 291:

Dynamic lifecycle support both for module lifecycle and service lifecycle
Service support (above and beyond the basic service provider support in Java SE)
Package import
Dynamic package import
Package attributes
Mandatory attributes
Split package support
Module classpath
Fragment modules
Extension modules
Pure Java implemented on top of Java SE
Compatible with Java ME (JSR 232)