Virgo 3.5.0 with Nano Technology

Virgo 3.5.0.RELEASE is available for download.

Although the deliverables are mostly backward compatible with Virgo 3.0.x, we decided to indicate the importance of the release by bumping the version from 3.0 to 3.5.

All Virgo deliverables are now launched using the Equinox launcher and have a corresponding directory structure which is slightly different from that of 3.0.x. All the deliverables can all be initially provisioned using p2, which is useful, for instance, when setting up remote nodes or automating installation.

There are two new deliverables which the team is pretty excited about. Virgo nano is a cut-down subset of the kernel. It supports hot deployment of bundles via the pickup directory, the Gogo shell, a single (kernel) region, and the medic component which provides Virgo's basic diagnostic capabilities.

The second new deliverable is Nano Web which is essentially Gemini Web -- the Apache Tomcat based reference implementation of the OSGi Web Applications specification -- running on Nano. In addition to Nano's features, Nano Web supports hot deployment of Web Application Bundles and WAR files.

Compared to the kernel-based deliverables, which are already pretty small and fast, Nano starts much faster and has a much smaller footprint as the following charts indicate.¹

Both Nano and Nano Web can be provisioned with application bundles at runtime using p2. The kernel based deliverables can only be initially provisioned using p2. Applications are then deployed into the user region in the usual way. This provisioning limitation in p2 triggered the initial development of Nano, although Nano soon appeared to have other benefits such as smaller runtime footprint (particularly important in the cloud) and the possibility of using PDE for bundle development. Also, Nano Web is likely to integrate additional enterprise Java components in the future to support applications written for the Java EE web profile.

There are a number of other interesting features in 3.5.0 including integrated support for the OSGi Blueprint Service, several significant kernel enhancements, and various dependency upgrades - all described in the release notes.

Additionally, Virgo Bundlor 1.1.1 and the Greenpages 2.5.0 sample application are released in association with Virgo 3.5.0.

In parallel with the development of Virgo 3.5.0, the Virgo IDE tooling has been overhauled and will be released in the next few weeks. Meanwhile you can use milestones and snapshots of the new tooling as described on the tooling wiki page.

Please see the release notes for further details of Virgo 3.5.0 and the migration notes for information about upgrading from Virgo 3.0.x.

Footnote:

1. Startup times are approximate. Measurements were taken using Mac OS X 10.7.4, and Java 1.6.0_33 on a Mac Pro with 12 GB RAM and 2 x 2.66 GHz dual core Intel Xeon processors.

New Releases: Virgo 3.0 and Gemini Web 2.0

Virgo 3.0 and Gemini Web 2.0, collectively known as the Maya release, are available for download and download, respectively.

The team of committers has expanded considerably and, although the team is now distributed geographically, we have a great working relationship. I'm delighted that these projects are now community collaborations rather than the work of a single vendor.

The theme of Virgo 3.0 is better integration with EclipseRT technologies. To that end, we have created a Jetty variant of the Virgo web server and have switched from Felix to Equinox implementations of some OSGi services. Integration with p2 is proving a tough nut to crack and is deferred to the Bondi (3.5) release, although we expect to issue a milestone soon to get some feedback from the user community.

Gemini Web and the Tomcat variant of the Virgo web server have been upgraded to Tomcat 7 and Servlet 3.0.

Gemini Web 2.0 passes the OSGi Web Applications compliance tests and will replace the SpringSource OSGi Web Container as the OSGi Web Applications reference implementation.

The snaps framework for modular web applications is also released as part of Maya. This started life as the SpringSource slices prototype.

Virgo kernel 3.0 has a new shell based on Apache Felix Gogo for both the kernel and user region and accessible via telnet or ssh.

The 3.0 kernel also uses a new implementation of regions, the region digraph, based on OSGi standard framework hooks. The digraph should position Virgo well for implementing the OSGi Subsystems specification in a future release. (The specification is still under development, but we expect a public draft from the OSGi Alliance before long.) The digraph has moved from Virgo to Equinox to enable it to be used by other projects, especially those that are aiming to implement the Subsystems spec.

Highlights:

• New Virgo Jetty Server
• Virgo Tomcat Server and Gemini Web upgraded to Tomcat 7 and Servlet 3.0
• New Snaps framework for modular web applications with new guide
• New Gogo shell
• New user guide for Gemini Web
• Updated GreenPages sample with new guide

See the release notes for details.

Equinox digraph ready for use

Previously I described how Virgo used OSGi framework hooks to implement a directed graph, or digraph, of isolated groups of bundles known as regions. The region digraph has now moved to Equinox and I'm modifying Virgo to use the Equinox region digraph in place of Virgo's own (see bug 343364 for details).

I always prefer making such changes in an incremental way, so the first step was to add the Equinox region digraph bundle to the Virgo kernel without exploiting it. The Virgo kernel runs fine with the Equinox region digraph in situ. This is because the Equinox digraph adds all bundles to its kernel region and so provides no isolation between those bundles. Later, Virgo's region digraph partitions the bundles in Virgo kernel and Virgo user regions with appropriate isolation and things continue to work as normal.

This is interesting, but not particularly useful. The whole point of the region digraph abstraction was to make it easy for multiple parties wishing to provide isolation via the OSGi framework hooks to be able to collaborate together by sharing and manipulating a single region digraph. It was never intended that multiple digraphs should co-exist.

But there is a useful point here. If you have an Equinox-based system, of which there are many, you can introduce the Equinox digraph with zero functional impact. All your bundles, including those installed before and after the Equinox region bundle, go in the Equinox kernel region and there is no isolation between those bundles.

When you want to start exploiting the digraph, you can use the RegionDigraph service to create a new region, attach it with appropriate filters, if any, to and from the kernel region, and then install a bundle into the new region using the Region.installBundle operation. After starting this bundle, you will then have a bundle context that can be used to install further bundles into the new region using the OSGi standard BundleContext API.

I hope this will inspire you to have a go at using the Equinox region digraph if you are an Equinox user and need to isolate groups of bundles.

Meanwhile, the standardisation of multi-bundle applications, including isolation, continues in the OSGi Alliance and a public draft of this spec should be available in the not too distant future. I hope the Equinox region digraph will be used more widely and this will help to provide feedback on the spec work.

Stumbling towards a better design

Some programs have a clear design and coding new features is quick and easy. Other programs are a patchwork quilt of barely comprehensible fragments, bug fixes, and glue. If you have to code new features for such programs, you're often better off rewriting them.

However there is a middle ground that I suspect is pretty common in these days of clean code and automated test suites: you have a good program with a clear design, but as you start to implement a new feature, you realise it's a force fit and you're not sure why. What to do?

I've recently being implementing a feature in Eclipse Virgo which raised this question and I hope sheds some light on how to proceed. Let's take a look.


The New Feature
I've been changing the way the Virgo kernel isolates itself from applications. Previously, Equinox supported nested OSGi frameworks and it was easy to isolate the kernel from applications by putting the applications in a nested framework (a "user region") and sharing selected packages and services with the kernel. However, the nested framework support is being withdrawn in favour of an OSGi standard set of framework hooks. These hooks let you control, or at least limit, the visibility of bundles, packages, and services -- all in a single framework.

So I set about re-basing Virgo on the framework hooks. The future looked good: eventually the hooks could be used to implement multiple user regions and even to rework the way application scoping is implemented in Virgo.


An Initial Implementation
One bundle in the Virgo kernel is responsible for region support, so I set about reworking it to use the framework hooks. After a couple of weeks the kernel and all its tests were running ok. However, the vision of using the hooks to implement multiple user regions and redo application scoping had receded into the distance given the rather basic way I had written the framework hooks. I had the option of ignoring this and calling "YAGNI!" (You Ain't Gonna Need It!). But I was certain that once I merged my branch into master, the necessary generalisation would drop down the list of priorities. Also, if I ever did prioritise the generalisation work, I would have forgotten much of what was then buzzing around my head.


Stepping Back
So the first step was to come up with a suitable abstract model. I had some ideas when we were discussing nested frameworks in the OSGi Alliance a couple of years ago: to partition the framework into groups of bundles and then to connect these groups together with one-way connections which would allow certain packages and services to be visible from one group to another.

Using Virgo terminology, I set about defining how I could partition the framework into regions and then connect the regions together using package, service, and bundle filters. At first it was tempting to avoid cycles in the graph, but it soon became clear that cycles are harmless and indeed necessary for modelling Virgo's existing kernel and user region, which need to be connected to each other with appropriate filters.


A Clean Abstraction
Soon I had a reasonably good idea of the kind of graph with filters that was necessary, so it was tempting to get coding and then refactor the thing into a reasonable shape. But I had very little idea of how the filtering of bundles and packages would interact. In the past I've found that refactoring from such a starting point can waste a lot of time, especially when tests have been written and need reworking. Code has inertia to being changed, so its often better to defer coding until I get a better understanding.

To get a clean abstraction and a clear understanding, while avoiding "analysis paralysis", I wrote a formal specification of these connected regions. This is essentially a mathematical model of the state of the graph and the operations on it. This kind of model enables properties of the system to be discovered before it is implemented in code. My friend and colleague, Steve Powell, was keen to review the spec and suggest several simplifications and before long, we had a formal spec with some rather nice algebraic properties for filtering and combining regions.

To give you a feel for how these properties look, take this example which says that "combining" two regions (used when describing the combined appearance of two regions) and then filtering is equivalent to filtering the two regions first and then combining the result:

Being a visual thinker and to make the formal spec more useful to non-mathematicians, I also drew plenty of pictures along the way. Here's an example graph of regions:

A New Implementation
I defined a RegionDigraph ("digraph" is short for "directed graph") interface, implemented it, and defined a suit of unit tests to give good code coverage. I then implemented a fresh collection of framework hooks in terms of the region digraph and then ripped out the old framework hooks and code supporting what in retrospect was a poorly formed notion of region membership and replaced this with the new framework hooks underpinned by the region digraph.


I Really Did Need It (IRDNI?)
It took a while to get all the kernel integration tests running again, mainly because the user region needs to be configured so that packages from the system bundle (which belongs in the kernel region) are imported along with some new services such as the region digraph service.

As problems occurred, I could step back and think in terms of the underlying graph. By writing appropriate toString methods on Region and RegionDigraph implementation classes, the model became easier to visualise in the debugger. This gives me hope that if and when other issues arise, I will have a better chance of debugging them because I can understand the underlying model.

A couple of significant issues turned up along the way, both related to the use of "side states" when Virgo deploys applications.

The first is the need to temporarily add bundle descriptions to the user region.

The second is the need to respect the region digraph when diagnosing resolver errors. This is relatively straightforward when deploying and diagnosing failures. It is less straightforward when dumping resolution failure states for offline analysis: the region digraph also needs to be dumped so it can also be used in the offline analysis.

These issues would have been much harder to address in the initial framework hooks implementation. The first issue would have involved some fairly arbitrary code to record and remove bundle descriptions from the user region. The second would have been much trickier as there was a poorly defined and overly static notion of region membership which wouldn't have lent itself to including in a state dump without looking like a gross hack. But with the region digraph it was easy to create a temporary "coregion" to contain the temporary bundle descriptions and it should be straightforward to capture the digraph alongside the state dump.

Ok, so I'm convinced that the region digraph is pulling its weight and isn't a bunch of YAGNI. But someone challenged me the other day by asking "Why do the framework hooks have to be so complex?".

Unnecessary Complexity?
Well, firstly the region digraph ensures consistent behaviour across the five framework hooks (bundle find, bundle event, service find, service event, and resolver hooks), especially regarding filtering behaviour, treatment of the system bundle, and transitive dependencies (i.e. across more than one region connection). This consistency should lead to fewer bugs, more consistent documentation, and ease of understanding for users.

Secondly, the region digraph is much more flexible than hooks based on a static notion of region membership: bundles may be added to the kernel after the user region has been created, application scoping should be relatively straightforward to rework in terms of regions thus giving scoping and regions consistent semantics (fewer bugs, better documentation etc), and multiple user regions should be  relatively tractable to implement.

Thirdly, the region digraph should be an excellent basis for implementing the notion of a multi-bundle application. In the OSGi Alliance, we are currently discussing how to standardise the multi-bundle application constructs in Virgo, Apache Aries, the Paremus Service Fabric, and elsewhere. Indeed I regard it as a proof of concept that the framework hooks can be used to implement certain basic kinds of multi-bundle application. As a nice spin-off, the development of the region digraph has resulted in several Equinox bugs being fixed and some clarifications being made to the framework hooks specification.


Next Steps
I am writing this while the region digraph is "rippling" through the Virgo repositories on its way into the 3.0 line. But this item is starting to have a broader impact. Last week I gave a presentation on the region digraph to the OSGi Alliance's Enterprise Expert Group. There was a lot of interest and subsequently there has even been discussion of whether the feature should be implemented in Equinox so that it can be reused by other projects outside Virgo.

Postscript (30 March 2010)
The region digraph is working out well in Virgo. We had to rework the function underlying the admin console because there is no longer a "surrogate" bundle representing the kernel packages and services in the user region. To better represent the connections from the user region to the kernel, the runtime artefact model inside Virgo needs to be upgraded to understand regions directly. This is work in progress in the 3.0 line.

Meanwhile, Tom Watson, an Equinox committer, is working with me to move the region digraph code to Equinox. The rationale is to ensure that multiple users of the framework hooks can co-exist (by using the region digraph API instead of directly using the framework hooks).

Tom contributed several significant changes to the digraph code in Virgo including persistence support. When Virgo  dumps a resolution failure state, it also dumps the region digraph. The dumped digraph is read back in later and used to provide a resolution hook for analysing the dumped state, which ensures consistency between the live resolver and the dumped state analysis.

"Strong" optionality

Richard Hall tells me Felix interprets optional imports as "strong", to use my earlier term. I believe there is enough latitude in the OSGi spec to permit this. Equinox takes the opposite interpretation and can discard optional imports of available packages in order to overcome uses constraints which would otherwise prevent resolution.

Regardless of that, I think that strong optionality is more likely to be what the user wants.

Without strong optionality, if the user provides a bundle which can satisfy some optional imports, they are not likely to be pleased if the resolver discards corresponding optional imports as the functions provided by the bundle will then be unavailable. Better to fail fast with diagnostics that allow the user to sort out the uses constraints.

Even worse, without strong optionality resolution may fail after a protracted attempt to discard combinations of optional imports in order to get a valid wiring. All the optional imports could be discarded before embarking upon a protracted search to ensure that the search is not in vain. Perhaps Equinox already does this, but I'm doubtful it as it could be criticised as optimising for the failure case.

OSGi resolution is NP-Complete. So what?

At last we have a proof, thanks to Robert Dunne, that the OSGi R4 resolution problem is NP-Complete. He showed that it is possible to represent a solution to the NP-Complete boolean satisfaction problem 3-SAT as a wiring of a carefully constructed set of OSGi bundles. After the idea arose of using a SAT solver to construct an OSGi resolver, some of us have been discussing the possibility of a SAT-based proof. The hard step was how to represent a boolean OR in terms of resolution, which Robert has now provided.

NP-Completeness tallies with the experience of those who have written an OSGi R4 resolver: it takes about a fortnight of intensive hacking, combined with sleepless nights, to crack the problem followed by a much longer period of bug fixing and optimisation. Now we know why: NP-Complete problems are hard to solve efficiently.

I'm partly to blame as I led the spec work on OSGi RFC 79 which added considerable function, and with it complexity, to the resolution algorithm. I presented some of the background at the 2004 OSGi Congress. The RFC 79 spec was one of the more interesting that the OSGi Core Platform Expert Group has worked on. There was a tension between addressing the basic use cases and creating something that was too hard to implement.

Fortunately, we developed a prototype resolver in the Equinox incubator. This provided useful feedback to the spec work, particularly when features were dropped from the spec which made the implementation more tractable. The team which created the prototype included Simon Burns, Steve Poole, and Tom Watson. Simon wrote the initial R4 resolver. Steve created a rigorous model-based test based on a slow, but functionally corrrect, resolver of his own. Tom provided much advice on the R3 resolver and eventually integrated the new resolver into Equinox and has been improving it ever since.

So what features emerged from this process that made the resolver NP-Complete? Well, crucial to the proof are the uses constraint and, its evil twin, optional imports. Steve Powell observes that the proof does not depend on the transitivity of the uses constraint. (Other, arguably much more evil, features in R4, require-bundle and fragments, also did not contribute to the resolver's intrinsic complexity.)

This suggests some practical ways round the problem. Examples occasionally crop up when the time to resolve a set of bundles becomes unacceptable. How can we work around such issues if we are willing to put in the effort?

Firstly, uses constraints can be avoided by preventing types from one package leaking out through the interface of another package. This is an application of the Law of Demeter.

Secondly, in some cases it may be possible to remove, or at least cut down, optional imports by using mandatory imports instead and splitting the function which has those dependencies across bundles so that the user's dependencies determine which bundles are needed.

These approaches are feasible only for new code. Bundles generated from popular Java libraries tend to exhibit both uses constraints and optional imports.

Another possibility we've been considering is a "strong" form of optionality in which the optional import cannot be discarded if it is locally satisfiable, i.e. before uses constraints are applied. Certainly the current definition of optional import is critical to Robert's proof. The challenge is to convince ourselves that R4 resolution with strong optionality in place of the current definition is no longer NP-Complete or, conversely, to prove the opposite.

I'm undecided. Strong optionality drastically reduces the search space in some cases. On the other hand, I feel there may be a proof based on 3-SAT in which multiple bundles import, but do not export, packages.

Upcoming 'webinar' on Equinox OSGi

Tom Watson and Simon Kaegi are giving an introduction to Eclipse Equinox and OSGi at noon EST on November 27th. Register if you're interested.

I hadn't come across Eclipse live before, but the archive has some OSGi podcasts which may be of interest. I guess Tom and Simon's talk will appear there in due course.

JSR 291 Reference Implementation candidate

Alex's mention of the release of Eclipse driver 3.3M5eh (Canadian for "3.3M5a") reminds me to mention that the 3.3M5 version of Equinox is the current candidate for a Reference Implementation for JSR 291.

Download it here if you want to try out any of the new features mentioned in the PR draft.

Note that Equinox hasn't changed between 3.3M5 and 3.3M5eh.

Thanks to Tom Watson for another interpretation of "eh".

Creating an OSGi bundle

Newcomers to OSGi may like a simple guide to developing a bundle. Peter Kriens has provided an extensive tutorial (December 2008: currently offline pending an overhaul) which is a must for anyone serious about learning OSGi. However, it has a long introduction and assumes you are happy to run Eclipse. So I thought I would provide a trival example that people could get going using only a Java SDK and their favourite text editor.

Here are instructions to create and run a trivial bundle, hopefully in about 10 minutes.

1. Download an OSGi framework (I used Equinox v3.2.1).

2. Paste the following code into the file org/foo/example/Example.java:

package org.foo.example;

import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;

public class Example implements BundleActivator {
public void start(BundleContext bc) {
System.out.println("Hello world");
}
public void stop(BundleContext bc) {
System.out.println("Goodbye world");  
}
}

3. Compile the code:

> javac -cp org.eclipse.osgi_3.2.1.R32x_v20060919.jar org/foo/example/Example.java

4. Paste the following bundle manifest into the file MANIFEST.MF:

Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-SymbolicName: org.foo.example.Example
Bundle-Version: 1
Bundle-Activator: org.foo.example.Example
Import-Package: org.osgi.framework;version="1.3.0"

Make sure you have a carriage return and/or newline character at the end of the last line as this is required (see manifest specification in the JAR specification).

5. Package the manifest and code into a bundle:

> jar cvfm example.jar MANIFEST.MF org

Note that the order of the flags is important - the f and m options need to appear in the same order as the jar file and manifest file parameters.

6. Run the resultant bundle example.jar interactively using the Equinox console, e.g.:

> java -jar org.eclipse.osgi_3.2.1.R32x_v20060919.jar -console

osgi> install file:///osgi/blog/example/example.jar
Bundle id is 6

osgi> start 6
Hello world

osgi> stop 6
Goodbye world

osgi> close

(close exits the framework).

That's it! Anyone care to comment how long it took them?