Virgo 3.0.2 released

Virgo 3.0.2 is available for download. Please see the release notes for details of this bug fix release.

Opening SourceTree from the command line

I've recently started using SourceTree to visualise and manage my git repos. It's very impressive.

The one thing I missed from gitx was the ability to launch SourceTree from the command line, in any directory of the repo. One solution is to set a git alias:

git config --global --add alias.sourcetree '!open -a SourceTree .'

and add an alias in the shell profile:

alias st='git sourcetree'

Now I can issue "st" from the command line and the relevant SourceTree window opens up.

(Thanks to Chris Frost for providing part of this solution.)

What did we learn in 100 sprints?

This week marks the 100th Sprint on Eclipse Virgo! That includes the precursor project, SpringSource dm Server. What have we learned about scrum? Well, obviously, it's pretty good otherwise we would have ditched it long ago (and believe me I've been tempted a couple of times). But specifically...


Firstly, that one week sprints are by far the easiest to manage, both in terms of planning effort, but also psychologically. Even in a two week sprint, it's easy to put off unpleasant tasks until it's too late.


Secondly, that the main value of scrum is to keep the team focussed on specific tasks during a sprint and enable progress to be shared easily at daily stand-up meetings.


Thirdly, that story points are more effort than they are worth, so it's more effective to go with gut feel.


Fourthly, that you have to take a relaxed attitude towards planning design work as it's almost impossible to predict the effort up front. Spikes are ok for small items, but sometimes you just need plenty of time to get your head round a large area.

Enterprise Bundle Repository Samples

After opening up the Enterprise Bundle Repository build system, I've added more sample bundlor templates along with Ant/Ivy build files to github.

So while we are trying to come up with a community-based successor to the EBR, if you need to get a new version of one of the bundles in the EBR, you may be able to find a bundlor template to use as a starting point.

Latest draft subsystems spec

The OSGi Alliance has made an early draft of some RFCs available. This includes the subsystems RFC 152 I blogged about earlier: see pages 72-144 of the PDF.

Eclipse Dance Steps

Sometime Eclipse fails to work as expected. The following 'dances', in increasing order of desperation, may help. Remember to maintain a fixed grin throughout and try not to tread on your own toes.

Refresh and clean all projects ("Quickstep")

Click on a project in the Package Explorer. Hit Apple-A (or Ctrl-A) to select all projects. Hit F5 to refresh. Select Project->Clean..., select "Clean all projects", and click Ok.

Open and close all projects ("Foxtrot")

Click on a project in the Package Explorer. Hit Apple-A (or Ctrl-A) to select all projects. Right click the projects and select "Close Project". Right click the projects again and select "Open Project".

Open and close all projects restarting Eclipse in between ("Tango")

Click on a project in the Package Explorer. Hit Apple-A (or Ctrl-A) to select all projects. Right click the projects and select "Close Project". Restart Eclipse. Click on a project in the Package Explorer. Hit Apple-A (or Ctrl-A) to select all projects. Right click the projects again and select "Open Project".

Delete and re-import all projects ("Paso Doble")

Click on a project in the Package Explorer. Hit Apple-A (or Ctrl-A) to select all projects. Press the delete key, select "Do not delete contents", and click Ok. Right click in the Package Explorer and select Import..., then choose General and "Existing Projects into Workspace" and click Next. Browse to select the platform checkout directory and click Choose. Select all the projects for import. Click Finish.

Rebuild the search indices ("Slow Waltz")

Exit the workbench and delete the Java search index:

• go into \plugins\.metadata\.plugins\org.eclipse.jdt.core
• delete 'savedIndexNames.txt'
• delete all *.index
  Restart

If all else fails

If you're still stuck, try the "Hokey Cokey" by repeating all of the above until you've had enough.

How Infor ION Uses Virgo

Infor have kindly made available a presentation on how their ION integration product is based on the Virgo kernel. They migrated from Equinox 3.5 to Virgo 2.1, added a lightweight web server (in preference to using the pre-built Virgo Tomcat or Jetty servers) and a service wrapper, added Spring integration, upgraded Spring framework, added a watched repository, and then added various enterprise service bundles. This is a perfect example of a Virgo kernel application, a so-called OSGi stackless stack.

The Infor team also helpfully pointed out some enhancements they'd like to see, some of which are already supported or planned for Virgo 3.5.

According to their website Infor is the world's third largest provider of enterprise applications and services, so it's nice to see another large company adopt Virgo.

Thanks to Ian Skerrett for bringing this user to my attention. See his blog for more.

Extenders: Pattern or Anti-Pattern?

The extender pattern has become popular in recent years and has even been utilised in OSGi standards such as the Blueprint service and the Web Applications specification. In Virgo, we've been working with extenders from the start, but in spite of their advantages, they have some significant downsides. Since the OSGi Alliance is considering using extenders in other specifications, I agreed to document some of the issues.

The first difficulty is knowing when an extender has finished processing a bundle. For example, a bundle containing a blueprint XML file will transition to ACTIVE state as soon as any bundle activator has been driven. But that's not the whole story. Administrators are interested in when the bundle is ready for use and so the management code in Virgo tracks the progress of the extender and presents an amalgamated state for the install artefact representing the bundle. The install artefact stays in STARTING state until the application context has been published, at which point it transitions to ACTIVE. Without such additional infrastructure, administrator cannot tell when a bundle processed by an extender really is ready for business.

That's the successful case, but there are complications in error cases too. The first complication is that since an extender runs in a separate thread to that which installed the bundle, if the extender throws an exception, this is not propagated to the code which installed the bundle. So the installer needs somehow to check for errors. Therefore Virgo has infrastructure to detect such errors and propagate them back to the thread which initiated deployment of the bundle: the deployment operation fails with a stack trace indicating what went wrong.

The other error complication is where there is a (possibly indefinite) delay in an extender processing a bundle. For this kind of error Virgo tracks the progress of extender processing and issues warnings to the event log (intended for the administrator's eyes) saying which pieces of processing have been delayed and in some common situations, for example when a blueprint is waiting for a dependency, what is causing the delay.

Extenders suffer from needing to be able to see bundle lifecycle events and so for systems that partition the framework, it is necessary to install each extender into multiple partitions. On the flip side it is crucial to prevent multiple instances of an extender from ever seeing the same bundle event otherwise they will both attempt to extend the bundle.

Another issue with extenders is the need to keep them running and healthy as there is little indication that an extender is down or sickly other than bundles not being processed by the extender. Virgo takes care to ensure its extenders are correctly started and its infrastructure for detecting delays helps to diagnose extender crashes or sickness (both of which are extremely rare situations).

There is also an issue in passing parameters to an extender to affect its behaviour. This is typically done by embedding extender configuration in the bundles being processed or by attaching a fragment containing configuration to the extender bundle. But since the extender is not driven by an API, the normal approach of passing parameters on a call is not available. Essentially, an extender model implies that the programming model for deployment is restricted to BundleContext.installBundle.

With considerable investment in additional infrastructure, Virgo has managed to support the Blueprint and Spring DM extenders reasonably well. But in the case of the Web Applications extender, Virgo couldn't make this sufficiently robust and so it drives the underlying web componentry directly from the Virgo deployment pipeline to avoid the above issues.

I understand at least one other server runtime project has encountered similar issues with extenders, so Virgo is not alone. There is a trade-off between loosely coupling the installer from the resource-specific processing, the main strength of the extender pattern (but far from unique to that pattern), and providing a robust programming model and usable management view -- crucial features of a server runtime -- which is far more straightforward without extenders.

OSGi Subsystems in Virgo

A public draft(*) of the OSGi subsystems RFC (152) should soon emerge from the OSGi Alliance. A subsystem is a multi-bundle application, not dissimilar to a PAR or plan in Virgo. IBM is leading the spec work and a number of other vendors, including SpringSource/VMware, are contributing. Quite a few projects have multi-bundle application constructs, so it makes sense to agree a standard form.

After going through the RFC again with my implementer's hat on, I listed the features necessary to support subsystems in Virgo. Most of the changes are in the Virgo kernel, although I hope to structure the support into non-subsystem specific generalisations of the kernel plus subsystem-specific code running in the user region.

Currently, it is not possible to deploy a plan which contain artefacts that are already deployed. This will need generalising so that we can support subsystems using a common data structure of deployed artefacts: a directed acyclic graph (DAG) rather than the current collection of trees.

The switch to a DAG has interesting implications for lifecycle management of shared subgraphs. With today's tree, when a node is stopped, started, or uninstalled, any subtrees are also stopped, started, or uninstalled, respectively. With a DAG, shared subgraphs need to be sensitive to all their parents. This boils down to keeping each shared subgraph at the maximum state required by any of its parents. States are ordered: ACTIVE > RESOLVED > UNINSTALLED.

Lifecycle management will also get interesting when a shared subgraph belongs to one or more atomic subgraphs as then lifecycle changes in the common subgraph will propagate to all the containing atomic subgraphs. I think that will just "work", but users might need to be careful in their use of atomic plans if they want to avoid management operations on one application affecting other applications.

I'm also considering using garbage collection as a means of uninstalling artefacts which are no longer needed. Given the number of types of dependency that are possible, this is likely to be more reliable than the alternative of maintaining reference counts.

I'm a little concerned about a possible race between garbage collection detecting an artefact as dead and a new dependency being created on the artefact just before garbage collection goes ahead and uninstalls the artefact, but there would be a similar concern for reference counting. The basic issue in this race is that a dead artefact may be found by a live bundle and a new dependency created before the dead artefact can be uninstalled. For instance, a dead bundle may be found by using the OSGi API to list all bundles. It may be possible to use some technique such as a special region in the region digraph to isolate dead bundles, although this issue is probably something to discuss among those working on the RFC.

Anyway, there's plenty of work to be getting on with. I haven't done detailed estimates of the features identified so far, but I guess there's a person year or so of effort needed, so I'm initially targeting Virgo 4.0. If you feel like lending a hand, please get in touch on virgo-dev.

* - RFC 152 has changed quite a bit since the version in the Enterprise spec early access draft dated 16 May 2011. A new draft is being prepared.

New Releases: Virgo 3.0 and Gemini Web 2.0

Virgo 3.0 and Gemini Web 2.0, collectively known as the Maya release, are available for download and download, respectively.

The team of committers has expanded considerably and, although the team is now distributed geographically, we have a great working relationship. I'm delighted that these projects are now community collaborations rather than the work of a single vendor.

The theme of Virgo 3.0 is better integration with EclipseRT technologies. To that end, we have created a Jetty variant of the Virgo web server and have switched from Felix to Equinox implementations of some OSGi services. Integration with p2 is proving a tough nut to crack and is deferred to the Bondi (3.5) release, although we expect to issue a milestone soon to get some feedback from the user community.

Gemini Web and the Tomcat variant of the Virgo web server have been upgraded to Tomcat 7 and Servlet 3.0.

Gemini Web 2.0 passes the OSGi Web Applications compliance tests and will replace the SpringSource OSGi Web Container as the OSGi Web Applications reference implementation.

The snaps framework for modular web applications is also released as part of Maya. This started life as the SpringSource slices prototype.

Virgo kernel 3.0 has a new shell based on Apache Felix Gogo for both the kernel and user region and accessible via telnet or ssh.

The 3.0 kernel also uses a new implementation of regions, the region digraph, based on OSGi standard framework hooks. The digraph should position Virgo well for implementing the OSGi Subsystems specification in a future release. (The specification is still under development, but we expect a public draft from the OSGi Alliance before long.) The digraph has moved from Virgo to Equinox to enable it to be used by other projects, especially those that are aiming to implement the Subsystems spec.

Highlights:

• New Virgo Jetty Server
• Virgo Tomcat Server and Gemini Web upgraded to Tomcat 7 and Servlet 3.0
• New Snaps framework for modular web applications with new guide
• New Gogo shell
• New user guide for Gemini Web
• Updated GreenPages sample with new guide

See the release notes for details.

OSGi Running in the Cloud: Virgo in CloudFoundry

I'm just been tryout out Hristo Iliev's support for Virgo in CloudFoundry, described on his blog. The VM took a few hours to download and a few minutes to unzip, but then starting CloudFoundry and deploying an application to a Virgo instance worked first time.

CloudFoundry takes a little while to get started:

But then it's trivial to deploy an application to a Virgo instance:



after which the application, in this case the Virgo web admin console, runs as expected at the application URL http://virgo.vcap.me:

and jconsole shows where the Virgo instance has been created:

The cloud nature of this setup becomes more apparent when another application is deployed. Since I'm lazy, I deployed the same application under a different name:

The application is then available at http://virgo2.vcap.me and jconsole shows there are now two Virgo instances running:

I'm looking forward to Hristo's pull requests become merged into the CloudFoundry base and maybe eventually Virgo showing up in some clouds near you.

Virgo 3.0 (Maya) Release Candidate 1

The first Release Candidate of Virgo 3.0 (Maya) is available for download. See the release notes for details.

RC1 finally brings the snaps framework for modular web applications out of prototype status. The separate snaps download includes a completely new Virgo Snaps User Guide and a comprehensive sample application. The User Guide describes how to install snaps on top of Virgo Tomcat Server and uses the sample application to verify the installation and step through all the major features of snaps.

RC1 of Virgo Tomcat Server embeds Gemini Web 2.0 RC1 which now comes with new documentation in the form of a User Guide plus a minimal Programmer Guide.

We encourage users to take RC1 for a spin and provide feedback via bugzilla or the Virgo forum. Unless critical bugs are found, we'll be focussing on the documentation until 3.0 is released, probably towards the end of August.

Exploiting p2 in EclipseRT projects

Virgo committer Borislav Kapukaranov has written an interesting blog entitled "RT meets p2". Here he gives some of the background behind the work to integrate p2 into Virgo (bug 343543) which is targetting the Azure (3.5) release of Virgo.

Virgo Maya (3.0) feature freeze at milestone 6

Milestone 6 of the Virgo Maya (3.0) release is available for download. This milestone marks the feature freeze of Maya, so we'll be working on bugs and documentation from now on. We plan to ship the release in August.

The milestone includes the new shell based on Apache Felix GoGo, accessible via telnet or ssh as described in the release notes. There are a few bugs in this support which need fixing, but apart from that the new function is in good shape for trying out.

This milestone also includes snaps for the first time. Snaps is a framework supporting modular OSGi web applications. We are bringing snaps out of prototype status in Maya, primarily by documenting its function and limitations in the next few weeks. Several groups are using snaps, so this should give them a more stable base. Including snaps in the release process will ensure it stays in step with Virgo Tomcat Server as the two evolve.

Finally, this milestone moves Virgo Jetty Server out of prototype status as the User and Programmer Guides now include appropriate documentation.

First milestone of Virgo tooling

The first milestone of the Virgo IDE tooling is available. See the wiki for instructions. It should provide approximately the same level of function as the Virgo tooling that was provided as part of SpringSource Tool Suite (STS). It works nicely on Eclipse Indigo and Helios SR2, for example.

The basic functions allow you to develop OSGi bundles, including Web Application Bundles, and PARs and deploy them to an instance of Virgo Tomcat Server (or Virgo Web Server if you prefer to run v2.1 rather than a 3.0 milestone) running under the control of the tooling. It is also possible to develop plans, but this needs documenting better as it's not immediately obvious how to do it. See an earlier blog post if you need to run the Virgo kernel instead of Virgo Tomcat Server.

Please direct discussion to the Virgo forum and raise bugs on bugzilla (thanks Daniël van 't Ooster for the bugs you raised while I was writing this blog!). The source is available in Eclipse git for anyone who wants to make changes and/or send in patches. We are keen to encourage new contributors to Virgo in general and the Virgo tooling in particular.

What next? Well, the most urgent feature, bug 341910 which is a pre-req to the first release, is to remove the dependency on Spring IDE.

Then, over time, we expect to align the Virgo tooling with the Eclipse Libra project. Libra provides tooling for OSGi standards, so we aim to move the OSGi standard features of the Virgo tooling to Libra and end up with the Virgo tooling being a Virgo-specific extension of Libra.

Congratulations to Martin Lippert who started with the STS code and single-handedly achieved this milestone! You can see him in last week's OSGi Development Tooling Panel.

Jigsaw déjà vu

I recently questioned one of the Jigsaw requirements on the jigsaw-dev mailing list. I agree with the general direction for interoperability with OSGi, but I wanted to suggest an improvement:

Hi Mark

I'm pleased to see the explicit acknowledgement of some basic OSGi interoperation requirements in the requirements document ([1]).

I agree with David Bosschaert ([2]), that it would make sense for OSGi to support the Java SE 8 module format and, for modules which can serve equally well as OSGi bundles, I'd like to avoid dual-maintenance of module metadata and OSGi manifest. I'd like to be able to "decorate" the standard metadata.

However, the requirement "The syntax must place all extended metadata after all standard metadata, with a clear delineation between them." precludes inline decorations. The result would be duplication and clunkiness.

I propose that this requirement be changed so that standard metadata could be decorated inline (the decorations would be ignored by the Java SE module system).

What do you think?

Regards,
Glyn
[1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12
[2] http://osgithoughts.blogspot.com/2011/05/java-se-8-modularity-requirements.html

The response: zilch!

After a follow-up, I got a single supportive response, but that was from OSGi aficionado Peter Kriens.

I would have preferred some defence of the current requirement rather than silence. This is awfully like the days of JSRs 277 and 294. Plus ça change, plus c'est la même chose...

Virgo 3.0 plan update

The feature to provision Virgo using p2 is taking a while to finalise. Also, it changes the directory layout of Virgo and so will need some changes to the tooling. So we've decided to defer the feature to a later release, possibly 3.5 to indicate the relatively big jump from 3.0.

This means the 3.0 release content is pretty much complete, so we just have to fix a few remaining bugs and polish the documentation before releasing, probably in July or August 2011.

See the plan on the Virgo wiki for details.

Eclipse Virgo 3.0.0 Milestone 5

Milestone 5 of Virgo 3.0.0 is available for download. This is an important milestone which adds significant functional enhancements, upgrades several dependencies to their latest levels including Spring 3.0.5, Tomcat 7.0.12, and Servlet 3.0, and fixes a number of bugs. Full details are available in the release notes.

Equinox digraph ready for use

Previously I described how Virgo used OSGi framework hooks to implement a directed graph, or digraph, of isolated groups of bundles known as regions. The region digraph has now moved to Equinox and I'm modifying Virgo to use the Equinox region digraph in place of Virgo's own (see bug 343364 for details).

I always prefer making such changes in an incremental way, so the first step was to add the Equinox region digraph bundle to the Virgo kernel without exploiting it. The Virgo kernel runs fine with the Equinox region digraph in situ. This is because the Equinox digraph adds all bundles to its kernel region and so provides no isolation between those bundles. Later, Virgo's region digraph partitions the bundles in Virgo kernel and Virgo user regions with appropriate isolation and things continue to work as normal.

This is interesting, but not particularly useful. The whole point of the region digraph abstraction was to make it easy for multiple parties wishing to provide isolation via the OSGi framework hooks to be able to collaborate together by sharing and manipulating a single region digraph. It was never intended that multiple digraphs should co-exist.

But there is a useful point here. If you have an Equinox-based system, of which there are many, you can introduce the Equinox digraph with zero functional impact. All your bundles, including those installed before and after the Equinox region bundle, go in the Equinox kernel region and there is no isolation between those bundles.

When you want to start exploiting the digraph, you can use the RegionDigraph service to create a new region, attach it with appropriate filters, if any, to and from the kernel region, and then install a bundle into the new region using the Region.installBundle operation. After starting this bundle, you will then have a bundle context that can be used to install further bundles into the new region using the OSGi standard BundleContext API.

I hope this will inspire you to have a go at using the Equinox region digraph if you are an Equinox user and need to isolate groups of bundles.

Meanwhile, the standardisation of multi-bundle applications, including isolation, continues in the OSGi Alliance and a public draft of this spec should be available in the not too distant future. I hope the Equinox region digraph will be used more widely and this will help to provide feedback on the spec work.

Opening up the Enterprise Bundle Repository

We'd like to enable the SpringSource Enterprise Bundle Repository to turn into a shared community resource.

As a first, small step in that direction, I've made available the EBR build system (a few Ant/Ivy scripts) and some sample bundle projects on github. If you want to try it out, clone the repository and refer to doc/README.txt.

The idea is that the community will be able to add bundles to this repository and then the content can be hosted in some cloud storage similar to the way the SpringSource EBR is hosted in Amazon S3.