New Releases: Virgo 3.0 and Gemini Web 2.0

Virgo 3.0 and Gemini Web 2.0, collectively known as the Maya release, are available for download and download, respectively.

The team of committers has expanded considerably and, although the team is now distributed geographically, we have a great working relationship. I'm delighted that these projects are now community collaborations rather than the work of a single vendor.

The theme of Virgo 3.0 is better integration with EclipseRT technologies. To that end, we have created a Jetty variant of the Virgo web server and have switched from Felix to Equinox implementations of some OSGi services. Integration with p2 is proving a tough nut to crack and is deferred to the Bondi (3.5) release, although we expect to issue a milestone soon to get some feedback from the user community.

Gemini Web and the Tomcat variant of the Virgo web server have been upgraded to Tomcat 7 and Servlet 3.0.

Gemini Web 2.0 passes the OSGi Web Applications compliance tests and will replace the SpringSource OSGi Web Container as the OSGi Web Applications reference implementation.

The snaps framework for modular web applications is also released as part of Maya. This started life as the SpringSource slices prototype.

Virgo kernel 3.0 has a new shell based on Apache Felix Gogo for both the kernel and user region and accessible via telnet or ssh.

The 3.0 kernel also uses a new implementation of regions, the region digraph, based on OSGi standard framework hooks. The digraph should position Virgo well for implementing the OSGi Subsystems specification in a future release. (The specification is still under development, but we expect a public draft from the OSGi Alliance before long.) The digraph has moved from Virgo to Equinox to enable it to be used by other projects, especially those that are aiming to implement the Subsystems spec.

Highlights:

• New Virgo Jetty Server
• Virgo Tomcat Server and Gemini Web upgraded to Tomcat 7 and Servlet 3.0
• New Snaps framework for modular web applications with new guide
• New Gogo shell
• New user guide for Gemini Web
• Updated GreenPages sample with new guide

See the release notes for details.

OSGi Running in the Cloud: Virgo in CloudFoundry

I'm just been tryout out Hristo Iliev's support for Virgo in CloudFoundry, described on his blog. The VM took a few hours to download and a few minutes to unzip, but then starting CloudFoundry and deploying an application to a Virgo instance worked first time.

CloudFoundry takes a little while to get started:

But then it's trivial to deploy an application to a Virgo instance:



after which the application, in this case the Virgo web admin console, runs as expected at the application URL http://virgo.vcap.me:

and jconsole shows where the Virgo instance has been created:

The cloud nature of this setup becomes more apparent when another application is deployed. Since I'm lazy, I deployed the same application under a different name:

The application is then available at http://virgo2.vcap.me and jconsole shows there are now two Virgo instances running:

I'm looking forward to Hristo's pull requests become merged into the CloudFoundry base and maybe eventually Virgo showing up in some clouds near you.

Virgo 3.0 (Maya) Release Candidate 1

The first Release Candidate of Virgo 3.0 (Maya) is available for download. See the release notes for details.

RC1 finally brings the snaps framework for modular web applications out of prototype status. The separate snaps download includes a completely new Virgo Snaps User Guide and a comprehensive sample application. The User Guide describes how to install snaps on top of Virgo Tomcat Server and uses the sample application to verify the installation and step through all the major features of snaps.

RC1 of Virgo Tomcat Server embeds Gemini Web 2.0 RC1 which now comes with new documentation in the form of a User Guide plus a minimal Programmer Guide.

We encourage users to take RC1 for a spin and provide feedback via bugzilla or the Virgo forum. Unless critical bugs are found, we'll be focussing on the documentation until 3.0 is released, probably towards the end of August.

Exploiting p2 in EclipseRT projects

Virgo committer Borislav Kapukaranov has written an interesting blog entitled "RT meets p2". Here he gives some of the background behind the work to integrate p2 into Virgo (bug 343543) which is targetting the Azure (3.5) release of Virgo.

Virgo Maya (3.0) feature freeze at milestone 6

Milestone 6 of the Virgo Maya (3.0) release is available for download. This milestone marks the feature freeze of Maya, so we'll be working on bugs and documentation from now on. We plan to ship the release in August.

The milestone includes the new shell based on Apache Felix GoGo, accessible via telnet or ssh as described in the release notes. There are a few bugs in this support which need fixing, but apart from that the new function is in good shape for trying out.

This milestone also includes snaps for the first time. Snaps is a framework supporting modular OSGi web applications. We are bringing snaps out of prototype status in Maya, primarily by documenting its function and limitations in the next few weeks. Several groups are using snaps, so this should give them a more stable base. Including snaps in the release process will ensure it stays in step with Virgo Tomcat Server as the two evolve.

Finally, this milestone moves Virgo Jetty Server out of prototype status as the User and Programmer Guides now include appropriate documentation.

First milestone of Virgo tooling

The first milestone of the Virgo IDE tooling is available. See the wiki for instructions. It should provide approximately the same level of function as the Virgo tooling that was provided as part of SpringSource Tool Suite (STS). It works nicely on Eclipse Indigo and Helios SR2, for example.

The basic functions allow you to develop OSGi bundles, including Web Application Bundles, and PARs and deploy them to an instance of Virgo Tomcat Server (or Virgo Web Server if you prefer to run v2.1 rather than a 3.0 milestone) running under the control of the tooling. It is also possible to develop plans, but this needs documenting better as it's not immediately obvious how to do it. See an earlier blog post if you need to run the Virgo kernel instead of Virgo Tomcat Server.

Please direct discussion to the Virgo forum and raise bugs on bugzilla (thanks Daniël van 't Ooster for the bugs you raised while I was writing this blog!). The source is available in Eclipse git for anyone who wants to make changes and/or send in patches. We are keen to encourage new contributors to Virgo in general and the Virgo tooling in particular.

What next? Well, the most urgent feature, bug 341910 which is a pre-req to the first release, is to remove the dependency on Spring IDE.

Then, over time, we expect to align the Virgo tooling with the Eclipse Libra project. Libra provides tooling for OSGi standards, so we aim to move the OSGi standard features of the Virgo tooling to Libra and end up with the Virgo tooling being a Virgo-specific extension of Libra.

Congratulations to Martin Lippert who started with the STS code and single-handedly achieved this milestone! You can see him in last week's OSGi Development Tooling Panel.

Jigsaw déjà vu

I recently questioned one of the Jigsaw requirements on the jigsaw-dev mailing list. I agree with the general direction for interoperability with OSGi, but I wanted to suggest an improvement:

Hi Mark

I'm pleased to see the explicit acknowledgement of some basic OSGi interoperation requirements in the requirements document ([1]).

I agree with David Bosschaert ([2]), that it would make sense for OSGi to support the Java SE 8 module format and, for modules which can serve equally well as OSGi bundles, I'd like to avoid dual-maintenance of module metadata and OSGi manifest. I'd like to be able to "decorate" the standard metadata.

However, the requirement "The syntax must place all extended metadata after all standard metadata, with a clear delineation between them." precludes inline decorations. The result would be duplication and clunkiness.

I propose that this requirement be changed so that standard metadata could be decorated inline (the decorations would be ignored by the Java SE module system).

What do you think?

Regards,
Glyn
[1] http://openjdk.java.net/projects/jigsaw/doc/draft-java-module-system-requirements-12
[2] http://osgithoughts.blogspot.com/2011/05/java-se-8-modularity-requirements.html

The response: zilch!

After a follow-up, I got a single supportive response, but that was from OSGi aficionado Peter Kriens.

I would have preferred some defence of the current requirement rather than silence. This is awfully like the days of JSRs 277 and 294. Plus ça change, plus c'est la même chose...

Virgo 3.0 plan update

The feature to provision Virgo using p2 is taking a while to finalise. Also, it changes the directory layout of Virgo and so will need some changes to the tooling. So we've decided to defer the feature to a later release, possibly 3.5 to indicate the relatively big jump from 3.0.

This means the 3.0 release content is pretty much complete, so we just have to fix a few remaining bugs and polish the documentation before releasing, probably in July or August 2011.

See the plan on the Virgo wiki for details.

Eclipse Virgo 3.0.0 Milestone 5

Milestone 5 of Virgo 3.0.0 is available for download. This is an important milestone which adds significant functional enhancements, upgrades several dependencies to their latest levels including Spring 3.0.5, Tomcat 7.0.12, and Servlet 3.0, and fixes a number of bugs. Full details are available in the release notes.

Equinox digraph ready for use

Previously I described how Virgo used OSGi framework hooks to implement a directed graph, or digraph, of isolated groups of bundles known as regions. The region digraph has now moved to Equinox and I'm modifying Virgo to use the Equinox region digraph in place of Virgo's own (see bug 343364 for details).

I always prefer making such changes in an incremental way, so the first step was to add the Equinox region digraph bundle to the Virgo kernel without exploiting it. The Virgo kernel runs fine with the Equinox region digraph in situ. This is because the Equinox digraph adds all bundles to its kernel region and so provides no isolation between those bundles. Later, Virgo's region digraph partitions the bundles in Virgo kernel and Virgo user regions with appropriate isolation and things continue to work as normal.

This is interesting, but not particularly useful. The whole point of the region digraph abstraction was to make it easy for multiple parties wishing to provide isolation via the OSGi framework hooks to be able to collaborate together by sharing and manipulating a single region digraph. It was never intended that multiple digraphs should co-exist.

But there is a useful point here. If you have an Equinox-based system, of which there are many, you can introduce the Equinox digraph with zero functional impact. All your bundles, including those installed before and after the Equinox region bundle, go in the Equinox kernel region and there is no isolation between those bundles.

When you want to start exploiting the digraph, you can use the RegionDigraph service to create a new region, attach it with appropriate filters, if any, to and from the kernel region, and then install a bundle into the new region using the Region.installBundle operation. After starting this bundle, you will then have a bundle context that can be used to install further bundles into the new region using the OSGi standard BundleContext API.

I hope this will inspire you to have a go at using the Equinox region digraph if you are an Equinox user and need to isolate groups of bundles.

Meanwhile, the standardisation of multi-bundle applications, including isolation, continues in the OSGi Alliance and a public draft of this spec should be available in the not too distant future. I hope the Equinox region digraph will be used more widely and this will help to provide feedback on the spec work.

Opening up the Enterprise Bundle Repository

We'd like to enable the SpringSource Enterprise Bundle Repository to turn into a shared community resource.

As a first, small step in that direction, I've made available the EBR build system (a few Ant/Ivy scripts) and some sample bundle projects on github. If you want to try it out, clone the repository and refer to doc/README.txt.

The idea is that the community will be able to add bundles to this repository and then the content can be hosted in some cloud storage similar to the way the SpringSource EBR is hosted in Amazon S3.

Virgo survey results

Thanks to everyone who responded to the recent Eclipse Virgo and SpringSource dm Server survey. The results make interesting reading.

Close to one quarter of respondents report they are "in production".

The transfer to Eclipse.org is a success. It is 6 months since the first Virgo release and 84% of respondents are now using Virgo exclusively with only 13% using dm Server exclusively.

Virgo is popular with early adopters. Of the 84% using Virgo exclusively, 59% are using 2.1.x or 3.0 milestones and 25% are on 3.0 milestones only (some of these already in production!). Please note that Virgo has versions 2.1.0 and greater whereas dm Server covers versions 1.x and 2.0.x.

The Virgo kernel is seeing significant usage. The kernel underpins the Virgo Web Server and provides Virgo's advanced deployment, provisioning, isolation, scoping, and diagnostic capabilities. The kernel has no web-specific code and so can be used to build messaging and other non-HTTP servers and to act as a container for non-server applications. 16% of respondents are using the kernel exclusively; 5% are in production.

Documentation and blogs were the most useful information sources closely followed by the Virgo forum, Virgo wiki, and Virgo development mailing list (1= not at all useful, 5 = very useful).

The survey was advertised on Twitter, the Virgo forum, the Virgo development mailing list, at www.springsource.org, and in this blog. 166 people responded. If you are interested, the raw results of the survey are available online or a summary can be downloaded.

Finally, I'd like to thank Chris Frost for coming up with the idea for the survey and then delivering the goods.

OSGi comes of age?

Early in my career at SpringSource, my colleagues presented me a T-shirt with "CICS does that!" printed on the front. However, nobody anticipated what that statement might eventually include.

IBM has just announced that the latest version of the CICS transaction monitor includes support for applications packaged as OSGi bundles. From the announcement letter:

The JVM server infrastructure with integrated OSGi support and tooling provides a new industry standard Java runtime environment in CICS.

[...] 

The JVM server environment supports the development and deployment of Java applications through the use of industry standard OSGi bundles. This will require Java applications to be packaged into an OSGi bundle, and then deployed as a CICS BUNDLE resource with reference to the OSGi bundle and its version, removing the need to load Java applications from a statically defined class path. This function will significantly improve the system management of Java applications in CICS through the provision of application isolation and versioning, cross-package prerequisite checking, and simplified package redeployment.

This is heart-warming news to me, both as an OSGi proponent and as a former member of the CICS development team. I'm guessing CICS is using Equinox as its OSGi framework, but I'm looking forward to reading the documentation for the details.

 

Java update breaks jps, jconsole, etc

The latest update of Java broke jps and jconsole when Java is launched specifying "-Djava.io.tmpdir" with a value of anything other than $TMPDIR. This affects Virgo and Tomcat.

The problem was introduced in Java 6 update 21 and is fixed by bug sunbug 7009828 in Java 6 update 25.

A workaround for Virgo is as follows. On Mac and Linux to delete the following line in dmk.sh:

-Djava.io.tmpdir=$TMP_DIR \

and on Windows to delete the following line in dmk.bat:

set KERNEL_JAVA_PARMS=%KERNEL_JAVA_PARMS% -Djava.io.tmpdir="%TMP_DIR%" 

Eclipse Virgo diagnostics for "uses" conflicts

Congratulations to Felix on producing some really nice diagnostics for uses conflicts, and thanks to Neil Bartlett for his blog on the subject.

I thought I'd try a similar example in Virgo for comparison. Here are Virgo's diagnostics:

Unable to satisfy dependencies of bundle 'org.example.foo.xydata' at version '0.0.0': Cannot resolve: org.example.foo.xydata
    Resolver report:
        Uses violation: ⟨Import-Package: org.example.foo.bar.datastore; version="0.0.0"⟩ in bundle ⟨org.example.foo.xydata_0.0.0[1301563814803]⟩
            Found conflicts:
                package        'org.example.foo.util.verify_1.0.0' in bundle 'org.example.foo.platform.util_0.0.0[1301563814806]' used by 'org.example.foo.bar.datastore_0.0.0' in bundle 'org.example.foo.bar.datastore_0.0.0[1301563814805]'
                conflicts with 'org.example.foo.util.verify_2.0.0' in bundle 'org.example.foo.framework_0.0.0[1301563814804]' imported by bundle 'org.example.foo.xydata_0.0.0[1301563814803]'

(The numbers in square brackets are bundle ids - they are enormous because Virgo is attempting resolution in a "side state". Bundles with enormous ids have either been deployed directly or auto-installed from the Virgo repository and are in the process of being resolved. They are committed to the OSGi framework only if the directly deployed bundle(s) can be resolved successfully.)

The Virgo formatting isn't quite as pretty as Felix's, but it looks reasonable on a wide terminal window. However, there are some subtle differences:

• Felix dumps out the dependency chains whereas Virgo does not attempt that. We had in mind examples where the dependency chains are unmanageably long, i.e. those where good diagnostics really are essential. But having seen Felix's diagnostics, we would like to dump out the chains in simple cases. On the other hand, that area of code is pretty complex, so this is not something we'll be rushing to do.
• Felix depicts wirings from import to export which could be especially useful for newcomers to OSGi.
• The Felix diagnostics are built in to the framework whereas non-Virgo users of Equinox do not get the Virgo diagnostics.
• The Virgo diagnostics highlight the fact that there is a uses violation whereas Felix leaves the user to infer that from the rest of the message. It would be nice if Felix was more explicit because there are other reasons a bundle cannot be resolved.
• Felix highlights the duplicated package whereas Virgo highlights the import that encountered the uses constraint. Virgo is designed to cope well when there are multiple duplicated packages. It's not clear what the Felix diagnostics would look like in such cases.

Note: on a resolution failure Virgo dumps the "side state" to disk for offline analysis. With this example, the offline analysis produced poorer diagnostics than were printed to the console, so I raised bug 341460 to investigate. The beauty of offline analysis is it lets you explore the details of the bundles. For example, the following screenshots shows the details of the duplicated package, including the import version ranges which result in the attempt to use the distinct versions of the duplicated package:

Anyway, I'm really pleased that we are starting to see some stiff competition on diagnostics. This will help OSGi users enormously.

Postscript
Bug 341460 is fixed in the Virgo 3.0 line. The admin console now displays the same diagnostics that were printed to the console.

Enterprise OSGi Runtimes and Tooling

It was great to see several enterprise OSGi runtimes in action at EclipseCon.

Readers of this blog will be familiar with Eclipse Virgo, so suffice to say that tutorials covered developing  applications using Eclipse Gemini components (reference implementations of several OSGi Enterprise standards) in Virgo and configuring and managing web applications in Virgo.

Several of the tutorials developed Web Application Bundles, or WABs, which are part of the OSGi Enterprise standard.  A Web Application Bundle is like a WAR file which specifies the web context path in its manifest and which obtains its dependencies using OSGi instead of having to include them all in WEB-INF/lib. The tutorials also implemented persistence using the OSGi Enterprise standard mapping of Java Persistence API (JPA).

Apache Aries provides a collection of components including implementations of several Enterprise OSGi standards and an application model similar to Virgo's. The EclipseCon tutorial had us creating a blogging application using the free IBM® Rational® tooling. The application consisted of an Aries application containing a plain OSGi bundle, a bundle utilising JPA persistence, and a WAB. The JPA bundle used Aries extensions to the standard blueprint service. The WAB used JNDI to look up the service published by the persistence bundle - the same approach that was taken in the Gemini and Virgo tutorial. We then deployed the application to a target platform which included Aries bundles.

Apache Karaf is similar in concept to the Virgo kernel. Karaf has several functions that Virgo lacks, some of which are good candidates to add to Virgo. Apache Geronimo is based on Karaf.

GlassFish now has good support for OSGi. The EclipseCon tutorial started with us developing a basic client bundle calling a service published by a back-end bundle. We then replaced the client bundle with a WAB and the back-end bundle by a JPA persistence bundle which published a service in the form of an EJB. The WAB used custom annotations to inject the service into the WAB's servlet. The EJB was used to demarcate transactions and security. The M2Eclipse "core" and "extras" plugins provided a basic Eclipse development environment, although the bundles were deployed via the command line. Kudos to the GlassFish team, including Arun Gupta and Sanjeeb Sahoo who led the tutorial, for possibly the most polished hands-on tutorial I've experienced.

I was also very pleased to see the Eclipse Libra project making progress. The version destined for inclusion in this year's Eclipse release train can be used to develop WABs. Runtime launching support is in the pipeline. We also hope to migrate standards based features from the Virgo tooling into Libra.

git tip: listing or editing the commits on a branch

Sometimes I need to list all the commits on a branch. I found the following command does the trick, even after I have merged from master into the branch one or more times:

git rev-list my-branch ^master

This lists all the commits reachable from my-branch but not reachable from master.

You can also change the committer or author info of commits on a branch by specifying  my-branch ^master as the on git filter-branch (see git help filter-branch). 

Quick survey of Virgo and SpringSource dm Server usage

I would be grateful if users of Virgo and SpringSource dm Server would take a short, anonymous survey to help us in our planning. Should take no more than 2 minutes to complete.

Correction: Oracle do not contribute to Virgo

A recent Oracle blog stated:

Also presented were Eclipse projects, that Oracle engineers are contributing to: Gemini, Virgo, Sapphire and Web Tool Platform.

I submitted a comment on the blog pointing out that Oracle engineers contribute to Gemini but not Virgo. Unfortunately, the comment was not published and the blog was not corrected so perhaps this blog will help to set the record straight.

That said, Virgo would welcome contributions from Oracle (and others). We've enjoyed collaborating with Oracle in the Gemini project and we really appreciate having Virgo committers from three different companies and a few more would be even better.

Incremental OSGi with Virgo, Gemini, and Libra

As I anticipate the start of EclipseCon 2011 later today, I'm reflecting on the current state of the Eclipse Virgo, Gemini, and Libra projects.

As I'll be discussing on Tuesday, Virgo evolved out of a relatively mature project whose aim was to make it easy for enterprise applications and enterprise application developers to adopt OSGi. The goal was to make the transition to OSGi incremental. Essentially you can start with a standard WAR file and deploy it to Virgo as-is. Then you can incrementally refactor the WAR file into an application comprising multiple OSGi bundles while continuing to use familiar technologies such as Spring, Hibernate, JPA, etc. We based Virgo on Tomcat suitably embedded into OSGi, again to provide familiarity for enterprise developers as well as systems administrators.

In parallel with the evolution of Virgo, the OSGi Alliance Enterprise Expert Group, being discussed on Tuesday, produced a series of enterprise specifications again with the goal of enabling enterprise applications to be migrate straightforwardly to OSGi. The reference implementations of these specifications became today's Eclipse Gemini project:

• Gemini Blueprint - an OSGi standard dependency injection container which also supports the popular Spring and Spring DM namespaces
• Gemini Web - support for servlets in the form of OSGi standard Web Application Bundles (WABs) or standard WAR files
• Gemini JPA - support for JPA persistence in OSGi
• Gemini DBAccess - modularised JDBC drivers for use in OSGi
• Gemini Management - JMX management for OSGi
• Gemini Naming - support for JNDI naming in OSGi

Until recently, integrating the Gemini projects into Virgo has been a fairly specialised skill. As we'll see in a workshop on Thursday, things are now really starting to come together and Gemini Blueprint, Gemini Web, and various other Gemini components can now be deployed and used in Virgo. We'll also be touching on how to manage and configure WABs in Virgo as part of a workshop later this morning.

So much for the past. What about the future? The primary goal of Virgo 3.0 is better integration with other EclipseRT technologies such as Equinox, Jetty, and p2. This will provide richer function and greater choice for enterprise applications as they migrate to OSGi. We'll be hearing about how Virgo and EclipseRT technologies work together on Wednesday. The recently released third milestone of this release shows the progress towards the goal of better integration with EclipseRT:

• Virgo now uses Equinox Configuration Admin and Event Admin services instead of their Apache equivalents.
• Virgo has moved to the new framework hooks model for isolation in Equinox which is part of the OSGi 4.3 standard soon to be issued for public review.
• Virgo now provides two web servers: Virgo Tomcat Server, which continues to be based on Gemini Web, and Virgo Jetty Server. Some exciting technology for modularising the view portion of web applications is also emerging into the Virgo mainstream as we'll hear on Wednesday.
• p2 support in Virgo is also making good progress although this is currently in prototype form.

Development tooling is also entering an exciting new phase which will greatly help applications migrate to enterprise OSGi:

• The Libra project, which will be presented on Wednesday, is providing development tooling for building WABs and will ship in the Eclipse Indigo release train later this year.
• The Virgo development tooling has entered the Eclipse IP process and will soon emerge in the Virgo project. Over time, the standards-based features of the Virgo tooling will be factored out and migrated into Libra.
• OSGi runtime launchers are also being contributed to Libra. I would hope to see a Virgo launcher for Libra emerge so that Libra can start to take over from the Virgo tooling for many aspects of enterprise OSGi application development.

If you're attending EclipesCon, come and hear the talks, join in the workshops and BoFs and help to shape incremental OSGi in the Virgo, Gemini, and Libra projects.