How to develop with the Virgo Kernel in Eclipse

A number of groups are discovering that the Virgo Kernel is a really useful OSGi runtime. For instance the Rich Ajax Platform project have documented how to run RAP on the kernel.

Others have steered clear of the kernel because the Eclipse-based Virgo tooling does not support the kernel directly (see bug 331101). However, it's really easy to get the tooling working with the kernel. Here's how...

First, set up the Virgo tooling in Eclipse as described in the Tooling chapter of the Virgo Programmer Guide.

However, when you create a new Virgo Web Server server runtime environment and point at a Virgo kernel installation, you'll see the following error:

Edit lib/.version in the kernel installation to contain the key 'virgo.server.version' instead of 'virgo.kernel.version'. Then the above error goes away and you can continue to set up the runtime environment and server as per the instructions in the Programmer Guide.

You can then use the Virgo tooling to do all the usual things: start and stop the kernel, create and deploy bundles, PARs and plans, examine bundles, and explore package and service wirings.

OSGi Adoption Experience

The Mule team have blogged about their decision not to adopt OSGi. The decision seems reasonable: OSGi isn't a panacea and I guess the benefits didn't outweigh the costs in their case.

One of their number, Ross Mason, correctly identifies some of the costs of adopting OSGi. He would like OSGi to be almost completely invisible to the developer, leaving him or her free to define an application in terms of modules without dealing directly with manifests, bundles, and build systems. The somewhat messy details of developing and building modular applications with OSGi are partly what led SpringSource to donate the dm Server project to Eclipse as Virgo. The aim is to grow the user community and smooth out the wrinkles in the developer experience.

Adoption is going pretty well. Virgo recently shipped its first release. SAP are adopting Virgo in a strategic way. VMware products in development are using dm Server/Virgo to obtain the benefits of modular application code. Others are using dm Server/Virgo to host substantial applications in sectors including investment banking, networking, healthcare, online gaming, and academic research.

As we prepare to donate the Virgo tooling to Eclipse and collaborate in the OSGi Enterprise Tools project inside Eclipse with the likes of SAP, EclipseSource, and Eteration, I'm optimistic about improving the developer experience.

Dissenting voices such as those of the Mule team are a good reality check. Efforts, such as Mule, to try to find intermediate modularity solutions with some of the benefits of OSGi but with smaller costs should also help by providing data points for comparison with full blown OSGi adoption.

I'm looking forward to more experience reports from application development groups that have adopted OSGi or technologies like Mule so others can make a more informed decision about the technology to adopt. Certainly OSGi users are starting to publish their experience. Perhaps users of Mule will soon do the same.

Eclipse Virgo 2.1.0 Release

Jump to Hyperspace, courtesy of Steve Jurvetson

Eclipse Virgo has shipped its first release after several months of work to contribute it to Eclipse.

The majority of the effort was to make the contribution acceptable from an Eclipse legal perspective. There were some close shaves, like when a file (mime.types) in the Spring Framework turned out to have a restrictive license. The file would have been virtually impossible to recreate from scratch. I am very grateful to the original author, Ian Graham, for solving this problem by making the content available under an Apache license. I would also like to thank Barb Cochrane and Sharon Corbett of the Eclipse legal team who cheerfully and efficiently handled over 200 "contribution questionnaires" covering the code donated to Virgo and its dependencies.

Apart from that, we have repackaged the code in the org.eclipse namespace, speeded up the startup considerably, upgraded several dependencies, fixed a few bugs, and made several enhancements.

The response from the community was particularly encouraging with thousands of downloads of the milestones and numerous forum posts and mailing list discussions. We are particularly grateful to the code contributors, listed in the release notes.

The main objective of contributing Virgo was to grow the community and thereby make the technology more easily consumed by application developers. The project seems to be meeting the community objective and we are now preparing to donate the Virgo tooling to Eclipse which will enable the community to help us make the developer experience smoother, especially for newcomers to modularity and OSGi.

Finally, I'd like to thank the other Virgo committers, Chris Frost and Steve Powell, for their persistence and thoroughness. I'm glad we can now move on to more interesting items for the next release. Thanks too to everyone else in SpringSource who supported the Virgo project in various ways. We couldn't have done it without you.

The future looks promising, starting with a Virgo face to face meeting at the end of this month. It's not too late to express your interest (on virgo-dev) if you would like to attend.

Eclipse Virgo: the ideal OSGi server runtime

Since the Eclipse Virgo web site is seeing more new visitors than ever, it seemed like a good time to summarise the benefits of Virgo.

Essentially Virgo is the culmination of 3 years of work in developing an application server based entirely on OSGi and exposing OSGi for use by applications. Virgo started inside SpringSource as the dm Server project which was licensed under the GNU Public License and shipped two major versions. About a year ago we started donating the codebase to Eclipse as the Virgo project licensed under the much more liberal Eclipse Public License. The transfer of the runtime code to Eclipse is now complete, we have satisfied the rather stringent IP checks required of all Eclipse projects, and Virgo is nearing its first formal release by the end of October.

Virgo is an ideal OSGi server runtime, possibly the ideal OSGi server runtime, from several perspectives. Firstly, it was designed from the ground up as OSGi bundles so it's extremely modular and extensible. For example, the core of the runtime is the Virgo kernel and the Virgo web server is constructed by configuring a web layer, an admin console, and some other utilities on top of the kernel. The kernel protects itself from interference from such additions, and applications, by running them in a nested framework. This nested framework approach also enables applications to use new versions of the Spring framework (regardless of the fact the kernel depends on Spring 3.0.0). All the major Java EE application servers are now built on top of OSGi, but only Virgo was designed for OSGi and didn't need OSGi to be retrofitted.

Secondly, Virgo is an ideal OSGi server runtime because it exposes OSGi in a usable way to applications. In addition to all the standard features of OSGi — encapsulated modules or bundles, a powerful service registry, versioning, class space isolation, sharing of dependencies, dynamic refreshing and updating, and much more — Virgo provides a multi-bundle application model to simplify the deployment and management of non-trivial applications. Virgo also provides a repository which can store dependencies such as OSGi bundles which are "faulted in" when needed. This results in cleaner definitions of applications and a small footprint compared to traditional Java EE servers which pre-load many features just in case an application needs them. The major Java EE application servers are beginning to follow suit in exposing an application model, so SpringSource is working with other vendors in the OSGi Alliance to produce a standard multi-bundle application construct.

Thirdly, Virgo is an ideal OSGi server runtime because it enables existing Java libraries to run successfully in an OSGi environment. The Virgo kernel supports thread context class loading, load time weaving, classpath scanning, and a number of other features which are commonly used by persistence providers and other common Java utilities. Essentially, we observed the commonly-occurring problems when people attempted to migrate to OSGi and implemented general solutions to those problems early on.

Fourthly, Virgo is an ideal OSGi server runtime because it integrates OSGi with Spring and Tomcat. Virgo uses Spring DM to wire application contexts to the OSGi service registry. Spring beans can be published as OSGi services and can consume OSGi services, both with minimal effort. The embedded form of Tomcat is used as a servlet engine in Virgo's web support and is configured and managed just like standard Tomcat.

Fifthly, Virgo is an ideal OSGi server runtime because of its excellent diagnostics:

• An admin console lets you examine the state of all artifacts deployed in Virgo as well as explore the state of bundles in the OSGi framework.
• Virgo provides multiple types of diagnostics: event logging aimed at administrators, diagnostics logging aimed at developers, as well as various types of diagnostic dumps.
• Virgo builds on Logback to support highly configurable and efficient logging.
• When an application is deployed, Virgo first resolves the application in a "side state" and, if resolution fails, no changes are committed to the OSGi framework and the side state is dumped to disk so that it can be analysed using the OSGi state inspector in the admin console.
• If a resolution failure occurs, Virgo also analyses the resolver error, including the root causes of any uses constraint violation, and extracts a meaningful message to include in an exception which is then thrown.
• Virgo adds advanced diagnostics to Spring DM to track the starting of bundles and their application contexts and issue error messages on failure and warnings when dependencies are delayed.
• Virgo automatically detects deadlocks and generates a thread stack dump.

Finally, Virgo is an ideal OSGi server runtime because it has advanced tooling support in the SpringSource Tool Suite or in standard Eclipse via an update site. This enables a Virgo server to run under the control of the tooling, applications to be deployed, debugged, and updated by the tooling, and package and service dependencies between bundles to be analysed. The Virgo tooling will also be donated to Eclipse in due course. Oh, I almost forgot to mention the obvious: Virgo is an open source project with a liberal license and active participation from multiple vendors, which positions it ideally for the future.

Footnote: The bulk of this article was originally posted to springsource.org. Reproducing here for the benefit of Planet Eclipse and others.

I Love OSGi, Now Please Change It!

I'm sitting in the OSGi DevCon where a panel are discussing what needs to change in OSGi. Several points are coming up that are relevant to OSGi standards and also to what we have been doing in Virgo.

One excellent question was posed by a panel member, IBM Distinguished Engineer Ian Robinson, who asked whether OSGi needs to distinguish better between APIs intended for applications and APIs intended for middleware implementers and how this squares with the OSGi tendency for all bundles to be created equal.

I believe the answer here is to use composite bundles or nested frameworks to provide a unit of modularity larger than an individual bundle. I don't think OSGi needs to prescribe which APIs fall into each category, but it should provide mechanisms for hiding APIs in a given environment which are not intended for application use.

Virgo uses an early prototype of nested frameworks to achieve this kind of isolation between the Virgo kernel and the user region, which is a nested framework in which applications run. Users are free to deploy bundles in the user region and can even install additional bundles into the kernel and configure the packages and services shared between the kernel and the user region.

Fortunately, the notion of composite bundles and nested frameworks is being actively developed in the OSGi standards work and so their mechanisms should become more broadly available in a standard form in due course.

There is also OSGi standards work going on to expose these mechanisms in a usable way to applications. Virgo provides plans and plan archives as a way of grouping together artefacts for deployment and management. Apache Aries has also been implementing some similar grouping mechanisms with their application model. Paremus and others have similar constructs. So the standardisation activity has plenty of implementation experience and exposure to users to inform it. Virgo has also introduced a scoping mechanism as a way of isolating groups of application bundles from other groups of application bundles. This will also feed into the standards work.

Another great question from another panel member, the OSGi consultant Neil Bartlett, was how much OSGi should accommodate and support legacy code and how much it should require legacy code to be restructured to be more modular.

The direction of the Enterprise Expert Group in recent years has been to accommodate porting Java EE applications to an OSGi environment and then modularising them.

Virgo has had support for legacy code as one of the main design points from its inception. The Virgo kernel implements solutions to several knotty problems which crop up regularly when using non-OSGi libraries in an OSGi environment. These are assumptions about thread context class loading, load time weaving, class path scanning, and implicit package use.

So using implementations of OSGi Enterprise specifications is one part of the answer as is providing the kind of support for legacy libraries like that available in Virgo.

Virgo and Ocean Observatories Initiative

Virgo seems to be finding its way into the scientific community. A research hospital and a networking research group already use Virgo heavily. It seems that an oceanography group are also considering Virgo for their "ioncore-java" runtime.

They are at the stage of comparing various application servers. The front-runners, at least in terms of their feature comparison table, are Virgo and WSO2. Fortunately, I believe the architecture and design of these products are quite different, so this should make the decision easier.

Virgo was built from the ground up as OSGi bundles and offers first class support for OSGi applications. The Virgo kernel is independent of any particular communications protocol. Tomcat is added as part of the Virgo web server which sits on top of the kernel. The whole of Virgo runs inside an OSGi container. Web services can be supported on Virgo, but it's not something Virgo has majored on as this is just one of the features available in the Spring portfolio running on Virgo.

WSO2 on the other hand is focussed on - and the clue's in the name here - web services. It is constructed as an interoperating network of web services components. WSO2 does support OSGi, but it hosts an OSGi container inside Tomcat.

Let's see what the Ocean Observatories Initiative regard as the key requirements for their component model. If it's OSGi components with primarily local communication (i.e. Java method call), then I reckon Virgo would be ideal. If on the other hand they want a primarily distributed component model (and it's a big if) with web services as the communication protocol, then WSO2 may be a good choice.

One last word. I felt a bit sorry for Glassfish as it didn't get a tick in the J2EE box even though it's the J2EE reference implementation. Also Tomcat is the servlet engine used by Virgo, so it deserves just as much of a J2EE tick in the box as Virgo. Similarly JBoss should get the J2EE tick.

Grails on Virgo

Wolfgang Schell recently released an OSGi plugin for Grails. This runs on Virgo but apparently uses Spring DM's web extender rather than Virgo's web support. This makes me wonder if it would be better run on the Virgo kernel to avoid loading Virgo's web support. Perhaps he'll go into more detail when he has experimented more with Virgo.

RAP on Virgo

Owen Ou blogged about deploying RAP (Eclipse Rich Ajax Platform) on an OSGi based server like Virgo. He says it's extremely easy, which is good to hear, but I'd love to see a bit more detail. For instance, assuming he's actually doing it on Virgo, is the RAP infrastructure launched as a plan in the initial list of artifacts for the user region? How are RAP applications then deployed? Fascinating stuff...

[The link to Owen's blog broke. I guess he deleted the blog. Anyway, Florian Waibel has also blogged about RAP on Virgo.]

Virgo one of the best and brightest Eclipse projects

from eWeek.

Virgo kernel checked in and ready for use

The Virgo kernel was checked in to Eclipse git earlier today and is ready for you to take it for a spin.



No, this isn't the Virgo kernel - it's a 5 Mb hard disk from 1956 which weighed over a ton. The Virgo kernel zip file would have needed a couple of tons of hard disk for storage. Thank goodness times have moved on.

IntelliJ IDEA Support for dm Server

IntelliJ just announced support for dm Server. Although I haven't tried using the support, it appears to be at a reasonable level of function and only a little behind the SpringSource Tools Suite (as far as dm Server support is concerned) with plans to fill the gap. IntelliJ haven't mentioned Virgo yet, but I'm hopeful that the Eclipse (RT) branding won't put them off.

This, in my opinion, is a sign of a healthy runtime: multiple vendors competing on tooling. I'd like to see the same on management tooling which could be implemented relatively easily layer to dm Server's JMX interface.

First Virgo components available for development

See this if you are interested in doing some Virgo development.

Special thanks go to the Eclipse IP team, especially Barb and Sharon, for making this possible at this stage.

Also thanks to Chris Frost for his help in raising CQs.

Virgo contribution underway

I just finished entering the main CQs (Contribution Questionnaires) for Virgo. The corresponding tracking bugs, which are visible to everyone, are at the bottom of the list here.

There are now two gating factors before we can commit the code: the workload of the small but committed Eclipse legal team and my team's ability to field any issues they raise...



Once we've gotten the code committed, it will be possible to others to clone the git repositories, build, test, develop improvements, and submit patches. Essentially we'll be in business as far as contributors and build-it-yourself users are concerned.

Later, when we've also gotten the 3rd party dependencies through the legal process, we'll be able to start doing releases out of eclipse.org.

EclipseCon 2010 Virgo Presentation

Eclipse Virgo

View more presentations from glynnormington.

Reflections on EclipseCon

EclipseCon has been a great conference - on a par with the Spring conferences for enthusiasm and quality of talks etc. There was an enthusiastic welcome to the Virgo project which made my attendance really worthwhile. Several people discussed possible contributions to the Virgo project and there were a number of useful chats with potential consumers of Virgo as well as committers on other EclipseRT components who are keen to see Virgo fully integrate with the rest of RT.

For me, the highlight was the Virgo BoF with a room full of people really interested in learning more about Virgo. I was able to give a bit more detail about the internals of Virgo, clarify the separation of the various components, and start to explain the concepts underpinning the kernel.

I'd like to congratulate the organisers for putting on such a great conference and I look forward with anticipation to future EclipseCons.

Eclipse Virgo home page

Eclipse Virgo. What more can I say, other than "Thanks Chris"?

EclipseCon Sessions

EclipseCon starts in next week and there are plenty of sessions relevant to Virgo.

I'm doing the main Virgo talk and there's a Virgo launch BoF session (and a proposed BoF on Virgo tooling). I'm on panels on Gemini and the future of application servers and co-leading a BoF on application models for OSGi. I'm also giving a brief update on Virgo at the Eclipse Membership Meeting.

Then there's the main Gemini talk, an EclipseRT BoF, and other sessions on OSGi, Apache Aries, Maven and OSGi, and JPA and OSGi.

"Strong" optionality

Richard Hall tells me Felix interprets optional imports as "strong", to use my earlier term. I believe there is enough latitude in the OSGi spec to permit this. Equinox takes the opposite interpretation and can discard optional imports of available packages in order to overcome uses constraints which would otherwise prevent resolution.

Regardless of that, I think that strong optionality is more likely to be what the user wants.

Without strong optionality, if the user provides a bundle which can satisfy some optional imports, they are not likely to be pleased if the resolver discards corresponding optional imports as the functions provided by the bundle will then be unavailable. Better to fail fast with diagnostics that allow the user to sort out the uses constraints.

Even worse, without strong optionality resolution may fail after a protracted attempt to discard combinations of optional imports in order to get a valid wiring. All the optional imports could be discarded before embarking upon a protracted search to ensure that the search is not in vain. Perhaps Equinox already does this, but I'm doubtful it as it could be criticised as optimising for the failure case.

OSGi resolution is NP-Complete. So what?

At last we have a proof, thanks to Robert Dunne, that the OSGi R4 resolution problem is NP-Complete. He showed that it is possible to represent a solution to the NP-Complete boolean satisfaction problem 3-SAT as a wiring of a carefully constructed set of OSGi bundles. After the idea arose of using a SAT solver to construct an OSGi resolver, some of us have been discussing the possibility of a SAT-based proof. The hard step was how to represent a boolean OR in terms of resolution, which Robert has now provided.

NP-Completeness tallies with the experience of those who have written an OSGi R4 resolver: it takes about a fortnight of intensive hacking, combined with sleepless nights, to crack the problem followed by a much longer period of bug fixing and optimisation. Now we know why: NP-Complete problems are hard to solve efficiently.

I'm partly to blame as I led the spec work on OSGi RFC 79 which added considerable function, and with it complexity, to the resolution algorithm. I presented some of the background at the 2004 OSGi Congress. The RFC 79 spec was one of the more interesting that the OSGi Core Platform Expert Group has worked on. There was a tension between addressing the basic use cases and creating something that was too hard to implement.

Fortunately, we developed a prototype resolver in the Equinox incubator. This provided useful feedback to the spec work, particularly when features were dropped from the spec which made the implementation more tractable. The team which created the prototype included Simon Burns, Steve Poole, and Tom Watson. Simon wrote the initial R4 resolver. Steve created a rigorous model-based test based on a slow, but functionally corrrect, resolver of his own. Tom provided much advice on the R3 resolver and eventually integrated the new resolver into Equinox and has been improving it ever since.

So what features emerged from this process that made the resolver NP-Complete? Well, crucial to the proof are the uses constraint and, its evil twin, optional imports. Steve Powell observes that the proof does not depend on the transitivity of the uses constraint. (Other, arguably much more evil, features in R4, require-bundle and fragments, also did not contribute to the resolver's intrinsic complexity.)

This suggests some practical ways round the problem. Examples occasionally crop up when the time to resolve a set of bundles becomes unacceptable. How can we work around such issues if we are willing to put in the effort?

Firstly, uses constraints can be avoided by preventing types from one package leaking out through the interface of another package. This is an application of the Law of Demeter.

Secondly, in some cases it may be possible to remove, or at least cut down, optional imports by using mandatory imports instead and splitting the function which has those dependencies across bundles so that the user's dependencies determine which bundles are needed.

These approaches are feasible only for new code. Bundles generated from popular Java libraries tend to exhibit both uses constraints and optional imports.

Another possibility we've been considering is a "strong" form of optionality in which the optional import cannot be discarded if it is locally satisfiable, i.e. before uses constraints are applied. Certainly the current definition of optional import is critical to Robert's proof. The challenge is to convince ourselves that R4 resolution with strong optionality in place of the current definition is no longer NP-Complete or, conversely, to prove the opposite.

I'm undecided. Strong optionality drastically reduces the search space in some cases. On the other hand, I feel there may be a proof based on 3-SAT in which multiple bundles import, but do not export, packages.

OSGi standards and adoption

The TSS thread on Virgo and whether or not OSGi is ready for mainstream enterprise application development contains some misconceptions.

Firstly, that SpringSource is somehow not involved in the OSGi standards process.

That couldn't be further from the truth. Recently we've led the standardisation of Spring DM into the blueprint service specification and reference implementation while IBM provided the compliance tests. We are developing the reference implementation of the OSGi web container while IBM wrote the specification and are providing the compliance tests. Then, as we look to the future, we are leading two, and involved in other, new specifications targeted at R4.3.

Secondly, it is easy to get the impression that other vendors, such as IBM and Oracle, are about to solve the adoption problem.

I certainly hope they do, but, at least in the case of IBM's WebSphere Application Server OSGi alpha, their programming models seem very similar to that of dm Server and their development and build tools, where the real adoption issues seem to be, do not seem to be particularly advanced.

Perhaps IBM's large customer base and marketing machine will encourage developers to jump the necessary hurdles. But the tricky thing about a product like WAS is that, after the alpha code has been rolled into the main product, it will be very hard to measure real usage. The only real evidence will be customer testimonials and APAR (i.e. external defect) rates against the OSGi features, which IBM typically do not publish.

Let's see how Virgo gets on. At least the adoption should be easier to measure as users will almost certainly be using it for its OSGi features.

Building OSGi applications

The recent SD Times article on Virgo paints an overly bleak picture of the problems of building an OSGi application.

Build certainly complicates adopting OSGi, but many users are willing to overcome that problem in order to get the benefits of modularity and the service registry.

Also various tools are available to help. We developed the bundlor tool for automatically generating bundle manifests from the code in a bundle. Peter Kriens's bnd tool provides similar function.

People have successfully developed Maven and Ant builds for OSGi projects and various helper plugins. for instance PAX Construct, are starting to emerge.

Also, SD Times fails to mention the Enterprise Bundle Repository which already makes available (directly or via other repository front ends) about a thousand of the most popular open source Java components packaged as OSGi bundles with carefully constructed bundle manifests.

Eclipse Virgo project

The news just broke. SpringSource are donating the dm Server code base to eclipse.org as part of the EclipseRT umbrella project. This should be just the server runtime that EclipseRT has been looking for. The Virgo project proposal is available and more details are on the SpringSource blog.