Virgo survey results

Thanks to everyone who responded to the recent Eclipse Virgo and SpringSource dm Server survey. The results make interesting reading.

Close to one quarter of respondents report they are "in production".

The transfer to Eclipse.org is a success. It is 6 months since the first Virgo release and 84% of respondents are now using Virgo exclusively with only 13% using dm Server exclusively.

Virgo is popular with early adopters. Of the 84% using Virgo exclusively, 59% are using 2.1.x or 3.0 milestones and 25% are on 3.0 milestones only (some of these already in production!). Please note that Virgo has versions 2.1.0 and greater whereas dm Server covers versions 1.x and 2.0.x.

The Virgo kernel is seeing significant usage. The kernel underpins the Virgo Web Server and provides Virgo's advanced deployment, provisioning, isolation, scoping, and diagnostic capabilities. The kernel has no web-specific code and so can be used to build messaging and other non-HTTP servers and to act as a container for non-server applications. 16% of respondents are using the kernel exclusively; 5% are in production.

Documentation and blogs were the most useful information sources closely followed by the Virgo forum, Virgo wiki, and Virgo development mailing list (1= not at all useful, 5 = very useful).

The survey was advertised on Twitter, the Virgo forum, the Virgo development mailing list, at www.springsource.org, and in this blog. 166 people responded. If you are interested, the raw results of the survey are available online or a summary can be downloaded.

Finally, I'd like to thank Chris Frost for coming up with the idea for the survey and then delivering the goods.

Quick survey of Virgo and SpringSource dm Server usage

I would be grateful if users of Virgo and SpringSource dm Server would take a short, anonymous survey to help us in our planning. Should take no more than 2 minutes to complete.

OSGi Adoption Experience

The Mule team have blogged about their decision not to adopt OSGi. The decision seems reasonable: OSGi isn't a panacea and I guess the benefits didn't outweigh the costs in their case.

One of their number, Ross Mason, correctly identifies some of the costs of adopting OSGi. He would like OSGi to be almost completely invisible to the developer, leaving him or her free to define an application in terms of modules without dealing directly with manifests, bundles, and build systems. The somewhat messy details of developing and building modular applications with OSGi are partly what led SpringSource to donate the dm Server project to Eclipse as Virgo. The aim is to grow the user community and smooth out the wrinkles in the developer experience.

Adoption is going pretty well. Virgo recently shipped its first release. SAP are adopting Virgo in a strategic way. VMware products in development are using dm Server/Virgo to obtain the benefits of modular application code. Others are using dm Server/Virgo to host substantial applications in sectors including investment banking, networking, healthcare, online gaming, and academic research.

As we prepare to donate the Virgo tooling to Eclipse and collaborate in the OSGi Enterprise Tools project inside Eclipse with the likes of SAP, EclipseSource, and Eteration, I'm optimistic about improving the developer experience.

Dissenting voices such as those of the Mule team are a good reality check. Efforts, such as Mule, to try to find intermediate modularity solutions with some of the benefits of OSGi but with smaller costs should also help by providing data points for comparison with full blown OSGi adoption.

I'm looking forward to more experience reports from application development groups that have adopted OSGi or technologies like Mule so others can make a more informed decision about the technology to adopt. Certainly OSGi users are starting to publish their experience. Perhaps users of Mule will soon do the same.

IntelliJ IDEA Support for dm Server

IntelliJ just announced support for dm Server. Although I haven't tried using the support, it appears to be at a reasonable level of function and only a little behind the SpringSource Tools Suite (as far as dm Server support is concerned) with plans to fill the gap. IntelliJ haven't mentioned Virgo yet, but I'm hopeful that the Eclipse (RT) branding won't put them off.

This, in my opinion, is a sign of a healthy runtime: multiple vendors competing on tooling. I'd like to see the same on management tooling which could be implemented relatively easily layer to dm Server's JMX interface.

EclipseCon 2010 Virgo Presentation

Eclipse Virgo

View more presentations from glynnormington.

SpringSource dm Server catching on

The recently announced alpha driver of IBM WebSphere Application Server v7 support for OSGi applications is covered in a brief dzone interview with Kirk Knoernschild of the Burton Group.

As Kirk notes in his blog, SpringSource dm Server has already brought OSGi to the enterprise. dm Server v1.0 shipped in September 2008 and v2.0 will enter RC1 in the next few weeks.

The WAS alpha has some features in common with dm Server, notably repositories and applications along similar lines to plans and PARs. Immitation is the sincerest form of flattery or, putting it another way, IBM has validated dm Server's approach.

I'm still waiting for technical details of WebLogic DM Server, but since it is microkernel based (unlike WAS) and with a name like that, it's likely to validate dm Server further.

Meanwhile GlassFish and JBoss support for OSGi is coming along nicely, so it'll be fascinating to see what kind of OSGi application models these projects end up providing.

Coding and hill-walking

I've recently been re-organising the internals of the SpringSource dm Kernel so that it can be extended to provide structured install artefacts known as "plans". The current code has been extended to provide a plan equivalent to the existing PAR files, but other variants of plans, unscoped and non-atomic, cannot be implemented without the necessary data structures being put in place.

From early in the development of dm Server 1.0, the team was using several terms which summarised the behaviour of the kernel, but which were not easily identified in the code. Terms such as "scope" and "deployment pipeline". As we started thinking about 2.0, we identified "plan", "install tree", and "install environment" as basic concepts, even if the latter was pretty fluffy.

Scopes were implemented as first class types quite a while ago now and were crucial to being able to make cloning work. One of the trickiest areas was performing operations on types which could be cloned or non-cloned. Explicit scopes enabled these operations to be coded to a non-cloned interface type and re-dispatched as necessary to the appropriate cloned type.

Recently, I've been turning deployment pipeline, install tree, and install environment into first class types. I really love this kind of work. It's rather like walking through a range of hills where the higher peaks are only visible when some of the lower summits are behind you.

Sometimes however things get much more strenuous. It's like the terrain is getting much steeper and there no obvious path to follow.

This afternoon I was struggling to structure the code to handle the starting of bundles. There's an interesting coordination and state management issue because starting a bundle has a synchronous phase defined by OSGi and often an asynchronous phase defined by Spring DM or the OSGi Blueprint Service. I made some progress, but I'm still not happy with the current code and it certainly isn't easily unit tested which is a good measure of code quality.

At this point, the only option is to take a break and come back to the problem with a clear head. I tried consulting colleagues and that helped quite a bit, but there are hopefully some really good abstractions waiting to be discovered.

I think the root cause of the complexity of this small area is that it is fairly low level and we're lacking some really good ways of thinking about the asynchronous processes involved. Occasionally, modelling processes in CSP and exploring the properties of the model using a model checker such as FDR has helped. But more often than not, the best approach seems to be iteratively cleaning up a bunch of code until a good structure starts to emerge.

In the worst case, I'll make the code as clean as possible and add it to the list of areas where future discoveries will hopefully make things cleaner. But I'm hoping for a much better outcome.

Well, that's enough for now. I put this blog together as I expect other people have similar experiences when working in complex areas of software and it's good to learn from each other. I'd be interested in your perspective and how you approach these kinds of activities.

I'll leave you with a hill-walking illustration of my current position: standing on one peak trying to make out the next one in the mist.

SpringSource dm Server 2.0 M1

As in Rob's blog, we recently shipped the first milestone of dm Server v2.0. Apart from some new function, this marks an opening up of the dm Server development process with publicly accessible subversion repositories and issue tracker.

The 2.0 roadmap is pretty exciting, and includes constructing a Reference Implementation for RFC 66 ("web container for OSGi") from the existing web components of dm Server.

I'm spending most of my time down in the dm Kernel implementing "cloning" which, roughly speaking, enables a bundle to be wired more than one way, which is not normally possible in OSGi. Cloning copies bundles into an application's scope. This enables the OSGi resolver to have another go at wiring the dependencies of the cloned bundles in a way which suits the application. More information on cloning is included in the above blog entries.

Application scopes can be thought of as an alternative to "nested" OSGi frameworks, which are being specified in the OSGi Alliance and implemented in Equinox. The contents of an application scope can "see" all the contents of the global scope which are not "shadowed" in the application scope. The contents of a nested framework can, by default, not see any of the content of the parent framework(s) unless the nested framework is explicitly defined to see some of the parents' contents.

Much of what we're learning about cloning into application scopes, such as how to calculate what needs cloning and how to manage cloned extenders, will also apply to cloning into nested frameworks if and when that becomes necessary.