CME Group and Virgo

Today CME Group released an official statement about their use of Virgo. This is posted under the "Virgo Powered" section of the Virgo home page, but the full text is also reproduced below.

I'm delighted both that CME Group are using Virgo and that such a large player in the finance area was happy to go public.

CME Group, formerly known as Chicago Mercantile Exchange Holdings Inc., was founded in 1898 and operates the CME, CBOT, NYMEX, and COMEX regulatory exchanges worldwide. It is the world’s leading derivatives marketplace with 2570 full time employees and a market capitalisation of over $15B.

CME Group kindly released the following information in January 2012.

At CME Group, the goals for our project were to use and create a platform that would be easy to maintain, expand and reuse. Our applications are of various sizes and complexities, but there are certain aspects that are common, which was the reason to find a very modular solution. In particular, our current efforts are in Surveillance tools which allow monitoring the state of the exchange and the market.

We started using Virgo 2.0.1 during a “proof of concept” in November of 2010 and spent 2011 working closely with them to build the application. Today we use version 3.0.1. with 48 service bundles and 20 web bundles. These services range from simple DAO type services, through authorization and authentication, web services, and even distributed caching. All web bundles use snaps and customized apache tiles views.

What Virgo and OSGi provides for us is complete modularity and plug-and-play capability. This enables us to work more intelligently with other teams who need to build bundles independently of us where the strong isolation of services and APIs allows for a clear separation of control. We have created a core set of services that will be a platform for future projects across CME Group and are already actively building two projects on top of that platform. Having Virgo bundled with Apache’s Tomcat also helped in deciding which container to use as our projects are mostly web applications.

Virgo Nano Technology

The first milestone of Virgo 3.5.0 is available. It introduces two significant new features: p2 support (covered previously) and a new Virgo distribution known as nano. (Apologies for the pun in the blog title - it was too hard to resist.)

Nano is essentially a cut-down version of Virgo which starts up really fast and has a single (kernel) region. It is intended for simpler scenarios where you don't need to isolate applications from each other or from the kernel.

The current nano distribution includes Gemini Web and so is a fully-functional OSGi web server which starts in about 3 seconds (compared to Virgo Tomcat/Jetty Server which take about 9-10 seconds). However, we plan to factor out Gemini Web so that nano becomes a small/fast subset of the Virgo kernel. We will then provide a separate distribution which adds Gemini Web to nano, as in the first milestone.

As well as introducing nano, the milestone re-bases the kernel and the Tomcat/Jetty servers on nano. Initial provisioning via p2, with full instructions in the user guide, is provided for all distributions in addition to the familiar ZIP file installs (the ZIP contents are now constructed using p2 at build time). All distributions use a regular Equinox directory layout and launcher which will simplify third party tooling integration (e.g. Pax Exam). A short-term downside is that we need to update the Virgo IDE tooling to work with Virgo 3.5.0.

The current nano distribution can also dynamically provision content via p2 which is particularly useful for automated or cloud deployments. (We tried to implement this for the multi-region kernel and Tomcat/Jetty servers, but that was beyond the scope of the current p2 design, so we've put it on hold.)

There is also a nice engineering side-effect of the introduction of nano: we are starting to refactor the kernel — by far the most complex Virgo component — and move the more general, region-agnostic features down into nano.

The following picture summarises the distributions we anticipate in Virgo 3.5.0:

As well as being the fruit of several months of effort, this first milestone kicks off Virgo development in 2012 in a very exciting way.

OSGi testing with Gradle and Pax Exam

I've recently been working on a Spring issue which involved writing a Gradle-built test to ensure that the Spring framework 3.1.x bundles successfully resolve in an OSGi container.

The Gradle side of this was a very simple and pleasant experience, as the following extract of the file build.gradle shows:

apply plugin: 'java'
apply plugin: 'eclipse'

repositories {
    mavenCentral()
}

dependencies {
    ...                
}

There were minor usability issues¹ with the Gradle Eclipse plugin, but I gather these will soon be resolved.

I decided to use Pax Exam to launch the testcase inside Equinox, and this require specifying some dependencies in build.gradle:

dependencies {
    paxExamVersion = '2.3.0.M1'
    equinoxVersion = '3.6.0.v20100517'
    
    testCompile "junit:junit:4.+",
                "org.ops4j.pax.exam:pax-exam-junit4:$paxExamVersion",
                "org.eclipse.osgi:org.eclipse.osgi:$equinoxVersion",
                "javax.inject:javax.inject:1"
                
    testRuntime "org.ops4j.pax.exam:pax-exam-container-native:$paxExamVersion",
                "org.ops4j.pax.exam:pax-exam-link-mvn:$paxExamVersion",
                "org.ops4j.pax.url:pax-url-aether:1.3.5"
}

Let's go through the dependencies in turn:

• JUnit 4 is the test framework.
• @RunWith(JUnit4TestRunner.class) marks the test for launching under Pax Exam.
• Equinox provides OSGi types at compile time and an OSGi container at runtime.
• A bundle context is injected into the test using @javax.inject.Inject.
• The test uses the native Pax Exam container - all the tests run in a single JVM.
• The test uses Pax Exam to install Spring bundles directly from Maven repositories.²
• The test caches bundles from the SpringSource Enterprise Bundle Repository using the Pax URL Mvn protocol.

The Pax Exam developers kindly helped me sort a basic setup issue which wasn't too obvious from the documentation. Thanks Harald and Toni! I needed to explicitly install a JUnit bundle, by calling junitBundles from the configuration method:

    @Configuration
    public static Option[] configuration() throws Exception {
        return options(//
            provisionSpringBundle("spring-core"), //
            provisionSpringBundle("spring-beans"), //
            // etc.
            junitBundles());
    }

The full source of the project is available on github (under the Apache 2.0 license) for anyone who wants to give it a spin or use it for their own purposes. If you'd prefer to write your tests in Groovy, take a look at Hamlet D'Arcy's blog.

Footnote:
1: The generated .classpath contains absolute paths rather than paths relative to the project and so is not suitable for committing and sharing with others. Also re-running gradle eclipse without first deleting .classpath can result in it containing duplicate paths. Luke Daley tells me that both these problems will soon be fixed, the latter fix appearing in gradle 1.0 milestone 7.
2: If the testcase was intended a a stand-alone test of published versions of Spring, it would cache Spring bundles in the local Maven repository for efficiency. But we plan to integrate the testcase into the Spring build in due course and so we'll want to grab the latest built version each time to cope with re-spins, so cacheing isn't helpful.

Virgo 3.0.2 released

Virgo 3.0.2 is available for download. Please see the release notes for details of this bug fix release.

Opening SourceTree from the command line

I've recently started using SourceTree to visualise and manage my git repos. It's very impressive.

The one thing I missed from gitx was the ability to launch SourceTree from the command line, in any directory of the repo. One solution is to set a git alias:

git config --global --add alias.sourcetree '!open -a SourceTree .'

and add an alias in the shell profile:

alias st='git sourcetree'

Now I can issue "st" from the command line and the relevant SourceTree window opens up.

(Thanks to Chris Frost for providing part of this solution.)

What did we learn in 100 sprints?

This week marks the 100th Sprint on Eclipse Virgo! That includes the precursor project, SpringSource dm Server. What have we learned about scrum? Well, obviously, it's pretty good otherwise we would have ditched it long ago (and believe me I've been tempted a couple of times). But specifically...


Firstly, that one week sprints are by far the easiest to manage, both in terms of planning effort, but also psychologically. Even in a two week sprint, it's easy to put off unpleasant tasks until it's too late.


Secondly, that the main value of scrum is to keep the team focussed on specific tasks during a sprint and enable progress to be shared easily at daily stand-up meetings.


Thirdly, that story points are more effort than they are worth, so it's more effective to go with gut feel.


Fourthly, that you have to take a relaxed attitude towards planning design work as it's almost impossible to predict the effort up front. Spikes are ok for small items, but sometimes you just need plenty of time to get your head round a large area.

Enterprise Bundle Repository Samples

After opening up the Enterprise Bundle Repository build system, I've added more sample bundlor templates along with Ant/Ivy build files to github.

So while we are trying to come up with a community-based successor to the EBR, if you need to get a new version of one of the bundles in the EBR, you may be able to find a bundlor template to use as a starting point.