Virgo survey results

Thanks to the ninety-seven people who filled in the Virgo survey. The survey is now closed and the results are in. I'll summarise the results here, but the raw data (in Open Document Spreadsheet format) are available for anyone who wants more detail.

88% of respondents are on Virgo 3.5.0 or later while only 5% are still using SpringSource dm Server.

Virgo Server for Apache Tomcat is by far the most popular runtime deliverable. I would expect this to change over time if SAP's cloud strategy based on Nano Web is successful, even if such success doesn't feature much in Virgo survey results.

A greater respondents (36%) are in production this time, compared to 23% in May 2011.

OSGi and Spring are both preferred programming models with Java EE being somewhat less preferred.  Admittedly this question doesn't capture those who are forced to use a programming model they would prefer not to, but hopefully that number is small given the choice of programming model provided by Virgo.

The future directions were all of interest, some a little more than others. I'm not going to comment in detail, but better support for OSGi standard applications is particularly strong with a high "Gimme gimme" ranking and a tiny snooze factor.

The written comments were illuminating on the whole. Here are the comments for question 7 "Any other future direction(s) you'd really like to see".

Better Maven integration: Automatic deployment, Automatic installation, etc. Better Documentation on How To do integration tests of a complex Web Appplication in Virgo.

I would like to see a better testing framework for Virgo unit and integration testing.

Ensuring that the tooling follows the development of the Virgo server, because it's quite slow right now and often leads to some null pointer that imply reconfiguring the server in STS :(

reliability and troubleshooting are a major concern. As Virgo exposes OSGi to applications - it's harder to debug classloading problems, compared to pure Tomcat or Jetty environments. xx

Question 11 "Anything else to get off your chest?" produced quite a bit of feedback. The tooling is fairly criticised, so we are hoping the upcoming 1.0.1 release will fix a number of common problems. There are repeated calls for sharing of experience with real world applications and technologies, something I feel the Virgo community could helpfully contribute, for instance via blogs or on the Virgo wiki: I'd invite anyone interested to go ahead and add to the Community section. Meanwhile, we have been revamping the samples and will be issuing a milestone soon.

Integration testing made easy (we talked about cargo integ' on the forum and it's for me the priority to leverage Virgo as a top level platform).

Virgo was not simple to shrink, we ended up with a number of components that we could not easily throw out. We did eventually get the right isolation and app working, but the project failed for other non technical reasons. There are generally too many frame works in java, and the owners of java have let opportunity to fix jar hell slip too many times. Look at other language ecosystems as examples. Erlang makes it super easy for operators to update code in a system.

Virgo: Don't push yourself so hard today. Try to find some relaxation time. If you had big plans, perhaps you should cancel them. Instead, schedule some time with friends or family. Or treat yourself to a little indulgence. Some wine tasting here or appetizers there won't hurt as long as you don't go overboard.

We really would like to see a much more stable version of the virgo tooling. Virgo must be restarted way too often during deployment because of tooling problems.

Move away from Spring. JavaEE + OSGi is a better combination. Nice to see that supported in the latest(?) release.

Generally I'm sorry to say I just don't see a future for Virgo. The container is too hard to use and the complexity is too much for the benefits delivered. Gettting a working build and keeping it working with manifest manipulation is still too hard and error prone. It seems unlikely it will ever get fixed or improved. The "next big thing" seems to be Typesafe. But for the most part, containers are going away, and we're going to move to something like AMQP based services. In fact, you can get JVM isolation just by running more JVMs in the cloud. So the unit of deployment is the Virtual Machine, not an OSGi bundle. Sorry for the negative feedback, personally I have always liked OSGi, but for most developers it's too hard to use for the benefits derived.

The first one thanks for your job, and support of this great Software called Virgo. We are developing in OSGi and Virgo about two years, developing Konekti aka Modular Web business platform. We suffer the bad initial tools in eclipse to develop, actually are better. Of course we would like more documentation :) We think it would be interesting adding any blog or tutorials about integration of other frameworks, or experience with Virgo. Our company spent many time integrating many frameworks like JBMP, Drools, ActiveMQ, Jersey, GWT, Vaadin, Jasper Reports, Quartz, ... , so we could provide our experience. Also we would like to know this experience of other companies that also are developing in OSGi and Virgo. We think that Virgo and OSGi in general are a great technology, but not many people knows about it, so any pedagogical work is necessary, and any must lead it. If in the future you add some meeting point, we can contribute it with our real experience in OSGi and Virgo. Best Regards.

Without Eclipse Virgo we would probably not have choosen to try OSGi in my project. Because we need to migrate a Java web application with quite a lot dependencies.

My take on the future - you need to branch out and ensure as much of the available Java technologies are instantly usable in Virgo. Classify the technologies (something like java-source.net does, but better) and pick a project for each one to start with, involve the community and build a library of addons and examples - you'll not only prevent users from bailing on using Virgo because of too much program infrastructure integration effort, but do just the opposite - attract them with ready-to-use examples and simple working projects. I started playing with Virgo since about 7 months. I am using Virgo to rebuild an existing security application platform which we provide to our customers and i find it very attractive to use for me as a developer. Using Virgo as an application server platform opens up a huge potential to me and allows me to build real modular applications based on osgi without having to deal with osgi runtime basics to get started, which is great. The features Virgo provides are good and do make sense to have in such an environment. Although, as mentioned above, Virgo Tooling should be improved and in my opinion this is the key element to attract more developers for using Virgo.

We use it as a managed osgi runtime and install everything into one region for sanity. Virgo rewriting bundle imports causes more problems than they are worth. Using different containers instead of a partitioned runtime makes more sense for disparate applications.

Virgo tools need a lot of improvements. In my company we ended up implementing a little set of eclipse plug-ins to perform similar tasks as those offered by Virgo tools. For example, ordering of projects deployed to the server, because whenever you open the Virgo server editor it takes ages to show up and you just have a 5 lines list for ordering bundles, with no possibility to perform a multiple selection.

Nice modular concept. Needs examples to host Virgo Nano in regular J2EE web app

I don't know what I'd do without you guys!

sorry but I feel you're apart from rest of the osgi community. Why not use or contribute to bnd, maven-bundle-plugin... (note : I have no idea about politics) Please keep thing simple, because It's not easy to defend osgi. I try in my compagny, but osgi feel complicated. In fact it's not. Just a problem of image.

First up - it's excellent. We were going to migrate to an OSGI stack but Virgo seems to have addressed a whole bunch of issues we knew we would hit. The down sides are: - eclipse tooling a bit unreliable / immature (still cannot deploy a ) - different approach from target platform standard eclipse stuff (more eclipse general issue maybe) - the whole spring, hibernate, cglib, resolve issues, class not found issues, blah blah is still a nightmare (nature of the game I guess) - *particularly if migrating to virgo* - greenfield may be easier

Hate the fact that there is no book on Virgo yet, hate that using it depends on other projects (Gemini) that also have sparse documentation. Even a simple hello world on one of the technologies can be a waste of a few hours when adapting to Virgo. Felix/Aries start to gain more traction now. No Paxrunner support for Virgo yet.

To me virgo nano with documentation regarding on how to build isolation would be perfect.

The whole jetty thing and not having an answer for regular Jdk app server model.

Virgo is great and has been very reliable. It is OSGi and integrating 3rd party software that is the pain. For example, getting DB2 datasources using a ConnectionPool framework and getting the datasources via JNDI lookup or a Declarative service takes a lot of work to configure. I still have not seen any good documentation about storing encrypted data in external properties files and then having a hook to decrypt the data so that it may be used in a blueprint.xml definition in the context of ${var} substitution. I do not want the id/password used for my database connections to be stored in plain text in some properties file. Seems that there should be a good recipe or API for handling such common security concerns. For the most part I am well pleased with what Virgo offers, and I am in the middle of transforming a JBoss J2EE multi-tier application to a pure OSGi application running under Virgo.

I love Virgo but the learning curve is steep. Not only Virgo's fault but overall to develop SpringDM applications I had many problems since there are many technologies involved none of which is particularly stable. Random problems: Bundles where there which were not seen, then they were seen. Spring services were found then were not found anymore. Then deleting the work directory solved a lot. Then a bundle was found with version 0.0.0 and you have no idea why. Then 1 day later you realise there was a missing dependency. Etc. Debugging is still quite tricky, when things work do work but many times to get there is a huge pain. Anyway Virgo rocks :) Also the problem with many JARs around which are not yet OSGfied. And to OSGi them you have to beg the developers. Or you have to wrap it yourself but then you have to do it everytime there's a release. There are ways to easily OSGfy a plugin but it would be very useful to make the process as seamless as possible and push the community to embrace OSGi.

I really appreciate what you guys doing. But I don't like the current state of maven support. We use maven for every project it is key technology at our company. It was not easy to setup maven multi-module project to work with eclipse virgo ide and be able to successfully build outside eclipse as well. (driving all osgi dependencies from maven side + generating manifest + providing transient dependencies so virgo can see them when we add dependency just to pom.xml) Do not have sources this days is just sad (and make it that bundles which comes with virgo to be source-aware is complete madness) - with maven it is just so easy. Anyway keep up good work.

Its still hard to managed those jar dependencies or change to more recent version of spring without running into conflicts.

Tooling tooling tooling. Is has been the biggest issue and time consuming process. We have finally crafted a build setup that support maven in CI and in Sts that works but at times can be fragile and difficult to troubleshoot. Many many times we ponder if there is a better way and aren't able to find the answer.

Startup time is the major pain point for us.

I think you need to do more community development. Work on describing how people can use vanilla technologies deployed in Virgo and then move into OSGi modules if they want to. My main concern is trying to stay within a technology stack that is either directly portable or easily ported to a different server as an option.

Seems like things took a while to transition to Eclipse, but feeling better about Virgo and tooling by the day. We're most interested in running this on tiny devices, so are happy (and hoping) to see continued support for Nano.

We are very happy with Virgo! We enjoy the fact that's pretty lightweight, easy to configure and provides good performance. A good thing would be if more than one pickup-folder could be defined.

In my opinion Virgo is lacking some really good documentation and programming examples. The examples available are all using maven and not the eclipse pde dependencies and tools, wich are the easiest way to develop within Eclipse.

Virgo Tooling with Eclipse is a nightmare. So many bugs, runtime parameters that can't be changed (try to change MaxPermSize...) or the lib\endorsed directory not included - took us hours to find out. You guys should eat your own dog food and start to use these tools not just in hobby projects.... I guess this would improve quality.

You should put more effort in the Virgo Tooling it's the more frustrating thing in Virgo. It is very slow and buggy. Keep the good work!

I want better virgo tooling in eclipse/sts. Each times i open tooling in eclipse, it's very slow still 30-40secs. If i stop virgo in error during starting then i need kill eclipse.

The technology section of the survey is rather long and hard to summarise, but it will provide useful input for future development. See the raw data if you are interested.

Finally, the quiz results were intriguing. The correct (or should I say my) answers are:

JOnAS was the biggest mystery with 49% responding "Dunno". WebLogic was second with 32%, just in front of WebSphere with 28%. One person thought that Virgo does not use OSGi or explicitly support it, but their preferred programming model was Java EE, so perhaps they see Virgo as just another enterprise server.

No solution for complexity

Neil Bartlett's blog of this title is shocking because it shows how slow to adopt proven technology the computing industry is.

David Parnas wrote his seminal paper "On the Criteria To Be Used in Decomposing Systems into Modules" (available here) in 1972. A large transaction processing monitor was modularised in the 1980's because the cost of servicing it was spiralling out of control. (Precise specifications were written for the new modules to clean up the interfaces and minimise dependencies on module internals, but that's another story.)

The OSGi module system for Java has been mature for at least five years, possibly longer depending on how you count it. Most of the major Java application servers are now constructed of OSGi modules and a number of them expose OSGi for applications to use. And yet we still see complaints such as the one Neil quotes. It seems application developers are stuck in their ways or stuck on systems which prevent them from using modularity.

Of course, modularity will eventually be adopted by all business critical software. There's really no rational alternative. But how long it will take is quite another matter.

Why I hate spikes

The recording of Dan North's recent talk on "Embracing Uncertainty" set me thinking again about what I like and dislike about agile methods such as scrum. As we end the 157th consecutive sprint of the Virgo project, I certainly appreciate the focus and sense of making progress that each sprint brings.

But what I really dislike are spikes.

The principle of a spike sounds fine: carve out some time to do some exploratory work. But I soon find myself too constrained by a spike. One definition of a spike is (emphasis added):

"a story that cannot be estimated until a development team runs a timeboxed investigation. The output of a spike story is an estimate for the original story."

There are some assumptions buried in that definition which I think show some of the problems:

• implicitly, and significantly, a spike is something you do in a sprint
• timeboxed - of course we can't commit an indefinite amount of time to a spike, but a timebox kind of implies that at the end, you'll come up with "the goods"
• estimate - the "goods" are an estimate
• original story - there's an assumption that the spike won't change your perception of what the story should be.

Here's where Dan's presentation helped me understand what was going on. Spikes are an attempt to fit some explorative work into an agile process which is designed for repeatedly churning out reliable software with minimal risk.

My natural way of doing exploratory work is rather different. First of all, I don't presume to know what it is I expect to find. I have some ideas to try out, some questions to answer, or maybe just something I need to learn about (and believe me, the longer I stay in this job, the more ignorant I realise I am).

So the actual "goods" are really answers, understanding, and more ideas. It might be possible to turn some of these into estimates for a story, but that's not really the goal.

What's wrong with trying to fit exploratory work into a sprint, you may ask? I think the psychology of reporting status in a stand-up meeting each day forces me to focus on things which will deliver immediate results. Sometimes the best way to explore something is to follow a number of lines of inquiry, get yourself thoroughly overwhelmed, take a break, and then (after several pennies have hopefully dropped), start to understand the underlying concepts.

Worse still, if I have at least one non-trivial task in the sprint besides a spike, it's really hard to get into the exploratory frame of mind. The only way I've found is to get the other stuff out of the way as quickly as possible and then try to switch modes.

So in the future, I propose to approach exploratory work differently. I'll treat it as something that's done outside a sprint. I'll still keep the idea of a timebox - it's no good going off indefinitely and losing contact with the rest of the team. But I definitely won't require that the output is an estimate of a story.

If this sounds too risky, then good. Innovation is supposed to be risky.

Restore Windows pop-up from Java on Mac OS X

I kept seeing the following pop-up during Java based builds on Mac OS X:

There was no obvious cause, but this was annoying as it paused the build until I dismissed the pop-up, although it never seemed to have any side-effect in the build.

I finally seem to have found the solution, which was to delete the Java related files in the directory ~/Library/Saved Application State:


$ rm -rf ~/Library/Saved\ Application\ State/com.apple.javajdk16.cmd.savedState/
$ rm -rf ~/Library/Saved\ Application\ State/net.java.openjdk.cmd.savedState/