Hello Fediverse

After several years of not blogging, I've started writing the occasional post, but this time on the Fediverse. Visit my new blog on wordsmith.social:

 

  

Postscript: in December 2023, the new blog moved to underlap.org.

The Go-2 Project

Following last week's public launch of the Go-2 project, this post describes the first release and invites contributions. Go-2 is a simplified, extended subset of Google's Go programming language which addresses many of Go's short-comings long recognized by the community. Some of these gaps were highlighted recently in a blog by Gary Willoughby, particularly the lack of generics.

A major design point of Go-2 is that it does not attempt to be fully backward compatible with Go, although the vast majority of cleanly written Go programs will compile and run perfectly under Go-2. See below for incompatible language features. Full backward binary compatibility is however guaranteed.

Go-2 adds generics. The benefits are clear: less duplicated boilerplate code and the ability to ship a general set of collection types (set, bag, map, relation, as well as specific concrete types such as list and tree) in the standard library. Go-2 preserves Go's map syntax, but introduces user-defined operators so that fewer builtins are required and map can be implemented without special support from the compiler.

Go-2 builds on Go's support for first class functions by adding some missing functional programming features: curried functions, partial function application, map/filter operations for applying functions to slices, and monads.

Go-2 also plugs Go's dependency management gap by adding relative imports (essential for centralised version control systems) and semantic versioning of imports. The version numbering format is configurable so, in addition to the usual decimal format, other formats may be used. Roman numerals are particularly intuitive to those coding in Latin character sets. A new Gradle plugin supports Go-2 projects and guarantees repeatable builds.

Go-2 exploits package versioning to introduce the notion of a module to encapsulate a group of packages and selectively export packages for use by other modules. Modules are separately compiled and dynamically linkable from Go-2 main programs.

Go eliminated many common concurrency issues by the prudent use of channels rather than data sharing. Go-2 takes this much further by introducing CSP-semantic verification at the compilation stage. It is virtually impossible to produce programs with any of the concurrency problems that remain: starvation, silent goroutine failures, infinite hidden chatter, etc. Although this extends compilation times somewhat, it makes Go-2 the perfect tool for hyper-concurrent systems. The Go-2 team is currently validating this part of the compiler (using the compiler to analyse itself), though the process is taking longer than expected.

Go-2 drops some Go language features which are usually indicative of dimness. The most notable such feature is the label. The break and continue keywords are supported, but only apply to the innermost loop and do not take a label. In a concession to Go programmers who still use the feature (e.g. here), the goto statement is supported, but in a limited form: goto takes an optional relative line number as its argument. If the line number is omitted, control flows to the next statement. Go's "select {}" is replaced by the more natural "goto 0".

Go-2 simplifies two confusing features of Go. nil values really are nil regardless of how they are initialised. Also, unexported methods may no longer be defined in exported interfaces unless they are also re-implemented local to use. Such interfaces can then be mocked, but only if their private methods are also implemented locally. This behaviour turns out to be much more intuitive.

The Go-2 compiler is written in Go-2 with semantics defined in the pure functional subset of Go-2, with monads. This approach simplifies the process of bootstrapping the compiler and, thanks to the use of a continuation monad, makes it trivial to add advanced features such as atomic panics and user defined deferrals. A Go-2 language specification will be provided for users not comfortable with functional programming. "Effective Go-2" will follow up with examples of idiomatic coding style and corrections to the language specification.

An experimental area of Go-2 is the introduction of bytecode to deliver platform-independent modules and avoid the complexities of cross-compilation. This is especially important for modules in the standard library, which would otherwise need to be compiled to multiple target architectures. The bytecode interpreter already runs reasonably efficiently but, once the project has addressed certain branding issues with the Go-2-in-time compiler (provisionally named "git"), we anticipate bytecode to deliver superior performance to compiled binaries via a couple of innovative features. Firstly, the git compiler is itself packaged as a bytecode module. Secondly, git (nicknamed "gradually-in-time" compiler by the project team) combines speculative partial recompilation with machine learning to deliver continual optimisation. These features trade off start-up speed against ultimate performance. Since the git compiler is itself subject to speculative recompilation, the performance of application code accelerates over time, as shown in the graph below:

The following features were deemed too boring for the initial release of Go-2, but are now eagerly anticipated:

• ternary ("bool ? trueValue : falseValue") and quaternary expressions ("*bool ? trueValue : falseValue ! nilValue"),
• non-shadowing short variable declaration syntax (::=) which will fail to compile if any variables are shadowed (anywhere),
• user-defined deferrals,
• the identity monad.

If you are interested in contributing to Go-2, please join the project mailing list or simply email a patch for the project's central cvs repository. The project name is still to be finalised. Currently "Go-2" is marginally ahead of "Go++", "Go#", and "Go-II". Go-2 is distributed under GPL v3.

A simple dependency injection convention for Go

Steve Powell and I are experimenting with a minimal DI convention for Go which simplifies how we construct values of Go interfaces and use mocks of dependencies during testing.

We are building a layered component and need to test the packages in isolation from each other. So we've introduced interfaces. Each interface usually has one exported creation function named something like NewXXX() which constructs a value of the interface. Packages in higher layers depend on interface values from packages in lower layers and so we pass these values into the creation function.

These NewXXX()functions are tied to the implementation of a package, so we wanted some way to organise the way in which these functions are used. The convention makes these functions private, with names like newXXX(), and then provides separate wiring functions which are used by higher layers and test code.

Although we work for SpringSource, we were hesitant to introduce code generation or reflective "magic" to do the wiring as those seemed contrary to Go's design philosophy of keeping things simple (even if a bit more code needs writing). So we made sure the wiring functions are patterned and simple to read and write.

The lowest layers have trivial wiring functions, for example:

func Wire() (Fileutils, error) {

    return WireWith()

}

func WireWith() (Fileutils, error) {

    return newFileutils()

}

whereas higher layers are somewhat more involved although extremely patterned and the wiring functions simply call lower level wiring functions, for example:

func Wire(depotPath string, rwBaseDir string) (warden.Backend, 

                                               error) {

    rootfs, err := rootfs.Wire(rwBaseDir)

    if err != nil {

        return nil, err

    }

    configBuilder, err := config_builder.Wire()

    if err != nil {

        return nil, err

    }

    return WireWith(depotPath, rootfs, configBuilder)

}

func WireWith(depotPath string, rootfs rootfs.RootFS, configBuilder 

              config_builder.ConfigBuilder) (warden.Backend, 

                                             error) {

return newGuardianBackend(depotPath, rootfs, configBuilder)

}

Note: the examples above use standard Go errors, but our code actually uses a special error type described in a previous blog.

For a detailed description of the convention, see this commit log.

Scoping the libcontainer API

In order to use libcontainer as the basis for a new Warden backend, it needs upgrading. The requirements for a Container API over and above what libcontainer already provides are:

1. Multiple user processes: It must be possible to run multiple user processes in the same Container, either serially or in parallel. All user processes must run in the Container’s namespaces and control groups (so that they are all subject to the same isolation requirements and resource limits). User processes are peers in the sense that they share a common parent and one may terminate without terminating any others.

2. Container life cycle: It must be possible to create a new Container with no user processes. A Container should continue to exist after user processes have terminated.

3. Dynamic reconfiguration: It must be possible to reconfigure the Container after it has been created. There are some fundamental limitations in what can be reconfigured, but these must be kept to a bare minimum. In particular, it must be possible to reconfigure the Container’s control groups and network settings.

4. Container file system: It must be possible to share a single read-only root file system across multiple Container instances. A Container’s file system must be updatable and any updates made in one Container instance must not be visible to other Container instances.

5. Copying and streaming files: It must be possible to copy files and directories between the host and Container file systems. It must also be possible to stream files from the Container’s file system back to the host’s, for example to tail a log file.

Some of these requirements turn out to be beyond the scope of libcontainer as envisaged by its maintainers. In particular, since libcontainer's only consumer is Docker and Docker deals with file system layers, libcontainer avoids managing the read-write layer necessary to satisfy requirement 4. Requirement 5 is closely related to requirement 4.

Similarly, naming and managing collections of Container instances is deemed to be out of scope for libcontainer. So we envisage building an "ideal" Container API at a higher level than libcontainer and reusing libcontainer to provide the core function for a single Container:

It's conceivable that in the future when libcontainer has more consumers, it could step up to requirements 4 and 5, in which case it may be possible to move some of our higher level components down into libcontainer.

With the scope of libcontainer clarified, we are producing another revision of our API proposal which should be ready tomorrow. The first version was captured in a Google document; the next version will be a pull request.

Containers in Cloud Foundry: warden meets libcontainer

Pivotal has decided to merge Cloud Foundry's container technology with Docker's. I'm delighted to be working on this project with my friend and colleague Steve Powell. This blog sketches our initial thoughts.

Steve and I have been exploring how to improve CF's Warden container component to make it a suitable base for future maintenance and extension. Warden is functionally rich, but needs some improvements in robustness, diagnostics, testing, and so on. Rewriting Warden was initially attractive, but would be unlikely to gain much traction in the open source community. Reusing Docker's container technology turned out to be the best option.

Now that Docker has split out the libcontainer project, basing a new Warden Linux backend on libcontainer is feasible: CF will benefit from container backend improvements in libcontainer and libcontainer will be strengthened by supporting the CF use case.

Introduction to Warden

Warden defines an external interface in terms of Google protocol buffers and implements a server which receives protocol buffer requests from clients such as Cloud Foundry's Droplet Execution Agent (DEA) component:

Detailed container management is delegated to a warden backend. There are backends for Linux and Windows. The Linux backend creates containers each of which contains a warden shell daemon (wshd) which is responsible for managing applications and performing other functions inside the container. The backend connects to wshd via TCP/IP using a file-based socket. The container is implemented in terms of Linux primitives such as namespaces and control groups.

Introduction to libcontainer

The docker daemon delegates to an execution driver to create and manage containers. The default "native" execution driver is based on the reusable library of container management functions known as libcontainer:

libcontainer implements its containers in terms of similar Linux primitives to those used by Warden: namespaces, control groups, etc.

The Way Forward

We plan to extend libcontainer on two fronts:

• functionally - to close the gaps relative to warden, and
• non-functionally - to improve robustness, maintainability, and serviceability.

We'll replace the existing Warden Linux backend with a new Warden backend based on the extended libcontainer. We hope that Docker will also be able to exploit the extensions to libcontainer, for instance by exposing new monitoring, management, and diagnostic capabilities.

In simple terms, the new Warden backend will have a layered structure:

Note that the libcontainer API does not yet exist, but is currently being proposed and discussed.

Initially, we plan to use the unextended libcontainer API to launch an agent which will play a similar role to that which wshd did previously:

Eventually, it may be possible to merge the agent functions into libcontainer, which would make the agent's active management functions more easily available to other users of libcontainer, such as Docker:

Gathering diagnostic context: an improved error idiom for Go

After discussing Better error handling idioms in Go and casting around a while, nothing turned up which was ideally suited to our project, so a colleague and I implemented the gerror package which captures stack traces when errors are created and enables errors to be identified without relying on the error message content.

So how does it look to a user? The first code to use it is a fileutils package which implements a file copy function in pure Go (no "shelling out" to cp). Let's take a look at a typical piece of error handling:

    src, err := os.Open(source)
    if err != nil {
        return gerror.NewFromError(ErrOpeningSourceDir, err)
    }

What does this achieve over and above the normal Go idiom of simply returning err if it is non-nil?

Firstly, it captures a stack trace in the error which appears when the error is logged (see below for an example). This gives the full context of the error which, in a large project, avoids guesswork and saves time locating the source of the error.

Secondly, it associates a "tag", in the form of an error identifier, with the error. Callers can use the tag if they want to check for particular errors programmatically:

    gerr := fileutils.Copy(destPath, srcPath)
    if gerr.EqualTag(fileutils.ErrOpeningSourceDir) {
        ...
    }

Thirdly, the resultant error conforms to the builtin error interface and so can be returned or passed around wherever an error is expected.

Fourthly, by defining a function's error return type to be gerror.Gerror, the compiler prevents a "vanilla" error being returned accidentally from the function, which is useful when we want to ensure that all errors have stack traces and tags.

So how are error identifier tags defined? It's easy, as this code from fileutils shows:

type ErrorId int

const (
  ErrFileNotFound ErrorId = iota
  ErrOpeningSourceDir
...
)

Note that this has an advantage over the approach of using variables to refer to specific errors - variables can be overwritten (see this example of issue 7885), whereas constants cannot.

When errors are constructed, the gerror package stores the tag and its type. Both the tag and its type are included in the error string (returned, as usual, by the Error method) and are used when checking for equality in the EqualTag method.

The tag type is logically of the form package.Type which could be ambiguous if two packages had the same name, but the stack trace avoids the ambiguity. For example the following stack trace of a "file not found" error makes it clear that the tag type fileutils.ErrorId refers to the type in the package github.com/cf-guardian/guardian/kernel/fileutils:

0 fileutils.ErrorId: Error caused by: lstat /tmp/fileutils_test-027950024/src.file: no such file or directory
goroutine 8 [running]:
github.com/cf-guardian/guardian/gerror.NewFromError(0xe49a0, 0x0, 0x3484c8, 0xc21000aa80, 0x3484c8, ...)
/Users/gnormington/go/src/github.com/cf-guardian/guardian/gerror/gerror.go:68 +0x8d
github.com/cf-guardian/guardian/kernel/fileutils.fileMode(0xc21000a960, 0x26, 0xc21000a9c0, 0x29, 0x0)
/Users/gnormington/go/src/github.com/cf-guardian/guardian/kernel/fileutils/fileutils.go:196 +0x6b
github.com/cf-guardian/guardian/kernel/fileutils.doCopy(0xc21000a9c0, 0x29, 0xc21000a960, 0x26, 0xc21000a960, ...)
/Users/gnormington/go/src/github.com/cf-guardian/guardian/kernel/fileutils/fileutils.go:68 +0x87
github.com/cf-guardian/guardian/kernel/fileutils.Copy(0xc21000a9c0, 0x29, 0xc21000a960, 0x26, 0x29, ...)
/Users/gnormington/go/src/github.com/cf-guardian/guardian/kernel/fileutils/fileutils.go:61 +0x14f
                        ...

The gerror package is available as open source on github and is licensed under the Apache v2 license. It's really a starting point and others are free to use it "as is" or adapt it to their own needs. If you have an improvement you think we might like, please read our contribution guidelines and send us a pull request.

Better error handling idioms in Go

How often have you seen, or written, Go code like this?

file, err := os.Open("someFile")

if err != nil {

    return err

}

Explicit, inline error handling is necessary since Go doesn't have exceptions. The code is sufficient for small programs, even if the error is returned from more than one level of function call. Hopefully at some point the error is logged and it's fairly easy then to guess what caused the error, especially since os.Open returns a failure helpful error string, for example:

"open someFile: No such file or directory"

However, in larger programs, this approach breaks down. There tend to be too many calls which could have returned the error and (an unbounded amount of) work has to be done to isolate the failing call.

We'd like to see a stack trace, so let's add one as soon as we detect the error.

file, err := os.Open("someFile")

if err != nil {

    var stack [4096]byte

    runtime.Stack(stack[:], false)

    log.Printf("%q\n%s\n", err, stack[:])

    return err

}

The resultant log looks something like this (at least in the playground):

2009/11/10 23:00:00 "open someFile: No such file or directory"

goroutine 1 [running]:
main.main()
 /tmpfs/gosandbox-xxx/prog.go:15 +0xe0
runtime.main()
 /tmp/sandbox/go/src/pkg/runtime/proc.c:220 +0x1c0
runtime.goexit()
 /tmp/sandbox/go/src/pkg/runtime/proc.c:1394

which is better than simply seeing the error message from os.Open.

Clearly this is too much code to write after each call, but some of the code can be moved into a custom error type. (Also 4K isn't enough to capture deep stack traces which is a shame when there is enough free memory available. Maybe there's room for improvement in the runtime package?)

An important consideration is that of "soft" errors - errors which don't appear to need diagnosing at one level of the stack, but which turn out to be more serious from the perspective of one (or more) of the callers. It will probably be too expensive to capture a stack trace every time an error is detected. But it may be sufficient for the first caller which regards the error as serious to capture the stack trace. The combination of a stack trace of this caller and a reasonably helpful error message may be good enough in most cases of soft errors.

Another consideration is logging of errors. It can be very distracting to see the same error logged over and over again. So it might be necessary to keep state in an error to record whether it has already been logged.

I'm interested to hear what error handling practices are evolving in the Go community. An early blog acknowledges the problem:

It is the error implementation's responsibility to summarize the context.

but doesn't address the difficulty of large codebases where the immediate program context isn't always sufficient to diagnose problems.

Some will argue for adding exceptions to Go, but I think that may be overkill, especially for soft errors. I like explicit error handling as it encourages good recovery logic. However, there may be room for improvement in the way the context of an error can be captured. Let's see what nice idioms are beginning to emerge...

Prototype Virgo charm

In the previous post, I mentioned a use case for the Virgo buildpack of running Virgo as an Ubuntu Juju charm. With some advice from the guys at Canonical, for which I'm grateful, a first prototype is now working.

The purpose of a charm is to instantiate a service in a cloud. I'm testing on Amazon EC2, although the charm should run without change on other clouds. The crucial contents of a charm are a simple metadata description and a series of hooks to control the lifecycle of the service associated with the charm and its relationships with other entities in the cloud.

So far, I've implemented two simple hooks. An install hook downloads and installs a JRE and Virgo as well as a test application. It does this by installing the test application and then running the Virgo buildpack compile operation against the application, which in turn downloads and installs a suitable JRE and Virgo (actually, Virgo Server for Apache Tomcat). A start hook starts Virgo using a command similar to that output by running the Virgo buildpack release operation against the test application.

The following pieces of work are necessary to tidy up the prototype:

• Make the start hook reuse the output of the release operation which is a piece YAML containing the basic shell command necessary to start Virgo. It will probably be simplest to rewrite the start script in Ruby so that parsing the YAML file will be trivial.
  
• Somehow synchronise the termination of the start script with the completion of starting Virgo. Currently, the start hook "fires and forgets" using an asynchronous invocation of the Virgo start script. The result is that the start hook returns before Virgo has started, which gives the wrong impression of the state of the charm, especially if Virgo startup fails for any reason.
  
• Implement any other hooks which are required as a bare minimum, including, presumably, a stop hook.
  
• Provide a mechanism to deploy a specified application to Virgo in place of the test application.

Please see the list of issues on github for all known bugs and restrictions.

There is also the small matter of the charm being checked into git. It is currently necessary to delete the .git directories of the charm itself and of the Virgo buildpack submodule since Juju uses git to version the uploaded charm and .git directories cause the install hook to fail. Canonical have some better support for git in mind, so this may just be a question of waiting.

If you want to try out the charm, you can clone it from github (tagged at the time of writing as v0.1). I've put some simple management scripts (which could also do with a bit of tidying up) in the scripts subdirectory which allow me to refresh a copy of the charm without .git directories, start up the Juju environment, deploy the charm, and tear down the Juju environment (currently the only way to stop the Virgo without using the EC2 console).

I've licensed the code under the Apache v2 license and would be delighted to accept contributions if anyone would like to raise issues, make some of the changes mentioned above, or otherwise improve the charm.

Virgo Buildpack Refreshed

I'm no longer leading the Virgo project, but I am still fond of Virgo, so I thought I would update the Virgo buildpack (previously mentioned in this blog) in my own time to re-base it on the current Java buildpack.

It now inherits all the benefits of the Java buildpack (more here):

• Ability to upgrade Virgo without changing the buildpack or the application - great when a security fix needs to be applied.
• Automatic calculation of JVM memory settings.
• Clean separation of components - Virgo is a new container (described here) and the other components including OpenJDK support are inherited for free.
• Helpful diagnostics including automatic JVM destruction on "out of memory" and debug logging of the buildpack's operations.

I decided to stick to the most general style of application layout that the previous version of the Virgo buildpack used. This consists of a pickup directory containing zero or more applications and, optionally, a repository/usr directory containing any additional dependencies of the application which are not provided by Virgo. This has the advantage that the detection criterion for Virgo (existence of the pickup directory) does not overlap with the detection criteria of the other containers inherited from the Java buildpack (Groovy, Play, Java Main, Spring Boot CLI, Tomcat) and so the Virgo buildpack is a functional superset of the Java buildpack.

There are two use cases that interest me. One is running Virgo on Cloud Foundry. The other is running Virgo as an Ubuntu Juju Charm - but more of that on another occasion.

Code Climate

We are using Code Climate to check the quality of the Ruby code in our Cloud Foundry Java buildpack. It's an excellent tool, but some improvements would really help.

1. Scans need to be triggered by git push so that I can rely on notifications rather than prodding/polling.

2. Sometimes code duplication detection is a bit harsh, IMO. I'd like an option to annotate a class/method to allow duplication. Filtering out whole files is too coarse-grained. For example, the following methods flag as duplicated but factoring out the duplication would, IMO, damage the readability of the code:

3. I'd like to be able to configure the "Repos" page to show just some of my projects for a quick team-specific summary. Preferably I'd like to be able to switch easily between filtered views.

4. I'd like to be able to add email addresses to the notification configuration, e.g. mailing lists and mail aliases so multiple people can be notified easily.

Virgo in the Cloud: a CloudFoundry buildpack

The recent Virgo community survey showed quite a bit of interest in running Virgo in the cloud. With CloudFoundry's forthcoming "buildpack" support, this was relatively easy to achieve.

A Virgo buildpack is available on github - see the README for detailed information. It currently supports two types of application: web applications or "overlays". A web application may be a standard WAR file or an OSGi Web Application Bundle. An overlay is a pair of directories -- pickup for the application(s) to be deployed and repository/usr for any additional dependencies not already provided by Virgo -- which are placed in an instance of Virgo Server for Apache Tomcat.

The support for buildpacks should ship later this month - see the CloudFoundry roadmap.

Virgo survey results

Thanks to the ninety-seven people who filled in the Virgo survey. The survey is now closed and the results are in. I'll summarise the results here, but the raw data (in Open Document Spreadsheet format) are available for anyone who wants more detail.

88% of respondents are on Virgo 3.5.0 or later while only 5% are still using SpringSource dm Server.

Virgo Server for Apache Tomcat is by far the most popular runtime deliverable. I would expect this to change over time if SAP's cloud strategy based on Nano Web is successful, even if such success doesn't feature much in Virgo survey results.

A greater respondents (36%) are in production this time, compared to 23% in May 2011.

OSGi and Spring are both preferred programming models with Java EE being somewhat less preferred.  Admittedly this question doesn't capture those who are forced to use a programming model they would prefer not to, but hopefully that number is small given the choice of programming model provided by Virgo.

The future directions were all of interest, some a little more than others. I'm not going to comment in detail, but better support for OSGi standard applications is particularly strong with a high "Gimme gimme" ranking and a tiny snooze factor.

The written comments were illuminating on the whole. Here are the comments for question 7 "Any other future direction(s) you'd really like to see".

Better Maven integration: Automatic deployment, Automatic installation, etc. Better Documentation on How To do integration tests of a complex Web Appplication in Virgo.

I would like to see a better testing framework for Virgo unit and integration testing.

Ensuring that the tooling follows the development of the Virgo server, because it's quite slow right now and often leads to some null pointer that imply reconfiguring the server in STS :(

reliability and troubleshooting are a major concern. As Virgo exposes OSGi to applications - it's harder to debug classloading problems, compared to pure Tomcat or Jetty environments. xx

Question 11 "Anything else to get off your chest?" produced quite a bit of feedback. The tooling is fairly criticised, so we are hoping the upcoming 1.0.1 release will fix a number of common problems. There are repeated calls for sharing of experience with real world applications and technologies, something I feel the Virgo community could helpfully contribute, for instance via blogs or on the Virgo wiki: I'd invite anyone interested to go ahead and add to the Community section. Meanwhile, we have been revamping the samples and will be issuing a milestone soon.

Integration testing made easy (we talked about cargo integ' on the forum and it's for me the priority to leverage Virgo as a top level platform).

Virgo was not simple to shrink, we ended up with a number of components that we could not easily throw out. We did eventually get the right isolation and app working, but the project failed for other non technical reasons. There are generally too many frame works in java, and the owners of java have let opportunity to fix jar hell slip too many times. Look at other language ecosystems as examples. Erlang makes it super easy for operators to update code in a system.

Virgo: Don't push yourself so hard today. Try to find some relaxation time. If you had big plans, perhaps you should cancel them. Instead, schedule some time with friends or family. Or treat yourself to a little indulgence. Some wine tasting here or appetizers there won't hurt as long as you don't go overboard.

We really would like to see a much more stable version of the virgo tooling. Virgo must be restarted way too often during deployment because of tooling problems.

Move away from Spring. JavaEE + OSGi is a better combination. Nice to see that supported in the latest(?) release.

Generally I'm sorry to say I just don't see a future for Virgo. The container is too hard to use and the complexity is too much for the benefits delivered. Gettting a working build and keeping it working with manifest manipulation is still too hard and error prone. It seems unlikely it will ever get fixed or improved. The "next big thing" seems to be Typesafe. But for the most part, containers are going away, and we're going to move to something like AMQP based services. In fact, you can get JVM isolation just by running more JVMs in the cloud. So the unit of deployment is the Virtual Machine, not an OSGi bundle. Sorry for the negative feedback, personally I have always liked OSGi, but for most developers it's too hard to use for the benefits derived.

The first one thanks for your job, and support of this great Software called Virgo. We are developing in OSGi and Virgo about two years, developing Konekti aka Modular Web business platform. We suffer the bad initial tools in eclipse to develop, actually are better. Of course we would like more documentation :) We think it would be interesting adding any blog or tutorials about integration of other frameworks, or experience with Virgo. Our company spent many time integrating many frameworks like JBMP, Drools, ActiveMQ, Jersey, GWT, Vaadin, Jasper Reports, Quartz, ... , so we could provide our experience. Also we would like to know this experience of other companies that also are developing in OSGi and Virgo. We think that Virgo and OSGi in general are a great technology, but not many people knows about it, so any pedagogical work is necessary, and any must lead it. If in the future you add some meeting point, we can contribute it with our real experience in OSGi and Virgo. Best Regards.

Without Eclipse Virgo we would probably not have choosen to try OSGi in my project. Because we need to migrate a Java web application with quite a lot dependencies.

My take on the future - you need to branch out and ensure as much of the available Java technologies are instantly usable in Virgo. Classify the technologies (something like java-source.net does, but better) and pick a project for each one to start with, involve the community and build a library of addons and examples - you'll not only prevent users from bailing on using Virgo because of too much program infrastructure integration effort, but do just the opposite - attract them with ready-to-use examples and simple working projects. I started playing with Virgo since about 7 months. I am using Virgo to rebuild an existing security application platform which we provide to our customers and i find it very attractive to use for me as a developer. Using Virgo as an application server platform opens up a huge potential to me and allows me to build real modular applications based on osgi without having to deal with osgi runtime basics to get started, which is great. The features Virgo provides are good and do make sense to have in such an environment. Although, as mentioned above, Virgo Tooling should be improved and in my opinion this is the key element to attract more developers for using Virgo.

We use it as a managed osgi runtime and install everything into one region for sanity. Virgo rewriting bundle imports causes more problems than they are worth. Using different containers instead of a partitioned runtime makes more sense for disparate applications.

Virgo tools need a lot of improvements. In my company we ended up implementing a little set of eclipse plug-ins to perform similar tasks as those offered by Virgo tools. For example, ordering of projects deployed to the server, because whenever you open the Virgo server editor it takes ages to show up and you just have a 5 lines list for ordering bundles, with no possibility to perform a multiple selection.

Nice modular concept. Needs examples to host Virgo Nano in regular J2EE web app

I don't know what I'd do without you guys!

sorry but I feel you're apart from rest of the osgi community. Why not use or contribute to bnd, maven-bundle-plugin... (note : I have no idea about politics) Please keep thing simple, because It's not easy to defend osgi. I try in my compagny, but osgi feel complicated. In fact it's not. Just a problem of image.

First up - it's excellent. We were going to migrate to an OSGI stack but Virgo seems to have addressed a whole bunch of issues we knew we would hit. The down sides are: - eclipse tooling a bit unreliable / immature (still cannot deploy a ) - different approach from target platform standard eclipse stuff (more eclipse general issue maybe) - the whole spring, hibernate, cglib, resolve issues, class not found issues, blah blah is still a nightmare (nature of the game I guess) - *particularly if migrating to virgo* - greenfield may be easier

Hate the fact that there is no book on Virgo yet, hate that using it depends on other projects (Gemini) that also have sparse documentation. Even a simple hello world on one of the technologies can be a waste of a few hours when adapting to Virgo. Felix/Aries start to gain more traction now. No Paxrunner support for Virgo yet.

To me virgo nano with documentation regarding on how to build isolation would be perfect.

The whole jetty thing and not having an answer for regular Jdk app server model.

Virgo is great and has been very reliable. It is OSGi and integrating 3rd party software that is the pain. For example, getting DB2 datasources using a ConnectionPool framework and getting the datasources via JNDI lookup or a Declarative service takes a lot of work to configure. I still have not seen any good documentation about storing encrypted data in external properties files and then having a hook to decrypt the data so that it may be used in a blueprint.xml definition in the context of ${var} substitution. I do not want the id/password used for my database connections to be stored in plain text in some properties file. Seems that there should be a good recipe or API for handling such common security concerns. For the most part I am well pleased with what Virgo offers, and I am in the middle of transforming a JBoss J2EE multi-tier application to a pure OSGi application running under Virgo.

I love Virgo but the learning curve is steep. Not only Virgo's fault but overall to develop SpringDM applications I had many problems since there are many technologies involved none of which is particularly stable. Random problems: Bundles where there which were not seen, then they were seen. Spring services were found then were not found anymore. Then deleting the work directory solved a lot. Then a bundle was found with version 0.0.0 and you have no idea why. Then 1 day later you realise there was a missing dependency. Etc. Debugging is still quite tricky, when things work do work but many times to get there is a huge pain. Anyway Virgo rocks :) Also the problem with many JARs around which are not yet OSGfied. And to OSGi them you have to beg the developers. Or you have to wrap it yourself but then you have to do it everytime there's a release. There are ways to easily OSGfy a plugin but it would be very useful to make the process as seamless as possible and push the community to embrace OSGi.

I really appreciate what you guys doing. But I don't like the current state of maven support. We use maven for every project it is key technology at our company. It was not easy to setup maven multi-module project to work with eclipse virgo ide and be able to successfully build outside eclipse as well. (driving all osgi dependencies from maven side + generating manifest + providing transient dependencies so virgo can see them when we add dependency just to pom.xml) Do not have sources this days is just sad (and make it that bundles which comes with virgo to be source-aware is complete madness) - with maven it is just so easy. Anyway keep up good work.

Its still hard to managed those jar dependencies or change to more recent version of spring without running into conflicts.

Tooling tooling tooling. Is has been the biggest issue and time consuming process. We have finally crafted a build setup that support maven in CI and in Sts that works but at times can be fragile and difficult to troubleshoot. Many many times we ponder if there is a better way and aren't able to find the answer.

Startup time is the major pain point for us.

I think you need to do more community development. Work on describing how people can use vanilla technologies deployed in Virgo and then move into OSGi modules if they want to. My main concern is trying to stay within a technology stack that is either directly portable or easily ported to a different server as an option.

Seems like things took a while to transition to Eclipse, but feeling better about Virgo and tooling by the day. We're most interested in running this on tiny devices, so are happy (and hoping) to see continued support for Nano.

We are very happy with Virgo! We enjoy the fact that's pretty lightweight, easy to configure and provides good performance. A good thing would be if more than one pickup-folder could be defined.

In my opinion Virgo is lacking some really good documentation and programming examples. The examples available are all using maven and not the eclipse pde dependencies and tools, wich are the easiest way to develop within Eclipse.

Virgo Tooling with Eclipse is a nightmare. So many bugs, runtime parameters that can't be changed (try to change MaxPermSize...) or the lib\endorsed directory not included - took us hours to find out. You guys should eat your own dog food and start to use these tools not just in hobby projects.... I guess this would improve quality.

You should put more effort in the Virgo Tooling it's the more frustrating thing in Virgo. It is very slow and buggy. Keep the good work!

I want better virgo tooling in eclipse/sts. Each times i open tooling in eclipse, it's very slow still 30-40secs. If i stop virgo in error during starting then i need kill eclipse.

The technology section of the survey is rather long and hard to summarise, but it will provide useful input for future development. See the raw data if you are interested.

Finally, the quiz results were intriguing. The correct (or should I say my) answers are:

JOnAS was the biggest mystery with 49% responding "Dunno". WebLogic was second with 32%, just in front of WebSphere with 28%. One person thought that Virgo does not use OSGi or explicitly support it, but their preferred programming model was Java EE, so perhaps they see Virgo as just another enterprise server.

No solution for complexity

Neil Bartlett's blog of this title is shocking because it shows how slow to adopt proven technology the computing industry is.

David Parnas wrote his seminal paper "On the Criteria To Be Used in Decomposing Systems into Modules" (available here) in 1972. A large transaction processing monitor was modularised in the 1980's because the cost of servicing it was spiralling out of control. (Precise specifications were written for the new modules to clean up the interfaces and minimise dependencies on module internals, but that's another story.)

The OSGi module system for Java has been mature for at least five years, possibly longer depending on how you count it. Most of the major Java application servers are now constructed of OSGi modules and a number of them expose OSGi for applications to use. And yet we still see complaints such as the one Neil quotes. It seems application developers are stuck in their ways or stuck on systems which prevent them from using modularity.

Of course, modularity will eventually be adopted by all business critical software. There's really no rational alternative. But how long it will take is quite another matter.

Why I hate spikes

The recording of Dan North's recent talk on "Embracing Uncertainty" set me thinking again about what I like and dislike about agile methods such as scrum. As we end the 157th consecutive sprint of the Virgo project, I certainly appreciate the focus and sense of making progress that each sprint brings.

But what I really dislike are spikes.

The principle of a spike sounds fine: carve out some time to do some exploratory work. But I soon find myself too constrained by a spike. One definition of a spike is (emphasis added):

"a story that cannot be estimated until a development team runs a timeboxed investigation. The output of a spike story is an estimate for the original story."

There are some assumptions buried in that definition which I think show some of the problems:

• implicitly, and significantly, a spike is something you do in a sprint
• timeboxed - of course we can't commit an indefinite amount of time to a spike, but a timebox kind of implies that at the end, you'll come up with "the goods"
• estimate - the "goods" are an estimate
• original story - there's an assumption that the spike won't change your perception of what the story should be.

Here's where Dan's presentation helped me understand what was going on. Spikes are an attempt to fit some explorative work into an agile process which is designed for repeatedly churning out reliable software with minimal risk.

My natural way of doing exploratory work is rather different. First of all, I don't presume to know what it is I expect to find. I have some ideas to try out, some questions to answer, or maybe just something I need to learn about (and believe me, the longer I stay in this job, the more ignorant I realise I am).

So the actual "goods" are really answers, understanding, and more ideas. It might be possible to turn some of these into estimates for a story, but that's not really the goal.

What's wrong with trying to fit exploratory work into a sprint, you may ask? I think the psychology of reporting status in a stand-up meeting each day forces me to focus on things which will deliver immediate results. Sometimes the best way to explore something is to follow a number of lines of inquiry, get yourself thoroughly overwhelmed, take a break, and then (after several pennies have hopefully dropped), start to understand the underlying concepts.

Worse still, if I have at least one non-trivial task in the sprint besides a spike, it's really hard to get into the exploratory frame of mind. The only way I've found is to get the other stuff out of the way as quickly as possible and then try to switch modes.

So in the future, I propose to approach exploratory work differently. I'll treat it as something that's done outside a sprint. I'll still keep the idea of a timebox - it's no good going off indefinitely and losing contact with the rest of the team. But I definitely won't require that the output is an estimate of a story.

If this sounds too risky, then good. Innovation is supposed to be risky.

Restore Windows pop-up from Java on Mac OS X

I kept seeing the following pop-up during Java based builds on Mac OS X:

There was no obvious cause, but this was annoying as it paused the build until I dismissed the pop-up, although it never seemed to have any side-effect in the build.

I finally seem to have found the solution, which was to delete the Java related files in the directory ~/Library/Saved Application State:


$ rm -rf ~/Library/Saved\ Application\ State/com.apple.javajdk16.cmd.savedState/
$ rm -rf ~/Library/Saved\ Application\ State/net.java.openjdk.cmd.savedState/

Eclipse Bundle Recipes project to supersede SpringSource Enterprise Bundle Repository (EBR)

IBM, Paremus, Rebaze, Red Hat, and VMware/SpringSource, with interested parties Peter Kriens and SAP, have submitted an Eclipse Bundle Recipes project proposal aiming to provide a repository of OSGi bundle templates for open source JARs. The intention is that this will supersede the SpringSource Enterprise Bundle Repository (EBR).

It's been nearly a couple of years since we made the templates from the SpringSource EBR available on github. Apart from several pull requests from Raman Gupta (a.k.a. rocketraman, pictured below), not much appeared to happen...

In fact, quite a lot was happening behind the scenes. I was in touch with various colleagues in other companies to see if there was interest in creating a community-based replacement for the SpringSource EBR. It turned out that several large vendors did want to join in, but some of them were very cautious about the licenses of the JAR files that the project would handle. For this reason, we went in the direction of a project to host just the manifest templates.

Clearly, this wasn't going to be much use to people needing to download OSGi bundles, so we also discussed how to make the resultant bundles available. One approach was to generate the bundles on the client machine to cope with most, if not all, open source licenses. The downside of this is the potential lack of reproducibility each time a bundle is generated.

The more people we talked to, the more interest there was in storing the generated bundles in Maven Central. It turns out this is exactly what the Pax Tipi project over at OPS4J started doing last year. So we hope to build on their experience, but perhaps achieve a higher profile.

So the plan is to collaborate on templates at the Eclipse Bundle Recipes project, but also have these published to Maven Central. The details need to be worked out, but now that the Eclipse project proposal has been submitted, we can finally talk about this in public. The project fits naturally into the Eclipse Runtime umbrella project.

Interestingly, another resource for people wanting to add OSGi manifests to existing project has recently become available - the metadata advice bugzilla category at the OSGi Alliance. This should be a valuable resource for anyone wanting to develop a high quality OSGi manifest for a third party JAR.

So the scene is finally set for some progress on superseding the SpringSource EBR. Watch this space or, better still, keep an eye on the Eclipse Bundle Recipes project and or get involved. There's a discussion topic on the proposals forum if you want to comment.

Virgo Community Survey

It's a couple of years since we ran the first Virgo survey and the project has moved on a lot since then. So we've just published a new Eclipse Virgo Community Survey. If you use Virgo, I'd really appreciate it if you could take a few minutes to complete the survey. If you are short of time, please just fill in the first six questions and skip the rest.

Question 12 is a fun quiz about the use of OSGi in application servers. You'll probably find it interesting even if you have no interest in Virgo. We'll provide the answers in a few weeks time along with the survey results.

Here's the survey if you want to get going right away.

Virgo 3.6.0 released

Virgo 3.6.0 is available for download. Although the release includes many small enhancements and bug fixes, some major new features are worth highlighting and there is one smaller one I'd like to draw to the attention of Windows users.

Mike Milinkovich, the Executive Director of the Eclipse Foundation, spoke of Virgo 3.6.0 in his recent JAX interview:

"A third important milestone was SAP shipping its Netweaver Cloud offering based on the Eclipse Virgo project. Having a major vendor basing such a significant product on Eclipse runtime technologies is a great endorsement of the work that the Eclipse RT community has been doing for several years."

So, let's look at the new features.

Java Enterprise APIs

The Virgo Nano Web distribution now integrates several open source Java Enterprise API implementations which will reduce the barrier to entry for a large class of existing applications.

The SAP NetWeaver Cloud platform, which uses Virgo Nano Web as its application server, has been certified against the Java EE Web Profile. Although Virgo has not been certified, I'm optimistic that if Virgo eventually gains access to the Web Profile TCK (discussed by the Eclipse Board in June), it shouldn't be too onerous to get it to pass. Until such time as we certify Virgo, we have to be careful not to claim support for the relevant APIs. Please see the release notes for further details of the APIs and implementations involved.

Virgo continues its first class support for Spring by embedding Spring framework 3.1 and supporting Spring framework 3.2 in the kernel-based deliverables.

Virgo users now have the choice of using Spring or Java Enterprise APIs to develop Virgo applications. Alternatively, they are free to take the minimal Nano or kernel deliverables and build their own custom runtimes, like Infor did with their ION server.

See the Virgo feature summary for a comparison of the various Virgo deliverables.

New Web Admin Console

This began life as a skunk-works project by my colleague Chris Frost. The new web admin console is an AJAJ application backed by the Jolokia OSGi agent paired with Gemini Management. Gemini Management and the Virgo kernel publish JMX mbeans which enable Virgo to be managed and inspected. Jolokia provides JSON access to those mbeans which are then available to the JavaScript running in the admin console. Again, more of this in the release notes and an upcoming blog from Chris - meanwhile here's an earlier blog.

We are particularly fond of the OSGi explorer which provides a graphical depiction of package and service wirings. The initial prototype was implemented by David Normington during a short internship here at SpringSource. Below is an example package wiring diagram.

We had lots of fun testing and fixing the admin console (especially on IE), but we are rather pleased with the result. It seems to push the SVG implementations fairly hard though. Take this example of large "fan-out":

If we scroll to the far left on Safari, the Bezier curves are nice and smooth:

Compare the result in Chrome:

Again see the release notes for more details and screenshots.

The master stroke of the new web admin console is that by minimising its dependencies (shown below), it is now available in all Virgo runtimes, right down to Nano, and smoothly degrades on Nano and Nano Web by omitting panels which are only relevant to kernel and above. Where a web container is not provided by the runtime, on Nano and the kernel, we use the Equinox HTTP Service to host the admin console.

Java 7 Support

This is boring but important since Java 6 public updates cease in February 2013. The Hudson jobs for Virgo run under Java 7, but we still do local builds and releases under Java 6 to ensure both versions work well.

Kernel Deployer Pathname Reduction

This is the smaller feature to make the lives of Windows users a bit easier and which satisfied a long-standing requirement.

Firstly, the kernel deployer uses shorter paths for storing copies of deployer artefacts.

Secondly, users can configure the kernel deployer to deploy bundles in packed form, which avoids the potentially long paths involved in exploded directories. Deploying in exploded form is still the default because deploying web applications packed changes their behaviour: certain servlet API methods return null when a web application is deployed packed. See "Configuring Deployment" in the section "Configuring the Kernel and User Region" in the Virgo User Guide for details.

Conclusion

All in all Virgo 3.6.0 is a significant release and should make Virgo attractive to a broader range of users and applications. We expect Virgo 3.7.0 to improve on 3.6.0 incrementally and we welcome bug and enhancement reports based on your experience of using 3.6.0.

Can you add your project to the Virgo powered list (on the Virgo home page) in 2013?