Scoping the libcontainer API

In order to use libcontainer as the basis for a new Warden backend, it needs upgrading. The requirements for a Container API over and above what libcontainer already provides are:

1. Multiple user processes: It must be possible to run multiple user processes in the same Container, either serially or in parallel. All user processes must run in the Container’s namespaces and control groups (so that they are all subject to the same isolation requirements and resource limits). User processes are peers in the sense that they share a common parent and one may terminate without terminating any others.

2. Container life cycle: It must be possible to create a new Container with no user processes. A Container should continue to exist after user processes have terminated.

3. Dynamic reconfiguration: It must be possible to reconfigure the Container after it has been created. There are some fundamental limitations in what can be reconfigured, but these must be kept to a bare minimum. In particular, it must be possible to reconfigure the Container’s control groups and network settings.

4. Container file system: It must be possible to share a single read-only root file system across multiple Container instances. A Container’s file system must be updatable and any updates made in one Container instance must not be visible to other Container instances.

5. Copying and streaming files: It must be possible to copy files and directories between the host and Container file systems. It must also be possible to stream files from the Container’s file system back to the host’s, for example to tail a log file.

Some of these requirements turn out to be beyond the scope of libcontainer as envisaged by its maintainers. In particular, since libcontainer's only consumer is Docker and Docker deals with file system layers, libcontainer avoids managing the read-write layer necessary to satisfy requirement 4. Requirement 5 is closely related to requirement 4.

Similarly, naming and managing collections of Container instances is deemed to be out of scope for libcontainer. So we envisage building an "ideal" Container API at a higher level than libcontainer and reusing libcontainer to provide the core function for a single Container:

It's conceivable that in the future when libcontainer has more consumers, it could step up to requirements 4 and 5, in which case it may be possible to move some of our higher level components down into libcontainer.

With the scope of libcontainer clarified, we are producing another revision of our API proposal which should be ready tomorrow. The first version was captured in a Google document; the next version will be a pull request.

Containers in Cloud Foundry: warden meets libcontainer

Pivotal has decided to merge Cloud Foundry's container technology with Docker's. I'm delighted to be working on this project with my friend and colleague Steve Powell. This blog sketches our initial thoughts.

Steve and I have been exploring how to improve CF's Warden container component to make it a suitable base for future maintenance and extension. Warden is functionally rich, but needs some improvements in robustness, diagnostics, testing, and so on. Rewriting Warden was initially attractive, but would be unlikely to gain much traction in the open source community. Reusing Docker's container technology turned out to be the best option.

Now that Docker has split out the libcontainer project, basing a new Warden Linux backend on libcontainer is feasible: CF will benefit from container backend improvements in libcontainer and libcontainer will be strengthened by supporting the CF use case.

Introduction to Warden

Warden defines an external interface in terms of Google protocol buffers and implements a server which receives protocol buffer requests from clients such as Cloud Foundry's Droplet Execution Agent (DEA) component:

Detailed container management is delegated to a warden backend. There are backends for Linux and Windows. The Linux backend creates containers each of which contains a warden shell daemon (wshd) which is responsible for managing applications and performing other functions inside the container. The backend connects to wshd via TCP/IP using a file-based socket. The container is implemented in terms of Linux primitives such as namespaces and control groups.

Introduction to libcontainer

The docker daemon delegates to an execution driver to create and manage containers. The default "native" execution driver is based on the reusable library of container management functions known as libcontainer:

libcontainer implements its containers in terms of similar Linux primitives to those used by Warden: namespaces, control groups, etc.

The Way Forward

We plan to extend libcontainer on two fronts:

• functionally - to close the gaps relative to warden, and
• non-functionally - to improve robustness, maintainability, and serviceability.

We'll replace the existing Warden Linux backend with a new Warden backend based on the extended libcontainer. We hope that Docker will also be able to exploit the extensions to libcontainer, for instance by exposing new monitoring, management, and diagnostic capabilities.

In simple terms, the new Warden backend will have a layered structure:

Note that the libcontainer API does not yet exist, but is currently being proposed and discussed.

Initially, we plan to use the unextended libcontainer API to launch an agent which will play a similar role to that which wshd did previously:

Eventually, it may be possible to merge the agent functions into libcontainer, which would make the agent's active management functions more easily available to other users of libcontainer, such as Docker:

Prototype Virgo charm

In the previous post, I mentioned a use case for the Virgo buildpack of running Virgo as an Ubuntu Juju charm. With some advice from the guys at Canonical, for which I'm grateful, a first prototype is now working.

The purpose of a charm is to instantiate a service in a cloud. I'm testing on Amazon EC2, although the charm should run without change on other clouds. The crucial contents of a charm are a simple metadata description and a series of hooks to control the lifecycle of the service associated with the charm and its relationships with other entities in the cloud.

So far, I've implemented two simple hooks. An install hook downloads and installs a JRE and Virgo as well as a test application. It does this by installing the test application and then running the Virgo buildpack compile operation against the application, which in turn downloads and installs a suitable JRE and Virgo (actually, Virgo Server for Apache Tomcat). A start hook starts Virgo using a command similar to that output by running the Virgo buildpack release operation against the test application.

The following pieces of work are necessary to tidy up the prototype:

• Make the start hook reuse the output of the release operation which is a piece YAML containing the basic shell command necessary to start Virgo. It will probably be simplest to rewrite the start script in Ruby so that parsing the YAML file will be trivial.
  
• Somehow synchronise the termination of the start script with the completion of starting Virgo. Currently, the start hook "fires and forgets" using an asynchronous invocation of the Virgo start script. The result is that the start hook returns before Virgo has started, which gives the wrong impression of the state of the charm, especially if Virgo startup fails for any reason.
  
• Implement any other hooks which are required as a bare minimum, including, presumably, a stop hook.
  
• Provide a mechanism to deploy a specified application to Virgo in place of the test application.

Please see the list of issues on github for all known bugs and restrictions.

There is also the small matter of the charm being checked into git. It is currently necessary to delete the .git directories of the charm itself and of the Virgo buildpack submodule since Juju uses git to version the uploaded charm and .git directories cause the install hook to fail. Canonical have some better support for git in mind, so this may just be a question of waiting.

If you want to try out the charm, you can clone it from github (tagged at the time of writing as v0.1). I've put some simple management scripts (which could also do with a bit of tidying up) in the scripts subdirectory which allow me to refresh a copy of the charm without .git directories, start up the Juju environment, deploy the charm, and tear down the Juju environment (currently the only way to stop the Virgo without using the EC2 console).

I've licensed the code under the Apache v2 license and would be delighted to accept contributions if anyone would like to raise issues, make some of the changes mentioned above, or otherwise improve the charm.

Virgo Buildpack Refreshed

I'm no longer leading the Virgo project, but I am still fond of Virgo, so I thought I would update the Virgo buildpack (previously mentioned in this blog) in my own time to re-base it on the current Java buildpack.

It now inherits all the benefits of the Java buildpack (more here):

• Ability to upgrade Virgo without changing the buildpack or the application - great when a security fix needs to be applied.
• Automatic calculation of JVM memory settings.
• Clean separation of components - Virgo is a new container (described here) and the other components including OpenJDK support are inherited for free.
• Helpful diagnostics including automatic JVM destruction on "out of memory" and debug logging of the buildpack's operations.

I decided to stick to the most general style of application layout that the previous version of the Virgo buildpack used. This consists of a pickup directory containing zero or more applications and, optionally, a repository/usr directory containing any additional dependencies of the application which are not provided by Virgo. This has the advantage that the detection criterion for Virgo (existence of the pickup directory) does not overlap with the detection criteria of the other containers inherited from the Java buildpack (Groovy, Play, Java Main, Spring Boot CLI, Tomcat) and so the Virgo buildpack is a functional superset of the Java buildpack.

There are two use cases that interest me. One is running Virgo on Cloud Foundry. The other is running Virgo as an Ubuntu Juju Charm - but more of that on another occasion.

Virgo in the Cloud: a CloudFoundry buildpack

The recent Virgo community survey showed quite a bit of interest in running Virgo in the cloud. With CloudFoundry's forthcoming "buildpack" support, this was relatively easy to achieve.

A Virgo buildpack is available on github - see the README for detailed information. It currently supports two types of application: web applications or "overlays". A web application may be a standard WAR file or an OSGi Web Application Bundle. An overlay is a pair of directories -- pickup for the application(s) to be deployed and repository/usr for any additional dependencies not already provided by Virgo -- which are placed in an instance of Virgo Server for Apache Tomcat.

The support for buildpacks should ship later this month - see the CloudFoundry roadmap.

OSGi Running in the Cloud: Virgo in CloudFoundry

I'm just been tryout out Hristo Iliev's support for Virgo in CloudFoundry, described on his blog. The VM took a few hours to download and a few minutes to unzip, but then starting CloudFoundry and deploying an application to a Virgo instance worked first time.

CloudFoundry takes a little while to get started:

But then it's trivial to deploy an application to a Virgo instance:



after which the application, in this case the Virgo web admin console, runs as expected at the application URL http://virgo.vcap.me:

and jconsole shows where the Virgo instance has been created:

The cloud nature of this setup becomes more apparent when another application is deployed. Since I'm lazy, I deployed the same application under a different name:

The application is then available at http://virgo2.vcap.me and jconsole shows there are now two Virgo instances running:

I'm looking forward to Hristo's pull requests become merged into the CloudFoundry base and maybe eventually Virgo showing up in some clouds near you.