Steve and I have been exploring how to improve CF's Warden container component to make it a suitable base for future maintenance and extension. Warden is functionally rich, but needs some improvements in robustness, diagnostics, testing, and so on. Rewriting Warden was initially attractive, but would be unlikely to gain much traction in the open source community. Reusing Docker's container technology turned out to be the best option.
Now that Docker has split out the libcontainer project, basing a new Warden Linux backend on libcontainer is feasible: CF will benefit from container backend improvements in libcontainer and libcontainer will be strengthened by supporting the CF use case.
Introduction to Warden
Warden defines an external interface in terms of Google protocol buffers and implements a server which receives protocol buffer requests from clients such as Cloud Foundry's Droplet Execution Agent (DEA) component:
Detailed container management is delegated to a warden backend. There are backends for Linux and Windows. The Linux backend creates containers each of which contains a warden shell daemon (wshd) which is responsible for managing applications and performing other functions inside the container. The backend connects to wshd via TCP/IP using a file-based socket. The container is implemented in terms of Linux primitives such as namespaces and control groups.
Introduction to libcontainer
The docker daemon delegates to an execution driver to create and manage containers. The default "native" execution driver is based on the reusable library of container management functions known as libcontainer:
libcontainer implements its containers in terms of similar Linux primitives to those used by Warden: namespaces, control groups, etc.
The Way Forward
We plan to extend libcontainer on two fronts:
- functionally - to close the gaps relative to warden, and
- non-functionally - to improve robustness, maintainability, and serviceability.
We'll replace the existing Warden Linux backend with a new Warden backend based on the extended libcontainer. We hope that Docker will also be able to exploit the extensions to libcontainer, for instance by exposing new monitoring, management, and diagnostic capabilities.
In simple terms, the new Warden backend will have a layered structure:
Initially, we plan to use the unextended libcontainer API to launch an agent which will play a similar role to that which wshd did previously:
Eventually, it may be possible to merge the agent functions into libcontainer, which would make the agent's active management functions more easily available to other users of libcontainer, such as Docker:
Looks great Glyn!
ReplyDeleteQuestion. Our organization is very interested in the "Security Groups" work being implemented in Cloud Foundry right now. (1)
Although initially it has been decided that security group changes will require an application restart. The idea of making security group changes to a container on the fly is a future enhancement our organization is very interested in. In a previous post of yours (2) you mention that one of the advantages of Warden is its dynamic configuration capabilities. Do you think the move to libcontainer will hurt the ability to have dynamic CF security groups in the future?
1: https://groups.google.com/a/cloudfoundry.org/forum/#!searchin/vcap-dev/security$20groups/vcap-dev/Dy4NVBiasxA/NLsISSGxmpUJ
2: https://groups.google.com/a/cloudfoundry.org/forum/#!searchin/vcap-dev/docker$20vs$20warden/vcap-dev/V-lVpMpNqL4/-kx8qDBPd7MJ
No, not as far as I can tell. The technologies underpinning libcontainer are the same as those underpinning warden, so they should both be capable of supporting the same sorts of changes, at least in principle. Clearly there is a bigger community in libcontainer/docker to approve any change, but also the benefit of more eyes to spot mistakes.
ReplyDeleteOTOH, there may be some practical constraints that get in the way. libcontainer's function is a bit broader than warden's with some support for SELinux, for example, which is likely to complicate any security changes. Also libcontainer seems to be designed to conform to systemd and is also looking towards exploiting the croups unified hierarchy when that ships, so these may also have implications for how easily security groups can be managed. However, even if warden had kept its own container implementation, there were requirements to support SELinux (in order to make some potential exploits more difficult) and also at least tolerate unified hierarchy, which may have drawn warden in the systemd direction. If these requirements had been satisfied by warden, then warden and libcontainer would have had similar complications for the kind of security changes you describe.
Essentially, warden and libcontainer are at similar levels of function and are subject to similar future requirements.
Perhaps you'd like to review the libcontainer API proposal and suggest how to ensure security group changes can be supported in the future?
I see, thanks for the explanation Glyn. I think for the "security groups" needs we don't need cgroups changes to be dynamic. We just need to be able to change the network iptables dynamically (not sure if that falls under the same constraints or not)
ReplyDeleteI would take a look at the api proposal but it would be way over my head. :) All I know is that as an end user I want to be able to change what a container's network interface can and cannot connect to without recreating/restarting the container.
If that isn't possible under the selinux constraints that's fine. Otherwise it would be nice if you could keep that use case in mind as you continue this work.
Thanks,
Mike
Great post and thanks for sharing your plans.
ReplyDeleteAre you going to lend a hand to support unprivileged containers?
Here is the link to PR
Great post and thanks for sharing your plans.
ReplyDeleteAre you going to lend a hand to support unprivileged containers?
Here is the link to PR
We'll be focussing mostly on bringing the libcontainer API up to parity with warden, so I guess that means we will be interested in unprivileged containers.
ReplyDeleteHi!
ReplyDeleteOne thing that is not quite clear to me is whether you guys are going to embrace the docker packaging format as well. IOW, would I every be able to use a Docker registry as a GitHub of sorts for managing the applications that my organization deploys to CF?
Thanks!
That's not on our (Steve's and mine) immediate radar, but I believe some work is going on elsewhere to add dockerfile support to CF and this may involve the docker repository. Probably best for you to ask on vcap-dev.
ReplyDeleteHi,
ReplyDeleteIs this change included in Diego project? I remember in the Diego's architecture, "Garden" will replace "Warden" which means warden is re-written in Golang. Is it still the plan?
Thanks!
In Diego, the Ruby code of Warden has been replaced by Go to form Garden (which is a Warden server minus a backend) and the Warden-linux backend. The Warden-linux backend combined with Garden replaces Warden. Apologies that the terminology is a little confusing!
ReplyDeleteRe-basing Warden-linux on libcontainer is a longer term item. I don't know whether the Diego project will run long enough to include the libcontainer work.