tag:blogger.com,1999:blog-346922332024-03-14T01:56:50.382+00:00Mind the GapGlynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.comBlogger227125tag:blogger.com,1999:blog-34692233.post-52374494650570614982022-02-25T17:32:00.003+00:002023-12-09T14:10:22.155+00:00Hello Fediverse<p>After several years of not blogging, I've started writing the occasional post, but this time on the Fediverse. Visit <a href="https://wordsmith.social/@underlap">my new blog on wordsmith.social</a>:</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEieJQHdmVHUzE_PREVrregc3LwWunEFFyGmM2Q3L3ICmm0pPMADdAwYzGWRi4lSR159feCUkGl7_pO_CmH02Dc4-Dc7P4-YA5jKoO7khvX3WDlc--YjGRi0vIWiB5Ga1C3e6iQanj0TwVQxZyfHYa8FVIFfASYU2TAXLav7nJCRr6D5VOYEb2Q=s1720" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="748" data-original-width="1720" height="278" src="https://blogger.googleusercontent.com/img/a/AVvXsEieJQHdmVHUzE_PREVrregc3LwWunEFFyGmM2Q3L3ICmm0pPMADdAwYzGWRi4lSR159feCUkGl7_pO_CmH02Dc4-Dc7P4-YA5jKoO7khvX3WDlc--YjGRi0vIWiB5Ga1C3e6iQanj0TwVQxZyfHYa8FVIFfASYU2TAXLav7nJCRr6D5VOYEb2Q=w640-h278" width="640" /> </a></div><div class="separator" style="clear: both; text-align: left;"> </div><div class="separator" style="clear: both; text-align: left;"><b>Postscript:</b> in December 2023, the new blog moved to <a href="https://underlap.org/">underlap.org</a>.<br /></div><p></p>Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com0tag:blogger.com,1999:blog-34692233.post-54552423727313073012015-04-01T06:04:00.000+01:002015-04-01T08:48:10.801+01:00The Go-2 ProjectFollowing last week's public launch of the Go-2 project, this post describes the first release and invites contributions. Go-2 is a simplified, extended subset of Google's <a href="http://golang.org/">Go</a> programming language which addresses many of Go's short-comings long recognized by the community. Some of these gaps were highlighted recently in a <a href="http://nomad.so/2015/03/why-gos-design-is-a-disservice-to-intelligent-programmers/">blog</a> by Gary Willoughby, particularly the lack of generics.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://ih2.redbubble.net/image.23826428.8000/sticker,375x360.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="Go-2 "toothy" go2phers" border="0" src="http://ih2.redbubble.net/image.23826428.8000/sticker,375x360.png" height="191" title="Go-2 "toothy" go2phers" width="200" /></a></div>
A major design point of Go-2 is that it does not attempt to be fully backward compatible with Go, although the vast majority of cleanly written Go programs will compile and run perfectly under Go-2. See below for incompatible language features. Full backward binary compatibility is however guaranteed.<br />
<br />
Go-2 adds generics. The benefits are clear: less duplicated boilerplate code and the ability to ship a general set of collection types (set, bag, map, relation, as well as specific concrete types such as list and tree) in the standard library. Go-2 preserves Go's map syntax, but introduces user-defined operators so that fewer builtins are required and map can be implemented without special support from the compiler.<br />
<br />
Go-2 builds on Go's support for first class functions by adding some missing functional programming features: curried functions, partial function application, map/filter operations for applying functions to slices, and monads.<br />
<br />
Go-2 also plugs Go's dependency management gap by adding relative imports (essential for centralised version control systems) and <a href="http://semver.org/">semantic versioning</a> of imports. The version numbering format is configurable so, in addition to the usual decimal format, other formats may be used. Roman numerals are particularly intuitive to those coding in Latin character sets. A new <a href="https://gradle.org/">Gradle</a> plugin supports Go-2 projects and guarantees repeatable builds.<br />
<br />
Go-2 exploits package versioning to introduce the notion of a <i>module</i> to encapsulate a group of packages and selectively export packages for use by other modules. Modules are separately compiled and dynamically linkable from Go-2 main programs.<br />
<br />
Go eliminated many common concurrency issues by the prudent use of channels rather than data sharing. Go-2 takes this much further by introducing CSP-semantic verification at the compilation stage. It is virtually impossible to produce programs with any of the concurrency problems that remain: starvation, silent goroutine failures, infinite hidden chatter, etc. Although this extends compilation times somewhat, it makes Go-2 the perfect tool for hyper-concurrent systems. The Go-2 team is currently validating this part of the compiler (using the compiler to analyse itself), though the process is taking longer than expected.<br />
<br />
Go-2 drops some Go language features which are usually indicative of dimness. The most notable such feature is the label. The <span style="font-family: Courier New, Courier, monospace;">break</span> and <span style="font-family: Courier New, Courier, monospace;">continue</span> keywords are supported, but only apply to the innermost loop and do not take a label. In a concession to Go programmers who still use the feature (e.g. <a href="https://golang.org/test/goto.go">here</a>), the <span style="font-family: Courier New, Courier, monospace;">goto</span> statement is supported, but in a limited form: <span style="font-family: Courier New, Courier, monospace;">goto</span> takes an optional relative line number as its argument. If the line number is omitted, control flows to the next statement. Go's "<span style="font-family: Courier New, Courier, monospace;">select {}</span>" is replaced by the more natural "<span style="font-family: Courier New, Courier, monospace;">goto 0</span>".<br />
<br />
Go-2 simplifies two confusing features of Go. <span style="font-family: Courier New, Courier, monospace;">nil</span> values really are <span style="font-family: Courier New, Courier, monospace;">nil</span> <a href="https://golang.org/doc/faq#nil_error">regardless</a> of how they are initialised. Also, unexported methods may no longer be defined in exported interfaces unless they are also re-implemented local to use. Such interfaces can then be mocked, but only if their private methods are also implemented locally. This behaviour turns out to be much more intuitive.<br />
<br />
The Go-2 compiler is written in Go-2 with semantics defined in the pure functional subset of Go-2, with monads. This approach simplifies the process of bootstrapping the compiler and, thanks to the use of a <a href="http://www.haskellforall.com/2012/12/the-continuation-monad.html">continuation monad</a>, makes it trivial to add advanced features such as atomic panics and user defined deferrals. A Go-2 language specification will be provided for users not comfortable with functional programming. "Effective Go-2" will follow up with examples of idiomatic coding style and corrections to the language specification.<br />
<br />
An experimental area of Go-2 is the introduction of bytecode to deliver platform-independent modules and avoid the complexities of cross-compilation. This is especially important for modules in the standard library, which would otherwise need to be compiled to multiple target architectures. The bytecode interpreter already runs reasonably efficiently but, once the project has addressed certain branding issues with the Go-2-in-time compiler (provisionally named "git"), we anticipate bytecode to deliver superior performance to compiled binaries via a couple of innovative features. Firstly, the git compiler is itself packaged as a bytecode module. Secondly, git (nicknamed "gradually-in-time" compiler by the project team) combines speculative partial recompilation with machine learning to deliver continual optimisation. These features trade off start-up speed against ultimate performance. Since the git compiler is itself subject to speculative recompilation, the performance of application code <i>accelerates</i> over time, as shown in the graph below:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgY4bJkM7HiXI9CsaMdVk0y_8vin3S4Ps-IxMBdOmeAfPcZfOVF8JkfARsGAU3MGrY2y460ouUuWoVhnH23xEQAFt8_Lm41He1slPd8uEq4pT5BJjl7d4rxZuYgxtOqHNGD3PTobw/s1600/Screen+Shot+2015-03-31+at+13.37.09.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="MMXV.IV.I" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgY4bJkM7HiXI9CsaMdVk0y_8vin3S4Ps-IxMBdOmeAfPcZfOVF8JkfARsGAU3MGrY2y460ouUuWoVhnH23xEQAFt8_Lm41He1slPd8uEq4pT5BJjl7d4rxZuYgxtOqHNGD3PTobw/s1600/Screen+Shot+2015-03-31+at+13.37.09.png" height="222" title="MMXV.IV.I" width="400" /></a></div>
<br />
The following features were deemed too boring for the initial release of Go-2, but are now eagerly anticipated:<br />
<ul>
<li>ternary ("<span style="font-family: Courier New, Courier, monospace;">bool ? trueValue : falseValue</span>") and quaternary expressions ("<span style="font-family: Courier New, Courier, monospace;">*bool ? trueValue : falseValue ! nilValue</span>"),</li>
<li>non-shadowing short variable declaration syntax (<span style="font-family: Courier New, Courier, monospace;">::=</span>) which will fail to compile if any variables are shadowed (anywhere),</li>
<li>user-defined deferrals,</li>
<li>the identity monad.</li>
</ul>
If you are interested in contributing to Go-2, please join the <a href="mailto:majordomo@go2lang.org"><span id="goog_1776962711"></span>project mailing list<span id="goog_1776962712"></span></a> or simply email a patch for the project's central cvs repository. The project name is still to be finalised. Currently "Go-2" is marginally ahead of "Go++", "Go#", and "Go-II". Go-2 is distributed under GPL v3.Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com8tag:blogger.com,1999:blog-34692233.post-35708520944097467092014-08-19T15:41:00.003+01:002014-08-19T15:41:46.828+01:00A simple dependency injection convention for GoSteve Powell and I are experimenting with a <a href="https://github.com/cf-guardian/guardian-backend/commit/a023fa493953e2010f40419dd2290f84e86d3960">minimal DI convention for Go</a> which simplifies how we construct values of Go interfaces and use mocks of dependencies during testing.<br />
<div>
<br /></div>
<div>
We are building a layered component and need to test the packages in isolation from each other. So we've introduced interfaces. Each interface usually has one exported creation function named something like <span style="font-family: Courier New, Courier, monospace;">NewXXX()</span> which constructs a value of the interface. Packages in higher layers depend on interface values from packages in lower layers and so we pass these values into the creation function.</div>
<div>
<br /></div>
<div>
These <span style="font-family: 'Courier New', Courier, monospace;">NewXXX()</span>functions are tied to the implementation of a package, so we wanted some way to organise the way in which these functions are used. The convention makes these functions private, with names like <span style="font-family: Courier New, Courier, monospace;">newXXX()</span>, and then provides separate <i>wiring functions</i> which are used by higher layers and test code.</div>
<div>
<br /></div>
<div>
Although we work for SpringSource, we were hesitant to introduce code generation or <a href="http://blog.parse.com/2014/05/13/dependency-injection-with-go/">reflective "magic"</a> to do the wiring as those seemed contrary to Go's design philosophy of keeping things simple (even if a bit more code needs writing). So we made sure the wiring functions are patterned and simple to read and write.</div>
<div>
<br /></div>
<div>
The lowest layers have trivial wiring functions, for example:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">func Wire() (Fileutils, error) {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> return WireWith()</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">}</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">func WireWith() (Fileutils, error) {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> return newFileutils()</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">}</span></div>
</div>
<div>
<br /></div>
<div>
whereas higher layers are somewhat more involved although extremely patterned and the wiring functions simply call lower level wiring functions, for example:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">func <b>Wire</b>(depotPath string, rwBaseDir string) (warden.Backend, </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> error) {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> rootfs, err := rootfs.<b>Wire</b>(rwBaseDir)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> if err != nil {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> return nil, err</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> }</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> configBuilder, err := config_builder.<b>Wire</b>()</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> if err != nil {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> return nil, err</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;"> }</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> return <b>WireWith</b>(depotPath, rootfs, configBuilder)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">}</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">func <b>WireWith</b>(depotPath string, rootfs rootfs.RootFS, configBuilder </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> config_builder.ConfigBuilder) (warden.Backend, </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> error) {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>return newGuardianBackend(depotPath, rootfs, configBuilder)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">}</span></div>
</div>
<div>
<br /></div>
<div>
Note: the examples above use standard Go errors, but our code actually uses a special error type described in <a href="http://underlap.blogspot.co.uk/2014/04/better-golang-error-idiom.html">a previous blog</a>.</div>
<div>
<br /></div>
<div>
For a detailed description of the convention, see <a href="https://github.com/cf-guardian/guardian-backend/commit/a023fa493953e2010f40419dd2290f84e86d3960">this commit log</a>.</div>
<div>
<br /></div>
Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com0tag:blogger.com,1999:blog-34692233.post-91337832593573333482014-07-01T16:07:00.001+01:002014-07-01T16:07:24.246+01:00Scoping the libcontainer APIIn order to use libcontainer as the basis for a new Warden backend, it needs upgrading. The requirements for a Container API over and above what libcontainer already provides are:<br />
<br />
<b>1. Multiple user processes:</b> It must be possible to run multiple user processes in the same Container, either serially or in parallel. All user processes must run in the Container’s namespaces and control groups (so that they are all subject to the same isolation requirements and resource limits). User processes are peers in the sense that they share a common parent and one may terminate without terminating any others.<br />
<br />
<b>2. Container life cycle:</b> It must be possible to create a new Container with no user processes. A Container should continue to exist after user processes have terminated.<br />
<br />
<b>3. Dynamic reconfiguration:</b> It must be possible to reconfigure the Container after it has been created. There are some fundamental limitations in what can be reconfigured, but these must be kept to a bare minimum. In particular, it must be possible to reconfigure the Container’s control groups and network settings.<br />
<br />
<b>4. Container file system:</b> It must be possible to share a single read-only root file system across multiple Container instances. A Container’s file system must be updatable and any updates made in one Container instance must not be visible to other Container instances.<br />
<br />
<b>5. Copying and streaming files:</b> It must be possible to copy files and directories between the host and Container file systems. It must also be possible to stream files from the Container’s file system back to the host’s, for example to tail a log file.<br />
<br />
Some of these requirements turn out to be beyond the scope of libcontainer as envisaged by its maintainers. In particular, since libcontainer's only consumer is Docker and Docker deals with file system layers, libcontainer avoids managing the read-write layer necessary to satisfy requirement 4. Requirement 5 is closely related to requirement 4.<br />
<br />
Similarly, naming and managing collections of Container instances is deemed to be out of scope for libcontainer. So we envisage building an "ideal" Container API at a higher level than libcontainer and reusing libcontainer to provide the core function for a single Container:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKRMsw61vS0vTK3zRdKiBbRi_l-fEjElcJNfDe85iiXhRfRfuwPvuFksXx8OSX_kJgDSvXaNwpGsw5EEZYpcFLeER_cNxh0eEYcP-C4N7d_QQnrv86gGfEiinxe2xSHKi_s6MS8A/s1600/idealised+Container.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKRMsw61vS0vTK3zRdKiBbRi_l-fEjElcJNfDe85iiXhRfRfuwPvuFksXx8OSX_kJgDSvXaNwpGsw5EEZYpcFLeER_cNxh0eEYcP-C4N7d_QQnrv86gGfEiinxe2xSHKi_s6MS8A/s1600/idealised+Container.png" height="208" width="640" /></a></div>
<br />
It's conceivable that in the future when libcontainer has more consumers, it could step up to requirements 4 and 5, in which case it may be possible to move some of our higher level components down into libcontainer.<br />
<br />
With the scope of libcontainer clarified, we are producing another revision of our API proposal which should be ready tomorrow. The <a href="https://docs.google.com/document/d/1vBNDhLdwK4YdP6uq08Xrc8bnF1g68oFbcEqDGErIYeA/edit?usp=sharing">first version</a> was captured in a Google document; the next version will be a pull request.Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com0tag:blogger.com,1999:blog-34692233.post-48783021532621814032014-06-19T13:23:00.002+01:002014-06-19T13:23:35.092+01:00Containers in Cloud Foundry: warden meets libcontainer<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgT6hxHktf8itKNPsKCuJSkU-Sx0YTNO9Sq8S4sS-qAH2Efi1K2Pns4fyrFR73Cty97zexnl8Hho4ooK_AQ_cZoZYtQHbXom6sb6MNm1clxVvRZYk8JqT2U3VbnWpWLk3n-iJ_MIw/s1600/CF+docker.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgT6hxHktf8itKNPsKCuJSkU-Sx0YTNO9Sq8S4sS-qAH2Efi1K2Pns4fyrFR73Cty97zexnl8Hho4ooK_AQ_cZoZYtQHbXom6sb6MNm1clxVvRZYk8JqT2U3VbnWpWLk3n-iJ_MIw/s1600/CF+docker.png" height="60" width="320" /></a></div>
<span style="font-family: Times, Times New Roman, serif;"><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Pivotal has decided to merge Cloud Foundry's container technology with Docker's. I'm delighted to be working on this project with my friend and colleague Steve Powell. </span><span style="font-size: 15px; white-space: pre-wrap;">This blog sketches our initial thoughts.</span></span><br />
<span style="font-family: Times, Times New Roman, serif;"><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span id="docs-internal-guid-a209164e-a967-be2d-a90c-43ca05e4ac48"><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Steve and I have been exploring how to improve CF's Warden container component to make it a suitable base for future maintenance and extension. Warden is functionally rich, but needs some improvements in robustness, diagnostics, testing, and so on. Rewriting Warden was initially attractive, but would be</span><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> unlikely to gain much traction in the open source community. Reusing Docker's container technology turned out to be the best option.</span></span></span><br />
<span style="font-family: Times, Times New Roman, serif;"><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Now that Docker has split out the</span></span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span><a href="https://github.com/docker/libcontainer"><span style="font-family: Consolas; font-size: 15px; line-height: 17.25px; text-indent: 21px; white-space: pre-wrap;">libcontainer</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span></a><span style="font-family: Times, Times New Roman, serif; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><a href="https://github.com/docker/libcontainer">project</a>, basing a new <a href="https://github.com/cloudfoundry-incubator/warden-linux">Warden Linux backend</a> on</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> <span style="font-family: Consolas; line-height: 17.25px; text-indent: 21px;">libcontainer</span> </span><span style="font-family: Times, Times New Roman, serif; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">is feasible: CF will benefit from container backend improvements in</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> <span style="font-family: Consolas; line-height: 17.25px; text-indent: 21px;">libcontainer</span> </span><span style="font-family: Times, Times New Roman, serif; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">and</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> <span style="font-family: Consolas; line-height: 17.25px; text-indent: 21px;">libcontainer</span> </span><span style="font-family: Times, Times New Roman, serif; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">will be strengthened by supporting the CF use case.</span><br />
<h3>
<span style="font-family: Times, Times New Roman, serif; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Introduction to Warden</span></h3>
<div>
<span style="font-family: Times, Times New Roman, serif; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Warden defines an external interface in terms of Google protocol buffers and implements a server which receives protocol buffer requests from clients such as Cloud Foundry's Droplet Execution Agent (DEA) component:</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBLspmCuQ_MDfP5_DnZBhRi74YAQ49ZBUgN3vAsly6QQGydFuLLCNi1k-TQyTwYckMuuvQXMHNLXJNxZySfAPKgLnQGSQTAHar-LHwvuYh5-EQzgFB72qkeH47Q44xlHAFZN4chQ/s1600/warden+intro.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBLspmCuQ_MDfP5_DnZBhRi74YAQ49ZBUgN3vAsly6QQGydFuLLCNi1k-TQyTwYckMuuvQXMHNLXJNxZySfAPKgLnQGSQTAHar-LHwvuYh5-EQzgFB72qkeH47Q44xlHAFZN4chQ/s1600/warden+intro.png" height="361" width="400" /></a></div>
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: left;">
Detailed container management is delegated to a warden <i>backend</i>. There are backends for Linux and Windows. The Linux backend creates containers each of which contains a <i>warden shell daemon</i> (wshd) which is responsible for managing applications and performing other functions inside the container. The backend connects to <span style="font-family: Courier New, Courier, monospace;">wshd</span> via TCP/IP using a file-based socket. The container is implemented in terms of Linux primitives such as namespaces and control groups.</div>
<h3>
<span style="font-family: Times, Times New Roman, serif; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Introduction to libcontainer</span></h3>
<div>
<span style="font-family: Times, Times New Roman, serif; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The docker daemon delegates to an <i>execution driver</i> to create and manage containers. </span><span style="font-family: Times, Times New Roman, serif; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The default "native" execution driver is based on the reusable library of container management functions known as</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-family: Consolas; font-size: 15px; line-height: 17.25px; white-space: pre-wrap;"><i>libcontainer</i></span><span style="font-family: Arial; font-size: 15px; white-space: pre-wrap;">:</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwWlmaZatTxW01ZQigI-TtRrhJYk6_o7iJNnbsojNyjXXv2r73EqbRl2MpJnmbUU-_AStPqP7BEyxcJ_tt_W1eR7llJdeYXpYNIvC_GNSAvJHDACInDP5EDFuQKxlCOJTS9B4gPw/s1600/libcontainer+intro.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwWlmaZatTxW01ZQigI-TtRrhJYk6_o7iJNnbsojNyjXXv2r73EqbRl2MpJnmbUU-_AStPqP7BEyxcJ_tt_W1eR7llJdeYXpYNIvC_GNSAvJHDACInDP5EDFuQKxlCOJTS9B4gPw/s1600/libcontainer+intro.png" height="298" width="320" /></a></div>
<span style="font-family: Times, Times New Roman, serif; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Consolas; line-height: 17.25px;">libcontainer</span> implements its containers</span> in terms of similar Linux primitives to those used by Warden: namespaces, control groups, etc.</div>
<h3>
<span style="font-family: Times, Times New Roman, serif; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The Way Forward</span></h3>
<div style="text-indent: 0px;">
<span style="color: black; font-family: Times, Times New Roman, serif; font-size: 15px; text-indent: 15.75pt; vertical-align: baseline; white-space: pre-wrap;">We plan to extend</span><span style="color: black; font-family: Arial; font-size: 15px; text-indent: 15.75pt; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="color: black; font-family: Consolas; font-size: 15px; font-weight: normal; line-height: 1.15; text-indent: 15.75pt; vertical-align: baseline; white-space: pre-wrap;">libcontainer</span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: normal; line-height: 1.15; text-indent: 15.75pt; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="color: black; font-family: Times, Times New Roman, serif; font-size: 15px; font-weight: normal; line-height: 1.15; text-indent: 15.75pt; vertical-align: baseline; white-space: pre-wrap;">on two fronts:</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Times, Times New Roman, serif; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">functionally - to close the gaps relative to warden, and</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Times, Times New Roman, serif; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">non-functionally - to improve robustness, maintainability, and serviceability.</span></div>
</li>
</ul>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif;"><span style="background-color: transparent; color: black; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span><span style="background-color: transparent; color: black; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We'll replace the existing Warden Linux backend with a new Warden backend based on the extended</span></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Consolas; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">libcontainer</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span><span style="font-family: Times, Times New Roman, serif;"><span style="background-color: transparent; color: black; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We hope that </span><span style="font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Docker will also be able to exploit the extensions to</span></span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-family: Consolas; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">libcontainer</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, </span><span style="font-family: Times, Times New Roman, serif; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">for instance by exposing new monitoring, management, and diagnostic capabilities.</span><br />
<span style="font-family: Times, Times New Roman, serif; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="font-family: Times, Times New Roman, serif; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">In simple terms, the new Warden backend will have a layered structure:</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnT1vo9H9INl6nBUZglV04-zCUjbUbtFms9SGMmhD3YxFXVyOxjfrUq9yOJzjdnxYb3lmBC_oM1NK9q2t6UH4OI5aLiJjCk53PFlo_HANBmiHa7LoXfN6guKOtxeMXUdvxz9bJ6g/s1600/garden-future.png" imageanchor="1"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnT1vo9H9INl6nBUZglV04-zCUjbUbtFms9SGMmhD3YxFXVyOxjfrUq9yOJzjdnxYb3lmBC_oM1NK9q2t6UH4OI5aLiJjCk53PFlo_HANBmiHa7LoXfN6guKOtxeMXUdvxz9bJ6g/s1600/garden-future.png" height="376" width="400" /></a></div>
Note that the libcontainer API does not yet exist, but is currently being proposed and discussed.<br />
<br />
Initially, we plan to use the unextended libcontainer API to launch an agent which will play a similar role to that which <span style="font-family: Consolas; font-size: 15px; white-space: pre-wrap;">wshd</span><span style="line-height: 1.15;"> did previously:</span><br />
<div class="separator" style="clear: both; text-align: center;">
<span style="line-height: 1.15;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNv11Rq4w0mt3Nhr-6hGth4QbRCJ6CSVibswIUZybBGrINwA5kL-tR6Ot6sZ0J79fw7nRtkzBGAPd5WG2Z7j9pIwUz7adC7r1sfJGxMovJ41tX74jWtHJtjtHxgbNJpCieyZaejA/s1600/libcontainer+future+0.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNv11Rq4w0mt3Nhr-6hGth4QbRCJ6CSVibswIUZybBGrINwA5kL-tR6Ot6sZ0J79fw7nRtkzBGAPd5WG2Z7j9pIwUz7adC7r1sfJGxMovJ41tX74jWtHJtjtHxgbNJpCieyZaejA/s1600/libcontainer+future+0.png" height="360" width="400" /></a></span></div>
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: left;">
Eventually, it may be possible to merge the agent functions into libcontainer, which would make the agent's active management functions more easily available to other users of libcontainer, such as Docker:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyhv2bF78AZBJDE2RpAwYhtZZzBgn4Yde_j4RZZj6eLus4pwQHPMROwXNg6VhbHcyI_iOzNjYembBmFi8HoQRUsTly-8tfj5Z7XQxTIS_gw7nIU-lxJXdGblmMy3u4wAOUSzzf2w/s1600/libcontainer+future+1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyhv2bF78AZBJDE2RpAwYhtZZzBgn4Yde_j4RZZj6eLus4pwQHPMROwXNg6VhbHcyI_iOzNjYembBmFi8HoQRUsTly-8tfj5Z7XQxTIS_gw7nIU-lxJXdGblmMy3u4wAOUSzzf2w/s1600/libcontainer+future+1.png" height="360" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com10Southampton, UK50.909700400000013 -1.40435090000005450.82961490000001 -1.565712400000054 50.989785900000015 -1.2429894000000541tag:blogger.com,1999:blog-34692233.post-41962188657083221122014-04-29T14:25:00.000+01:002014-04-29T14:33:22.181+01:00Gathering diagnostic context: an improved error idiom for GoAfter discussing <a href="http://underlap.blogspot.co.uk/2014/04/better-error-handling-idioms-in-go.html">Better error handling idioms in Go</a> and casting around a while, nothing turned up which was ideally suited to our project, so a colleague and I implemented the <a href="https://github.com/cf-guardian/guardian/tree/master/gerror">gerror package</a> which captures stack traces when errors are created and enables errors to be identified without relying on the error message content.<br />
<br />
So how does it look to a user? The first code to use it is a <a href="https://github.com/cf-guardian/guardian/tree/master/kernel/fileutils">fileutils package</a> which implements a file copy function in pure Go (no "shelling out" to <span style="font-family: Courier New, Courier, monospace;"><b>cp</b></span>). Let's take a look at a typical piece of error handling:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"> <b>src, err := os.Open(source)</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b> if err != nil {</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b> return gerror.NewFromError(ErrOpeningSourceDir, err)</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b> }</b></span><br />
<br />
What does this achieve over and above the normal Go idiom of simply returning <span style="font-family: Courier New, Courier, monospace;"><b>err</b></span> if it is non-nil?<br />
<br />
Firstly, it captures a stack trace in the error which appears when the error is logged (see below for an example). This gives the full context of the error which, in a large project, avoids guesswork and saves time locating the source of the error.<br />
<br />
Secondly, it associates a "tag", in the form of an error identifier, with the error. Callers can use the tag if they want to check for particular errors programmatically:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"><b> gerr := fileutils.Copy(destPath, srcPath)</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b> if gerr.EqualTag(fileutils.ErrOpeningSourceDir) {</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b> ...</b></span><br />
<b><span style="font-family: Courier New, Courier, monospace;"> }</span></b><br />
<br />
Thirdly, the resultant error conforms to the builtin <span style="font-family: Courier New, Courier, monospace;"><b>error</b></span> interface and so can be returned or passed around wherever an <span style="font-family: Courier New, Courier, monospace;"><b>error</b></span> is expected.<br />
<br />
Fourthly, by defining a function's error return type to be <span style="font-family: Courier New, Courier, monospace;"><b>gerror.Gerror</b></span>, the compiler prevents a "vanilla" error being returned accidentally from the function, which is useful when we want to ensure that all errors have stack traces and tags.<br />
<br />
So how are error identifier tags defined? It's easy, as this code from <span style="font-family: Courier New, Courier, monospace;"><b>fileutils</b></span> shows:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"><b>type ErrorId int</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><br /></b></span>
<span style="font-family: Courier New, Courier, monospace;"><b>const (</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span style="white-space: pre;"> </span>ErrFileNotFound ErrorId = iota</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span style="white-space: pre;"> </span>ErrOpeningSourceDir</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>...</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>)</b></span><br />
<br />
Note that this has an advantage over the approach of using variables to refer to specific errors - variables can be overwritten (see <a href="http://play.golang.org/p/PQPr9jcLqU">this example</a> of <a href="https://code.google.com/p/go/issues/detail?id=7885">issue 7885</a>), whereas constants cannot.<br />
<br />
When errors are constructed, the <span style="font-family: Courier New, Courier, monospace;"><b>gerror</b></span> package stores the tag <i>and</i> its type. Both the tag and its type are included in the error string (returned, as usual, by the <span style="font-family: Courier New, Courier, monospace;"><b>Error</b></span> method) and are used when checking for equality in the <span style="font-family: Courier New, Courier, monospace;"><b>EqualTag</b></span> method.<br />
<br />
The tag type is logically of the form <span style="font-family: Courier New, Courier, monospace;"><b>package.Type</b></span> which could be ambiguous if two packages had the same name, but the stack trace avoids the ambiguity. For example the following stack trace of a "file not found" error makes it clear that the tag type <span style="font-family: Courier New, Courier, monospace;"><b>fileutils.ErrorId</b></span> refers to the type in the package <b style="font-family: 'Courier New', Courier, monospace;">github.com/cf-guardian/guardian/kernel/fileutils</b>:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">0 </span><b style="font-family: 'Courier New', Courier, monospace; font-size: small;">fileutils.ErrorId</b><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">: Error caused by: lstat /tmp/fileutils_test-027950024/src.file: no such file or directory</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>goroutine 8 [running]:</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>github.com/cf-guardian/guardian/gerror.NewFromError(0xe49a0, 0x0, 0x3484c8, 0xc21000aa80, 0x3484c8, ...)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>/Users/gnormington/go/src/github.com/cf-guardian/guardian/gerror/gerror.go:68 +0x8d</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span><b>github.com/cf-guardian/guardian/kernel/fileutils</b>.fileMode(0xc21000a960, 0x26, 0xc21000a9c0, 0x29, 0x0)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>/Users/gnormington/go/src/github.com/cf-guardian/guardian/kernel/fileutils/fileutils.go:196 +0x6b</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>github.com/cf-guardian/guardian/kernel/fileutils.doCopy(0xc21000a9c0, 0x29, 0xc21000a960, 0x26, 0xc21000a960, ...)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>/Users/gnormington/go/src/github.com/cf-guardian/guardian/kernel/fileutils/fileutils.go:68 +0x87</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>github.com/cf-guardian/guardian/kernel/fileutils.Copy(0xc21000a9c0, 0x29, 0xc21000a960, 0x26, 0x29, ...)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>/Users/gnormington/go/src/github.com/cf-guardian/guardian/kernel/fileutils/fileutils.go:61 +0x14f</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> ...</span><br />
<br />
The <span style="font-family: Courier New, Courier, monospace;"><b>gerror</b></span> package is available as open source on github and is licensed under the Apache v2 license. It's really a starting point and others are free to use it "as is" or adapt it to their own needs. If you have an improvement you think we might like, please read our <a href="https://github.com/cf-guardian/guardian/blob/master/CONTRIBUTING.md">contribution guidelines</a> and send us a pull request.Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com0tag:blogger.com,1999:blog-34692233.post-81809421779818065832014-04-11T10:23:00.002+01:002014-04-11T10:23:32.985+01:00Better error handling idioms in GoHow often have you seen, or written, Go code like this?<br />
<div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>file, err := os.Open("someFile")</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>if err != nil {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> return err</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>}</b></span></div>
<div>
<br />
Explicit, inline error handling is necessary since Go doesn't have exceptions. The code is sufficient for small programs, even if the error is returned from more than one level of function call. Hopefully at some point the error is logged and it's fairly easy then to guess what caused the error, especially since <span style="font-family: Courier New, Courier, monospace;"><b>os.Open</b></span> returns a failure helpful error string, for example:<br />
<br />
<pre class="" style="font-size: 15px;"><span class="stdout"><span style="font-family: Courier New, Courier, monospace;"><b>"open someFile: No such file or directory"</b></span></span></pre>
</div>
</div>
<div>
<br /></div>
<div>
However, in larger programs, this approach breaks down. There tend to be too many calls which could have returned the error and (an unbounded amount of) work has to be done to isolate the failing call.<br />
<br />
We'd like to see a stack trace, so let's add one as soon as we detect the error.</div>
<div>
<br /></div>
<div>
<div>
<b style="font-family: 'Courier New', Courier, monospace;">file, err := os.Open("someFile")</b></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>if err != nil {</b></span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> var stack [4096]byte</b></span></div>
<span style="font-family: Courier New, Courier, monospace;"><b>
</b></span>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> runtime.Stack(stack[:], false)</b></span></div>
<span style="font-family: Courier New, Courier, monospace;"><b>
<div>
log.Printf("%q\n%s\n", err, stack[:])</div>
<div>
return err</div>
</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>}</b></span></div>
</div>
<div>
<br /></div>
<div>
The resultant log looks something like this (at least in the playground):</div>
<div>
<br /></div>
<div>
<pre class="" style="font-size: 15px;"><span style="font-family: Courier New, Courier, monospace;"><b>2009/11/10 23:00:00 "open someFile: No such file or directory"</b></span></pre>
<pre class="" style="font-size: 15px;"><pre class=""><span class="stdout"><span style="font-family: Courier New, Courier, monospace;"><b>goroutine 1 [running]:
main.main()
/tmpfs/gosandbox-xxx/prog.go:15 +0xe0
runtime.main()
/tmp/sandbox/go/src/pkg/runtime/proc.c:220 +0x1c0
runtime.goexit()
/tmp/sandbox/go/src/pkg/runtime/proc.c:1394</b></span></span></pre>
</pre>
</div>
<div>
which is better than simply seeing the error message from <span style="font-family: Courier New, Courier, monospace;"><b>os.Open</b></span>.<br />
<br />
Clearly this is too much code to write after each call, but some of the code can be moved into a custom error type. (Also 4K isn't enough to capture deep stack traces which is a shame when there is enough free memory available. Maybe there's room for improvement in the runtime package?)<br />
<br />
An important consideration is that of "soft" errors - errors which don't appear to need diagnosing at one level of the stack, but which turn out to be more serious from the perspective of one (or more) of the callers. It will probably be too expensive to capture a stack trace every time an error is detected. But it may be sufficient for the first caller which regards the error as serious to capture the stack trace. The combination of a stack trace of this caller and a reasonably helpful error message may be good enough in most cases of soft errors.<br />
<br />
Another consideration is logging of errors. It can be very distracting to see the same error logged over and over again. So it might be necessary to keep state in an error to record whether it has already been logged.<br />
<br />
I'm interested to hear what error handling practices are evolving in the Go community. An early <a href="http://blog.golang.org/error-handling-and-go">blog</a> acknowledges the problem:<br />
<blockquote class="tr_bq">
<span style="color: #222222; font-family: Helvetica, Arial, sans-serif; font-size: 16px;">It is the error implementation's responsibility to summarize the context.</span></blockquote>
but doesn't address the difficulty of large codebases where the immediate program context isn't always sufficient to diagnose problems.<br />
<br />
Some will argue for adding exceptions to Go, but I think that may be overkill, especially for soft errors. I like explicit error handling as it encourages good recovery logic. However, there may be room for improvement in the way the context of an error can be captured. Let's see what nice idioms are beginning to emerge...</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com9tag:blogger.com,1999:blog-34692233.post-21000570924033198412013-11-20T03:37:00.002+00:002013-11-30T09:28:14.022+00:00Prototype Virgo charmIn the <a href="http://underlap.blogspot.co.uk/2013/11/virgo-buildpack-refreshed.html">previous post</a>, I mentioned a use case for the Virgo buildpack of running Virgo as an <a href="https://juju.ubuntu.com/">Ubuntu Juju</a> charm. With some advice from the guys at Canonical, for which I'm grateful, a first prototype is now working.<br />
<br />
The purpose of a charm is to instantiate a service in a cloud. I'm testing on Amazon EC2, although the charm should run without change on other clouds. The crucial contents of a charm are a simple metadata description and a series of hooks to control the lifecycle of the service associated with the charm and its relationships with other entities in the cloud.<br />
<br />
So far, I've implemented two simple hooks. An <i>install hook</i> downloads and installs a JRE and Virgo as well as a test application. It does this by installing the test application and then running the Virgo buildpack <i>compile</i> operation against the application, which in turn downloads and installs a suitable JRE and Virgo (actually, Virgo Server for Apache Tomcat). A <i>start hook</i> starts Virgo using a command similar to that output by running the Virgo buildpack <i>release</i> operation against the test application.<br />
<br />
The following pieces of work are necessary to tidy up the prototype:<br />
<br />
<ul>
<li>Make the start hook reuse the output of the release operation which is a piece YAML containing the basic shell command necessary to start Virgo. It will probably be simplest to rewrite the start script in Ruby so that parsing the YAML file will be trivial.<br /></li>
<li>Somehow synchronise the termination of the start script with the completion of starting Virgo. Currently, the start hook "fires and forgets" using an asynchronous invocation of the Virgo start script. The result is that the start hook returns before Virgo has started, which gives the wrong impression of the state of the charm, especially if Virgo startup fails for any reason.<br /></li>
<li>Implement any other hooks which are required as a bare minimum, including, presumably, a <i>stop hook</i>.<br /></li>
<li>Provide a mechanism to deploy a specified application to Virgo in place of the test application.</li>
</ul>
<div>
Please see the <a href="https://github.com/glyn/virgo-charm/issues">list of issues</a> on github for all known bugs and restrictions.</div>
<div>
<br /></div>
<div>
There is also the small matter of the charm being checked into git. It is currently necessary to delete the <span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>.git</b></span> directories of the charm itself and of the Virgo buildpack submodule since Juju uses git to version the uploaded charm and <span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>.git</b></span> directories cause the install hook to fail. Canonical have some better support for git in mind, so this may just be a question of waiting.</div>
<div>
<br /></div>
<div>
If you want to try out the charm, you can <a href="https://github.com/glyn/virgo-charm">clone it from github</a> (tagged at the time of writing as v0.1). I've put some simple management scripts (which could also do with a bit of tidying up) in the <span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>scripts</b></span> subdirectory which allow me to <u>refresh</u> a copy of the charm without .git directories, <u>start up</u> the Juju environment, <u>deploy</u> the charm, and <u>tear down</u> the Juju environment (currently the only way to stop the Virgo without using the EC2 console).<br />
<br />
I've licensed the code under the Apache v2 license and would be delighted to accept contributions if anyone would like to raise issues, make some of the changes mentioned above, or otherwise improve the charm.</div>
Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com1tag:blogger.com,1999:blog-34692233.post-38495372659658940642013-11-15T06:22:00.000+00:002013-11-15T06:22:54.353+00:00Virgo Buildpack RefreshedI'm no longer leading the Virgo project, but I am still fond of Virgo, so I thought I would update the <a href="https://github.com/glyn/virgo-buildpack">Virgo buildpack</a> (previously mentioned in <a href="http://underlap.blogspot.co.uk/2013/04/virgo-in-cloud-cloudfoundry-buildpack.html">this blog</a>) in my own time to re-base it on the current <a href="https://github.com/cloudfoundry/java-buildpack">Java buildpack</a>.<br />
<br />
It now inherits all the benefits of the Java buildpack (more <a href="http://blog.cloudfoundry.com/2013/09/06/introducing-the-cloud-foundry-java-buildpack/">here</a>):<br />
<br />
<ul>
<li>Ability to upgrade Virgo without changing the buildpack or the application - great when a security fix needs to be applied.</li>
<li>Automatic calculation of JVM memory settings.</li>
<li>Clean separation of components - Virgo is a new <i>container </i>(described <a href="https://github.com/glyn/virgo-buildpack/blob/master/docs/container-virgo.md">here</a>) and the other components including OpenJDK support are inherited for free.</li>
<li>Helpful diagnostics including automatic JVM destruction on "out of memory" and debug logging of the buildpack's operations.</li>
</ul>
<div>
I decided to stick to the most general style of application layout that the previous version of the Virgo buildpack used. This consists of a <span style="font-family: Courier New, Courier, monospace; font-size: x-small;">pickup</span> directory containing zero or more applications and, optionally, a <span style="font-family: Courier New, Courier, monospace; font-size: x-small;">repository/usr</span> directory containing any additional dependencies of the application which are not provided by Virgo. This has the advantage that the detection criterion for Virgo (existence of the pickup directory) does not overlap with the detection criteria of the other containers inherited from the Java buildpack (Groovy, Play, Java Main, Spring Boot CLI, Tomcat) and so the Virgo buildpack is a functional superset of the Java buildpack.</div>
<div>
<br /></div>
<div>
There are two use cases that interest me. One is running Virgo on Cloud Foundry. The other is running Virgo as an <a href="https://juju.ubuntu.com/">Ubuntu Juju</a> Charm - but more of that on another occasion.</div>
Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com0tag:blogger.com,1999:blog-34692233.post-63629936230801765942013-09-03T08:54:00.002+01:002013-09-03T08:54:51.188+01:00Code ClimateWe are using <a href="https://codeclimate.com/">Code Climate</a> to check the quality of the Ruby code in our <a href="https://github.com/cloudfoundry/java-buildpack">Cloud Foundry Java buildpack</a>. It's an excellent tool, but some improvements would really help.<br />
<div class="p2">
<br /></div>
<div class="p1">
1. Scans need to be triggered by git push so that I can rely on notifications rather than prodding/polling.</div>
<div class="p2">
<br /></div>
<div class="p1">
2. Sometimes code duplication detection is a bit harsh, IMO. I'd like an option to annotate a class/method to allow duplication. Filtering out whole files is too coarse-grained. For example, the following methods flag as duplicated but factoring out the duplication would, IMO, damage the readability of the code:</div>
<div class="p2">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidBvtUfOBQ01y57jVADI5-p_msMbAaoXX4Xw-dmZ5qa-lbze2Jq0OGYexXs_Bfln1t5vmB15MnAgG-HQreRXVPzgpnzYiEgJE4yQg9DZn2biDymLFM5GC7bAboKSgldnk4Xcl1wA/s1600/Screen+Shot+2013-09-03+at+08.46.32.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="291" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidBvtUfOBQ01y57jVADI5-p_msMbAaoXX4Xw-dmZ5qa-lbze2Jq0OGYexXs_Bfln1t5vmB15MnAgG-HQreRXVPzgpnzYiEgJE4yQg9DZn2biDymLFM5GC7bAboKSgldnk4Xcl1wA/s640/Screen+Shot+2013-09-03+at+08.46.32.png" width="640" /></a></div>
<div class="p2">
<br /></div>
<div class="p2">
3. I'd like to be able to configure the "Repos" page to show just some of my projects for a quick team-specific summary. Preferably I'd like to be able to switch easily between filtered views.</div>
<div class="p2">
<br /></div>
<br />
<div class="p1">
4. I'd like to be able to add email addresses to the notification configuration, e.g. mailing lists and mail aliases so multiple people can be notified easily.</div>
Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com1tag:blogger.com,1999:blog-34692233.post-90021626334692545622013-04-03T16:55:00.001+01:002013-04-04T12:15:11.803+01:00Virgo in the Cloud: a CloudFoundry buildpackThe recent Virgo community survey showed quite a bit of interest in running Virgo in the cloud. With CloudFoundry's forthcoming "buildpack" support, this was relatively easy to achieve.<br />
<br />
A <a href="https://github.com/glyn/virgo-buildpack">Virgo buildpack</a> is available on github - see the <a href="https://github.com/glyn/virgo-buildpack/blob/master/README.md">README</a> for detailed information. It currently supports two types of application: web applications or "overlays". A web application may be a standard WAR file or an OSGi Web Application Bundle. An overlay is a pair of directories -- <span style="font-family: Courier New, Courier, monospace;">pickup</span> for the application(s) to be deployed and <span style="font-family: Courier New, Courier, monospace;">repository/usr</span> for any additional dependencies not already provided by Virgo -- which are placed in an instance of Virgo Server for Apache Tomcat.<br />
<br />
The support for buildpacks should ship later this month - see the <a href="http://cloudfoundry.github.com/docs/roadmap.html">CloudFoundry roadmap</a>.Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com0tag:blogger.com,1999:blog-34692233.post-13013044878956732912013-02-18T16:18:00.000+00:002013-02-18T16:21:13.623+00:00Virgo survey resultsThanks to the ninety-seven people who filled in the Virgo survey. The survey is now closed and the results are in. I'll summarise the results here, but the <a href="http://www.eclipse.org/virgo/project-info/Survey-2013.ods">raw data</a> (in Open Document Spreadsheet format) are available for anyone who wants more detail.<br />
<br />
88% of respondents are on Virgo 3.5.0 or later while only 5% are still using SpringSource dm Server.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiac-8aFaz2OjFzQBmGLSjWC-SaH4_sQjAkip2MzgYCXaT_JK7MrPZpQ2iakEqb9DwZIPBZStthtVBGVAKlxuliHocEiBRBhzXF6MZwAeGWxcBdI5fClkugwQiNnX8Roi8uDDnYwA/s1600/Screen+Shot+2013-02-18+at+14.07.59.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="294" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiac-8aFaz2OjFzQBmGLSjWC-SaH4_sQjAkip2MzgYCXaT_JK7MrPZpQ2iakEqb9DwZIPBZStthtVBGVAKlxuliHocEiBRBhzXF6MZwAeGWxcBdI5fClkugwQiNnX8Roi8uDDnYwA/s640/Screen+Shot+2013-02-18+at+14.07.59.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Virgo Server for Apache Tomcat is by far the most popular runtime deliverable. I would expect this to change over time if SAP's cloud strategy based on Nano Web is successful, even if such success doesn't feature much in Virgo survey results.</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOyAKFEgzdieMCUzxTTc1yuO0URL3RAJGn6ahl6tjnL5Ja0uVP7bomDVO_deuNGb2g03IFICXpyXAqEZhJPJ715OM8aBt2LgZ7bniohPUSKov8HGW-TRyWHr42cp5iTwqbhgQkWA/s1600/Screen+Shot+2013-02-18+at+14.08.40.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="354" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOyAKFEgzdieMCUzxTTc1yuO0URL3RAJGn6ahl6tjnL5Ja0uVP7bomDVO_deuNGb2g03IFICXpyXAqEZhJPJ715OM8aBt2LgZ7bniohPUSKov8HGW-TRyWHr42cp5iTwqbhgQkWA/s640/Screen+Shot+2013-02-18+at+14.08.40.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
A greater respondents (36%) are in production this time, compared to 23% in May 2011.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLAVYBf-Wz42ooeC5v0Rxd_0a67RYx9kTvlCQsGcsrwxA0BMBIAoiyIIi-6TZpAHEvz8gokqne7DhgC0AyNu1uzZ_rkKBNZn1yXHz8o4588AiT9UC6bx-ui5vzkDszyLPVjqggpQ/s1600/Screen+Shot+2013-02-18+at+14.09.51.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="168" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLAVYBf-Wz42ooeC5v0Rxd_0a67RYx9kTvlCQsGcsrwxA0BMBIAoiyIIi-6TZpAHEvz8gokqne7DhgC0AyNu1uzZ_rkKBNZn1yXHz8o4588AiT9UC6bx-ui5vzkDszyLPVjqggpQ/s640/Screen+Shot+2013-02-18+at+14.09.51.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
OSGi and Spring are both preferred programming models with Java EE being somewhat less preferred. Admittedly this question doesn't capture those who are forced to use a programming model they would prefer not to, but hopefully that number is small given the choice of programming model provided by Virgo.</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiU2b8H2n2TbrYlKCB43OsjhAZKgVe_8CdTxAHYrPa8cizsNBULOxkVI6gWt-zb4Bcvg3hZdFox2tmf9rE3bsHv21ser66adxdS3fGbm8h5VQU5TbrTwAdIqCf7rfxQy9Kg_bdaAw/s1600/Screen+Shot+2013-02-18+at+14.10.42.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="174" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiU2b8H2n2TbrYlKCB43OsjhAZKgVe_8CdTxAHYrPa8cizsNBULOxkVI6gWt-zb4Bcvg3hZdFox2tmf9rE3bsHv21ser66adxdS3fGbm8h5VQU5TbrTwAdIqCf7rfxQy9Kg_bdaAw/s640/Screen+Shot+2013-02-18+at+14.10.42.png" width="640" /></a></div>
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNqsD_vhJMBYROvaK0MQsULSg8CWhhZz94-FP_WpP-b8GjeNa1nBEbAV0Av8up2ynt_RysTrb2zqETV8yJZPpJhJJ8koWQyON287L70bGr6W0XmO3SHvCbZXk7bjnfQx4hgTYqAg/s1600/Screen+Shot+2013-02-18+at+14.11.41.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="198" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNqsD_vhJMBYROvaK0MQsULSg8CWhhZz94-FP_WpP-b8GjeNa1nBEbAV0Av8up2ynt_RysTrb2zqETV8yJZPpJhJJ8koWQyON287L70bGr6W0XmO3SHvCbZXk7bjnfQx4hgTYqAg/s640/Screen+Shot+2013-02-18+at+14.11.41.png" width="640" /></a></div>
<br />
The future directions were all of interest, some a little more than others. I'm not going to comment in detail, but better support for OSGi standard applications is particularly strong with a high "Gimme gimme" ranking and a tiny snooze factor.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiczFHxvaleHb4lZB05GVdibafXHUl5To3bXBKbJx7yWfE5O1EpOCumWdhaDIsEzQvVX3HnkNJXgxEjIe0f72MU9xEF-5QZvjF3YkvLFuBWUchPlOwNv22eGHLI0-R3UsXFBOX-Pg/s1600/Screen+Shot+2013-02-18+at+14.20.31.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiczFHxvaleHb4lZB05GVdibafXHUl5To3bXBKbJx7yWfE5O1EpOCumWdhaDIsEzQvVX3HnkNJXgxEjIe0f72MU9xEF-5QZvjF3YkvLFuBWUchPlOwNv22eGHLI0-R3UsXFBOX-Pg/s1600/Screen+Shot+2013-02-18+at+14.20.31.png" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0MRMNt2Y7aKPPf60BxBr6Ra19mUdcEiwV30pT47ywRpyctZCJHOGYilD9uIQnaOdaefLHUIyWTdLr9FxvlyzZd3uKIm0r2TXY23wB8PXpQZuwhppyxADTjdWI2QXvXM6_liO3IQ/s1600/Screen+Shot+2013-02-18+at+14.20.47.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0MRMNt2Y7aKPPf60BxBr6Ra19mUdcEiwV30pT47ywRpyctZCJHOGYilD9uIQnaOdaefLHUIyWTdLr9FxvlyzZd3uKIm0r2TXY23wB8PXpQZuwhppyxADTjdWI2QXvXM6_liO3IQ/s1600/Screen+Shot+2013-02-18+at+14.20.47.png" /></a></div>
<br />
The written comments were illuminating on the whole. Here are the comments for question 7 "Any other future direction(s) you'd really like to see".<br />
<br />
<b>Better Maven integration: Automatic deployment, Automatic installation, etc.
Better Documentation on How To do integration tests of a complex Web Appplication in Virgo.</b><br />
<b><br /></b>
<b>I would like to see a better testing framework for Virgo unit and integration testing.
</b><br />
<b><br /></b>
<b>Ensuring that the tooling follows the development of the Virgo server, because it's quite slow right now and often leads to some null pointer that imply reconfiguring the server in STS :(
</b><br />
<b><br /></b>
<b>reliability and troubleshooting are a major concern. As Virgo exposes OSGi to applications - it's harder to debug classloading problems, compared to pure Tomcat or Jetty environments.
xx</b><br />
<br />
Question 11 "Anything else to get off your chest?" produced quite a bit of feedback. The tooling is fairly criticised, so we are hoping the upcoming 1.0.1 release will fix <a href="https://bugs.eclipse.org/bugs/buglist.cgi?list_id=4462177&classification=RT&query_format=advanced&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=CLOSED&component=tooling&product=Virgo&target_milestone=1.0.1.RELEASE">a number of common problems</a>. There are repeated calls for sharing of experience with real world applications and technologies, something I feel the Virgo community could helpfully contribute, for instance via blogs or on the Virgo wiki: I'd invite anyone interested to go ahead and add to the <a href="http://wiki.eclipse.org/Virgo/Community">Community section</a>. Meanwhile, we have been <a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=329368">revamping the samples</a> and will be issuing a milestone soon.<br />
<br />
<b>Integration testing made easy (we talked about cargo integ' on the forum and it's for me the priority to leverage Virgo as a top level platform).</b><br />
<b><br /></b>
<b>Virgo was not simple to shrink, we ended up with a number of components that we could not easily throw out. We did eventually get the right isolation and app working, but the project failed for other non technical reasons.
There are generally too many frame works in java, and the owners of java have let opportunity to fix jar hell slip too many times. Look at other language ecosystems as examples. Erlang makes it super easy for operators to update code in a system.</b><br />
<b><br /></b>
<b>Virgo:
Don't push yourself so hard today. Try to find some relaxation time. If you had big plans, perhaps you should cancel them. Instead, schedule some time with friends or family. Or treat yourself to a little indulgence. Some wine tasting here or appetizers there won't hurt as long as you don't go overboard.</b><br />
<b><br /></b>
<b>We really would like to see a much more stable version of the virgo tooling. Virgo must be restarted way too often during deployment because of tooling problems.</b><br />
<b><br /></b>
<b>Move away from Spring. JavaEE + OSGi is a better combination. Nice to see that supported in the latest(?) release.</b><br />
<b><br /></b>
<b>Generally I'm sorry to say I just don't see a future for Virgo. The container is too hard to use and the complexity is too much for the benefits delivered.
Gettting a working build and keeping it working with manifest manipulation is still too hard and error prone. It seems unlikely it will ever get fixed or improved.
The "next big thing" seems to be Typesafe. But for the most part, containers are going away, and we're going to move to something like AMQP based services.
In fact, you can get JVM isolation just by running more JVMs in the cloud. So the unit of deployment is the Virtual Machine, not an OSGi bundle.
Sorry for the negative feedback, personally I have always liked OSGi, but for most developers it's too hard to use for the benefits derived.</b><br />
<b><br /></b>
<b>The first one thanks for your job, and support of this great Software called Virgo.
We are developing in OSGi and Virgo about two years, developing Konekti aka Modular Web business platform. We suffer the bad initial tools in eclipse to develop, actually are better. Of course we would like more documentation :)
We think it would be interesting adding any blog or tutorials about integration of other frameworks, or experience with Virgo. Our company spent many time integrating many frameworks like JBMP, Drools, ActiveMQ, Jersey, GWT, Vaadin, Jasper Reports, Quartz, ... , so we could provide our experience. Also we would like to know this experience of other companies that also are developing in OSGi and Virgo.
We think that Virgo and OSGi in general are a great technology, but not many people knows about it, so any pedagogical work is necessary, and any must lead it. If in the future you add some meeting point, we can contribute it with our real experience in OSGi and Virgo.
Best Regards.</b><br />
<b><br /></b>
<b>Without Eclipse Virgo we would probably not have choosen to try OSGi in my project. Because we need to migrate a Java web application with quite a lot dependencies.</b><br />
<b><br /></b>
<b>My take on the future - you need to branch out and ensure as much of the available Java technologies are instantly usable in Virgo. Classify the technologies (something like java-source.net does, but better) and pick a project for each one to start with, involve the community and build a library of addons and examples - you'll not only prevent users from bailing on using Virgo because of too much program infrastructure integration effort, but do just the opposite - attract them with ready-to-use examples and simple working projects.
I started playing with Virgo since about 7 months. I am using Virgo to rebuild an existing security application platform which we provide to our customers and i find it very attractive to use for me as a developer. Using Virgo as an application server platform opens up a huge potential to me and allows me to build real modular applications based on osgi without having to deal with osgi runtime basics to get started, which is great. The features Virgo provides are good and do make sense to have in such an environment. Although, as mentioned above, Virgo Tooling should be improved and in my opinion this is the key element to attract more developers for using Virgo.</b><br />
<b><br /></b>
<b>We use it as a managed osgi runtime and install everything into one region for sanity. Virgo rewriting bundle imports causes more problems than they are worth. Using different containers instead of a partitioned runtime makes more sense for disparate applications.</b><br />
<b><br /></b>
<b>Virgo tools need a lot of improvements.
In my company we ended up implementing a little set of eclipse plug-ins to perform similar tasks as those offered by Virgo tools. For example, ordering of projects deployed to the server, because whenever you open the Virgo server editor it takes ages to show up and you just have a 5 lines list for ordering bundles, with no possibility to perform a multiple selection.</b><br />
<b><br /></b>
<b>Nice modular concept. Needs examples to host Virgo Nano in regular J2EE web app</b><br />
<b><br /></b>
<b>I don't know what I'd do without you guys!</b><br />
<b><br /></b>
<b>sorry but I feel you're apart from rest of the osgi community. Why not use or contribute to bnd, maven-bundle-plugin... (note : I have no idea about politics)
Please keep thing simple, because It's not easy to defend osgi. I try in my compagny, but osgi feel complicated. In fact it's not. Just a problem of image.</b><br />
<b><br /></b>
<b>First up - it's excellent. We were going to migrate to an OSGI stack but Virgo seems to have addressed a whole bunch of issues we knew we would hit.
The down sides are:
- eclipse tooling a bit unreliable / immature (still cannot deploy a )
- different approach from target platform standard eclipse stuff (more eclipse general issue maybe)
- the whole spring, hibernate, cglib, resolve issues, class not found issues, blah blah is still a nightmare (nature of the game I guess) - *particularly if migrating to virgo* - greenfield may be easier</b><br />
<b><br /></b>
<b>Hate the fact that there is no book on Virgo yet, hate that using it depends on other projects (Gemini) that also have sparse documentation. Even a simple hello world on one of the technologies can be a waste of a few hours when adapting to Virgo.
Felix/Aries start to gain more traction now. No Paxrunner support for Virgo yet.</b><br />
<b><br /></b>
<b>To me virgo nano with documentation regarding on how to build isolation would be perfect.</b><br />
<b><br /></b>
<b>The whole jetty thing and not having an answer for regular Jdk app server model.
</b><br />
<b><br /></b>
<b>Virgo is great and has been very reliable. It is OSGi and integrating 3rd party software that is the pain. For example, getting DB2 datasources using a ConnectionPool framework and getting the datasources via JNDI lookup or a Declarative service takes a lot of work to configure.
I still have not seen any good documentation about storing encrypted data in external properties files and then having a hook to decrypt the data so that it may be used in a blueprint.xml definition in the context of ${var} substitution. I do not want the id/password used for my database connections to be stored in plain text in some properties file. Seems that there should be a good recipe or API for handling such common security concerns.
For the most part I am well pleased with what Virgo offers, and I am in the middle of transforming a JBoss J2EE multi-tier application to a pure OSGi application running under Virgo.</b><br />
<b><br /></b>
<b>I love Virgo but the learning curve is steep. Not only Virgo's fault but overall to develop SpringDM applications I had many problems since there are many technologies involved none of which is particularly stable. Random problems: Bundles where there which were not seen, then they were seen. Spring services were found then were not found anymore. Then deleting the work directory solved a lot. Then a bundle was found with version 0.0.0 and you have no idea why. Then 1 day later you realise there was a missing dependency. Etc.
Debugging is still quite tricky, when things work do work but many times to get there is a huge pain. Anyway Virgo rocks :)
Also the problem with many JARs around which are not yet OSGfied. And to OSGi them you have to beg the developers. Or you have to wrap it yourself but then you have to do it everytime there's a release. There are ways to easily OSGfy a plugin but it would be very useful to make the process as seamless as possible and push the community to embrace OSGi.</b><br />
<b><br /></b>
<b>I really appreciate what you guys doing. But I don't like the current state of maven support. We use maven for every project it is key technology at our company.
It was not easy to setup maven multi-module project to work with eclipse virgo ide and be able to successfully build outside eclipse as well. (driving all osgi dependencies from maven side + generating manifest + providing transient dependencies so virgo can see them when we add dependency just to pom.xml)
Do not have sources this days is just sad (and make it that bundles which comes with virgo to be source-aware is complete madness) - with maven it is just so easy.
Anyway keep up good work.</b><br />
<b><br /></b>
<b>Its still hard to managed those jar dependencies or change to more recent version of spring without running into conflicts.
</b><br />
<b><br /></b>
<b>Tooling tooling tooling. Is has been the biggest issue and time consuming process. We have finally crafted a build setup that support maven in CI and in Sts that works but at times can be fragile and difficult to troubleshoot. Many many times we ponder if there is a better way and aren't able to find the answer.</b><br />
<b><br /></b>
<b>Startup time is the major pain point for us.</b><br />
<b><br /></b>
<b>I think you need to do more community development. Work on describing how people can use vanilla technologies deployed in Virgo and then move into OSGi modules if they want to. My main concern is trying to stay within a technology stack that is either directly portable or easily ported to a different server as an option.</b><br />
<b><br /></b>
<b>Seems like things took a while to transition to Eclipse, but feeling better about Virgo and tooling by the day. We're most interested in running this on tiny devices, so are happy (and hoping) to see continued support for Nano.</b><br />
<b><br /></b>
<b>We are very happy with Virgo! We enjoy the fact that's pretty lightweight, easy to configure and provides good performance. A good thing would be if more than one pickup-folder could be defined.</b><br />
<b><br /></b>
<b>In my opinion Virgo is lacking some really good documentation and programming examples. The examples available are all using maven and not the eclipse pde dependencies and tools, wich are the easiest way to develop within Eclipse.
</b><br />
<b><br /></b>
<b>Virgo Tooling with Eclipse is a nightmare. So many bugs, runtime parameters that can't be changed (try to change MaxPermSize...) or the lib\endorsed directory not included - took us hours to find out. You guys should eat your own dog food and start to use these tools not just in hobby projects.... I guess this would improve quality.</b><br />
<b><br /></b>
<b>You should put more effort in the Virgo Tooling it's the more frustrating thing in Virgo. It is very slow and buggy. Keep the good work!</b><br />
<b><br /></b>
<b>I want better virgo tooling in eclipse/sts. Each times i open tooling in eclipse, it's very slow still 30-40secs.
If i stop virgo in error during starting then i need kill eclipse.</b><br />
<br />
The technology section of the survey is rather long and hard to summarise, but it will provide useful input for future development. See the <a href="http://www.eclipse.org/virgo/project-info/Survey-2013.ods">raw data</a> if you are interested.<br />
<br />
Finally, the quiz results were intriguing. The correct (or should I say <i>my</i>) answers are:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwFaTM8PLWF9TCz3DjP6573Me8tSiCdxAqo4sYqoHojY3pr-ZqW5OdCfCu9UjF5H8WT3Cuoh7evGSGvy5KFv9pM7lh8-2bWo5swxuHYpoiFPnqhlBGL6aNXRmO6bpCFudfS7lKhQ/s1600/Screen+Shot+2013-02-18+at+14.57.43.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="500" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwFaTM8PLWF9TCz3DjP6573Me8tSiCdxAqo4sYqoHojY3pr-ZqW5OdCfCu9UjF5H8WT3Cuoh7evGSGvy5KFv9pM7lh8-2bWo5swxuHYpoiFPnqhlBGL6aNXRmO6bpCFudfS7lKhQ/s640/Screen+Shot+2013-02-18+at+14.57.43.png" width="640" /></a></div>
<br />
<a href="http://jonas.ow2.org/xwiki/bin/view/Main/">JOnAS</a> was the biggest mystery with 49% responding "Dunno". <a href="https://blogs.oracle.com/cloudappfoundation/entry/oracle_weblogic_server_12c_launch">WebLogic</a> was second with 32%, just in front of <a href="http://www.javaworld.com/javaworld/jw-03-2009/jw-03-osgi-in-websphere.html">WebSphere</a> with 28%. One person thought that Virgo does not use OSGi or explicitly support it, but their preferred programming model was Java EE, so perhaps they see Virgo as just another enterprise server.<br />
<br />Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com0tag:blogger.com,1999:blog-34692233.post-45853244709471880062013-02-04T13:50:00.004+00:002013-02-04T13:59:23.421+00:00No solution for complexityNeil Bartlett's <a href="http://njbartlett.name/2013/02/04/no-solution-for-complexity.html">blog</a> of this title is shocking because it shows how slow to adopt proven technology the computing industry is.<br />
<br />
David Parnas wrote his seminal paper "On the Criteria To Be Used in Decomposing Systems into Modules" (available <a href="http://sunnyday.mit.edu/16.355/parnas-criteria.html">here</a>) in 1972. A large transaction processing monitor was modularised in the 1980's because the cost of servicing it was spiralling out of control. (<a href="http://en.wikipedia.org/wiki/CICS#Z_notation">Precise specifications</a> were written for the new modules to clean up the interfaces and minimise dependencies on module internals, but that's another story.)<br />
<br />
The OSGi module system for Java has been mature for at least five years, possibly longer depending on how you count it. Most of the major Java application servers are now constructed of OSGi modules and a number of them expose OSGi for applications to use. And yet we still see complaints such as the one Neil quotes. It seems application developers are stuck in their ways or stuck on systems which prevent them from using modularity.<br />
<br />
Of course, modularity <i>will</i> eventually be adopted by all business critical software. There's really no rational alternative. But how long it will take is quite another matter.Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com0tag:blogger.com,1999:blog-34692233.post-53197711403764451292013-02-01T16:20:00.002+00:002013-02-01T16:20:40.172+00:00Why I hate spikesThe <a href="http://www.infoq.com/presentations/Embracing-Uncertainty">recording</a> of Dan North's recent talk on "Embracing Uncertainty" set me thinking again about what I like and dislike about agile methods such as scrum. As we end the 157th consecutive sprint of the Virgo project, I certainly appreciate the focus and sense of making progress that each sprint brings.<br />
<br />
But what I really dislike are spikes.<br />
<br />
The principle of a spike sounds fine: carve out some time to do some exploratory work. But I soon find myself too constrained by a spike. One <a href="http://searchsoftwarequality.techtarget.com/definition/spike">definition</a> of a spike is (emphasis added):<br />
<blockquote class="tr_bq">
"a story that cannot be estimated until a development team runs a <i>timeboxed</i> investigation. The output of a spike story is an <i>estimate</i> for the <i>original story</i>."</blockquote>
There are some assumptions buried in that definition which I think show some of the problems:<br />
<br />
<ul>
<li>implicitly, and significantly, a spike is something you do <i>in</i> a sprint</li>
<li><i>timeboxed</i> - of course we can't commit an indefinite amount of time to a spike, but a timebox kind of implies that at the end, you'll come up with "the goods"</li>
<li><i>estimate</i> - the "goods" are an estimate</li>
<li><i>original story</i> - there's an assumption that the spike won't change your perception of what the story should be.</li>
</ul>
<br />
Here's where Dan's presentation helped me understand what was going on. Spikes are an attempt to fit some <i>explorative</i> work into an agile process which is designed for <i>repeatedly</i> churning out reliable software with <i>minimal risk</i>.<br />
<br />
My natural way of doing exploratory work is rather different. First of all, I don't presume to know what it is I expect to find. I have some ideas to try out, some questions to answer, or maybe just something I need to learn about (and believe me, the longer I stay in this job, the more ignorant I realise I am).<br />
<br />
So the actual "goods" are really answers, understanding, and more ideas. It might be possible to turn some of these into estimates for a story, but that's not really the goal.<br />
<br />
What's wrong with trying to fit exploratory work into a sprint, you may ask? I think the psychology of reporting status in a stand-up meeting each day forces me to focus on things which will deliver immediate results. Sometimes the best way to explore something is to follow a number of lines of inquiry, get yourself thoroughly overwhelmed, take a break, and then (after several pennies have hopefully dropped), start to understand the underlying concepts.<br />
<br />
Worse still, if I have at least one non-trivial task in the sprint besides a spike, it's really hard to get into the exploratory frame of mind. The only way I've found is to get the other stuff out of the way as quickly as possible and then try to switch modes.<br />
<br />
So in the future, I propose to approach exploratory work differently. I'll treat it as something that's done outside a sprint. I'll still keep the idea of a timebox - it's no good going off indefinitely and losing contact with the rest of the team. But I definitely won't require that the output is an estimate of a story.<br />
<br />
If this sounds too risky, then good. Innovation is supposed to be risky.<br />
<br />
<br />Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com4tag:blogger.com,1999:blog-34692233.post-51346083284548342652013-02-01T13:56:00.003+00:002013-02-01T13:56:42.364+00:00Restore Windows pop-up from Java on Mac OS XI kept seeing the following pop-up during Java based builds on Mac OS X:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9ar1Ty1BVQhCYh3psqp8Yc6IO_h57qRBwILg3vW597NvLAJdhTmHKD1vdiKsHal4dXnuFD5B8jkA4gPJqdHHQfXrGmu3x8iesFDgh4EZcjzbI2wGryuMTBtAugeDEpWe8nbzBzA/s1600/Screen+Shot+2013-02-01+at+13.09.40.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9ar1Ty1BVQhCYh3psqp8Yc6IO_h57qRBwILg3vW597NvLAJdhTmHKD1vdiKsHal4dXnuFD5B8jkA4gPJqdHHQfXrGmu3x8iesFDgh4EZcjzbI2wGryuMTBtAugeDEpWe8nbzBzA/s1600/Screen+Shot+2013-02-01+at+13.09.40.png" /></a></div>
<br />
There was no obvious cause, but this was annoying as it paused the build until I dismissed the pop-up, although it never seemed to have any side-effect in the build.<br />
<br />
I finally seem to have found the solution, which was to delete the Java related files in the directory <span style="font-family: Courier New, Courier, monospace; font-size: x-small;">~/Library/Saved Application State:</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">$ rm -rf ~/Library/Saved\ Application\ State/com.apple.javajdk16.cmd.savedState/</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">$ rm -rf ~/Library/Saved\ Application\ State/net.java.openjdk.cmd.savedState/</span><br />
Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com0tag:blogger.com,1999:blog-34692233.post-78352348543449765312013-01-29T09:40:00.001+00:002013-01-29T12:39:07.613+00:00Eclipse Bundle Recipes project to supersede SpringSource Enterprise Bundle Repository (EBR)IBM, Paremus, Rebaze, Red Hat, and VMware/SpringSource, with interested parties Peter Kriens and SAP, have submitted an <a href="http://eclipse.org/proposals/rt.ebr/">Eclipse Bundle Recipes</a> project proposal aiming to provide a repository of OSGi bundle templates for open source JARs. The intention is that this will supersede the <a href="http://ebr.springsource.com/repository/app/">SpringSource Enterprise Bundle Repository</a> (EBR).
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4hqJokyNhCRxzZvBftmoqlcJ1zgyDYZ0GcmebEkEDVshhEhhSJc6HUih12bUGQDUUI5XsJ9cl9bx0wVLG_l689tO5eBdr0TG2MDISyJuUxdDlJIT26-_UzQH1hKSYjWhaldO5WA/s1600/Screen+Shot+2013-01-28+at+13.18.58.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="62" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4hqJokyNhCRxzZvBftmoqlcJ1zgyDYZ0GcmebEkEDVshhEhhSJc6HUih12bUGQDUUI5XsJ9cl9bx0wVLG_l689tO5eBdr0TG2MDISyJuUxdDlJIT26-_UzQH1hKSYjWhaldO5WA/s640/Screen+Shot+2013-01-28+at+13.18.58.png" width="640" /></a></div>
<br />
It's been nearly a couple of years since we made the templates from the SpringSource EBR available on <a href="https://github.com/glyn/bundlerepo">github</a>. Apart from several pull requests from Raman Gupta (a.k.a. rocketraman, pictured below), not much appeared to happen...<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://secure.gravatar.com/avatar/c2aa2c9f527afea12a7fef458588b2a1?s=400&d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-user-420.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="200" src="https://secure.gravatar.com/avatar/c2aa2c9f527afea12a7fef458588b2a1?s=400&d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-user-420.png" width="200" /></a></div>
<br />
In fact, quite a lot was happening behind the scenes. I was in touch with various colleagues in other companies to see if there was interest in creating a community-based replacement for the SpringSource EBR. It turned out that several large vendors did want to join in, but some of them were very cautious about the licenses of the JAR files that the project would handle. For this reason, we went in the direction of a project to host just the manifest templates.<br />
<br />
Clearly, this wasn't going to be much use to people needing to download OSGi bundles, so we also discussed how to make the resultant bundles available. One approach was to generate the bundles on the client machine to cope with most, if not all, open source licenses. The downside of this is the potential lack of reproducibility each time a bundle is generated.<br />
<br />
The more people we talked to, the more interest there was in storing the generated bundles in Maven Central. It turns out this is exactly what the <a href="http://team.ops4j.org/wiki/display/PAXTIPI/Pax+Tipi">Pax Tipi</a> project over at <a href="http://team.ops4j.org/wiki/display/ops4j/Open+Participation+Software+for+Java">OPS4J</a> started doing last year. So we hope to build on their experience, but perhaps achieve a higher profile.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://team.ops4j.org/wiki/download/attachments/10387457/global.logo?version=1&modificationDate=1229120180422" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://team.ops4j.org/wiki/download/attachments/10387457/global.logo?version=1&modificationDate=1229120180422" /></a></div>
<br />
So the plan is to collaborate on templates at the Eclipse Bundle Recipes project, but also have these published to Maven Central. The details need to be worked out, but now that the Eclipse project proposal has been submitted, we can finally talk about this in public. The project fits naturally into the Eclipse Runtime umbrella project.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjMPGZyWjrR5XkGIAnO-jz_q0_8m3E_p0Fhy7LhgDyYg1EsHQNtPpKSgHfi-eLJhT2sD3toK9tD3zdoPHAb4m-m0suOmggJUYt528E-M1AwGfcs4UMjYshuxXUU6CvKeZ-s2bZagw/s1600/Screen+Shot+2013-01-28+at+13.43.30.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjMPGZyWjrR5XkGIAnO-jz_q0_8m3E_p0Fhy7LhgDyYg1EsHQNtPpKSgHfi-eLJhT2sD3toK9tD3zdoPHAb4m-m0suOmggJUYt528E-M1AwGfcs4UMjYshuxXUU6CvKeZ-s2bZagw/s1600/Screen+Shot+2013-01-28+at+13.43.30.png" /></a></div>
<br />
Interestingly, another resource for people wanting to add OSGi manifests to existing project has recently become available - the <a href="http://blog.osgi.org/2013/01/get-help-adding-osgi-metadata-to-your.html">metadata advice</a> bugzilla category at the OSGi Alliance. This should be a valuable resource for anyone wanting to develop a high quality OSGi manifest for a third party JAR.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXI2G24LGqEWmsKSJ2M6s9MwqKtsDdQb6KSszaUjHRNy9fgTQSg9WYc5vQkL3cqirsIT6NUC2D0pq1qa_rt3GzsvsUsm2opF4WeJ8lnvJ_AZBR0i6ibH_XgjVwYBNJycIZyRiK1g/s1600/Metadata_Advice_via_Bugzilla.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="382" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXI2G24LGqEWmsKSJ2M6s9MwqKtsDdQb6KSszaUjHRNy9fgTQSg9WYc5vQkL3cqirsIT6NUC2D0pq1qa_rt3GzsvsUsm2opF4WeJ8lnvJ_AZBR0i6ibH_XgjVwYBNJycIZyRiK1g/s640/Metadata_Advice_via_Bugzilla.png" width="640" /></a></div>
So the scene is finally set for some progress on superseding the SpringSource EBR. Watch this space or, better still, keep an eye on the Eclipse Bundle Recipes project and or get involved. There's a <a href="http://www.eclipse.org/forums/index.php/t/452224/">discussion topic</a> on the proposals forum if you want to comment.Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com1tag:blogger.com,1999:blog-34692233.post-65958752669503739262013-01-16T14:54:00.001+00:002013-01-16T14:54:20.885+00:00Virgo Community SurveyIt's a couple of years since we ran the first Virgo survey and the project has moved on a lot since then. So we've just published a new <a href="http://bit.ly/ZX73GL">Eclipse Virgo Community Survey</a>. If you use Virgo, I'd really appreciate it if you could take a few minutes to complete the survey. If you are short of time, please just fill in the first six questions and skip the rest.<br />
<br />
Question 12 is a fun quiz about the use of OSGi in application servers. You'll probably find it interesting even if you have no interest in Virgo. We'll provide the answers in a few weeks time along with the survey results.
<br />
<br />
Here's the survey if you want to get going right away.<br />
<br />
<br />
<br />
<br />
<iframe frameborder="0" height="5272" marginheight="0" marginwidth="0" src="https://docs.google.com/spreadsheet/embeddedform?formkey=dC1JMXRCbzd4a2pVN285ZU5fRlJoM2c6MQ" width="760">Loading...</iframe>Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com0tag:blogger.com,1999:blog-34692233.post-17946520116305684962012-12-21T11:28:00.000+00:002012-12-21T14:00:17.454+00:00Virgo 3.6.0 releasedVirgo 3.6.0 is available for <a href="http://www.eclipse.org/virgo/download/">download</a>. Although the release includes many small enhancements and bug fixes, some major new features are worth highlighting and there is one smaller one I'd like to draw to the attention of Windows users.<br />
<br />
Mike Milinkovich, the Executive Director of the Eclipse Foundation, spoke of Virgo 3.6.0 in his recent <a href="http://jaxenter.com/mike-milinkovich-s-year-in-eclipse-45862.html">JAX interview</a>:<br />
<blockquote>
<b>"A third important milestone was SAP shipping its Netweaver Cloud offering based on the Eclipse Virgo project. Having a major vendor basing such a significant product on Eclipse runtime technologies is a great endorsement of the work that the Eclipse RT community has been doing for several years."</b></blockquote>
So, let's look at the new features.<br />
<br />
<b>Java Enterprise APIs</b><br />
<br />
The Virgo Nano Web distribution now integrates several open source Java Enterprise API implementations which will reduce the barrier to entry for a large class of existing applications.<br />
<br />
The <a href="http://scn.sap.com/community/developer-center/cloud-platform">SAP NetWeaver Cloud</a> platform, which uses Virgo Nano Web as its application server, <a href="http://www.information-management.com/news/sap-netweaver-cloud-now-java-ee-web-profile-compatible-10023689-1.html">has been certified</a> against the Java EE Web Profile. Although Virgo has not been certified, I'm optimistic that if Virgo eventually gains access to the Web Profile TCK (<a href="http://www.eclipse.org/org/foundation/boardminutes/2012_06_19-20_Minutes.php">discussed</a> by the Eclipse Board in June), it shouldn't be too onerous to get it to pass. Until such time as we certify Virgo, we have to be careful not to claim support for the relevant APIs. Please see the <a href="http://www.eclipse.org/virgo/download/release-notes/3.6.0.RELEASE.php">release notes</a> for further details of the APIs and implementations involved.<br />
<br />
Virgo continues its first class support for Spring by embedding Spring framework 3.1 and supporting Spring framework 3.2 in the kernel-based deliverables.<br />
<br />
Virgo users now have the choice of using Spring or Java Enterprise APIs to develop Virgo applications. Alternatively, they are free to take the minimal Nano or kernel deliverables and build their own custom runtimes, like Infor did with their <a href="http://www.infor.com/infor-ion/">ION server</a>.<br />
<br />
See the <a href="http://www.eclipse.org/virgo/feature-summary/">Virgo feature summary</a> for a comparison of the various Virgo deliverables.<br />
<br />
<b>New Web Admin Console</b><br />
<br />
This began life as a skunk-works project by my colleague Chris Frost. The new web admin console is an <a href="http://en.wikipedia.org/wiki/AJAJ">AJAJ</a> application backed by the <a href="http://www.jolokia.org/agent.html">Jolokia OSGi agent</a> paired with Gemini Management. Gemini Management and the Virgo kernel publish JMX mbeans which enable Virgo to be managed and inspected. Jolokia provides JSON access to those mbeans which are then available to the JavaScript running in the admin console. Again, more of this in the release notes and an upcoming blog from Chris - meanwhile here's an <a href="http://codewax.org/eclipse-rt/new-web-admin-console-in-virgo/">earlier blog</a>.<br />
<br />
We are particularly fond of the OSGi explorer which provides a graphical depiction of package and service wirings. The initial prototype was implemented by David Normington during a short internship here at SpringSource. Below is an example package wiring diagram.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjU2v0fv2n5B05MaKl3R_toDs0T-m7hbtq4CeTndZGK4V6glJypgWHOt6KaOsAltYs25LzlvK54JjTfAr44o2hQEhdTOKYkFZhAJ3SnVqrxQJ2tfILLCbhFg8kOkjLWI6DUP5LrFQ/s1600/Screen+Shot+2012-12-18+at+15.27.19.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="548" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjU2v0fv2n5B05MaKl3R_toDs0T-m7hbtq4CeTndZGK4V6glJypgWHOt6KaOsAltYs25LzlvK54JjTfAr44o2hQEhdTOKYkFZhAJ3SnVqrxQJ2tfILLCbhFg8kOkjLWI6DUP5LrFQ/s640/Screen+Shot+2012-12-18+at+15.27.19.png" width="640" /></a></div>
We had lots of fun testing and fixing the admin console (especially on IE), but we are rather pleased with the result. It seems to push the SVG implementations fairly hard though. Take this example of large "fan-out":<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgO29rJj_sfdH9uadmqUa4IOjVk43k9qFdhcNxvxHSQSYRSoF-lhPX_4NX9WO21tEByCWRM1gwcWL4NlAflzsuVV0QDdX5dmtR2_I-eYEIdycd1yWGCNzkJaSyUFNF1L9YJq777XA/s1600/Screen+Shot+2012-12-18+at+15.33.00.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="232" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgO29rJj_sfdH9uadmqUa4IOjVk43k9qFdhcNxvxHSQSYRSoF-lhPX_4NX9WO21tEByCWRM1gwcWL4NlAflzsuVV0QDdX5dmtR2_I-eYEIdycd1yWGCNzkJaSyUFNF1L9YJq777XA/s640/Screen+Shot+2012-12-18+at+15.33.00.png" width="640" /></a></div>
<br />
If we scroll to the far left on Safari, the Bezier curves are nice and smooth:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXpGtpab2i0PyfitMKK4yNSkm-brkHZP2lIxXLn26eCdhhFMvkLNNE9nmFk7IOsfC_WH7IvmGMowVvcXMcEIOD0cWagYF02I-N_Yg2NH6zjyXw3lwxj4kMPpNUUZ-Tmu36QgbTpw/s1600/Screen+Shot+2012-12-18+at+15.34.22.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="82" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXpGtpab2i0PyfitMKK4yNSkm-brkHZP2lIxXLn26eCdhhFMvkLNNE9nmFk7IOsfC_WH7IvmGMowVvcXMcEIOD0cWagYF02I-N_Yg2NH6zjyXw3lwxj4kMPpNUUZ-Tmu36QgbTpw/s640/Screen+Shot+2012-12-18+at+15.34.22.png" width="640" /></a></div>
<br />
Compare the result in Chrome:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQD8v_963TdksloMZFv0Ch2iP1N6wDgks-L8qed71eXM4qFLSmlBt2uzxzCW5hBseWr-atvnq9kMq_6fHhVaPrtTUQ8-r4jJDZcFw_hyphenhyphenSX-cgXWDdZ8VUtzbD11jJ3ZHVAWVREJg/s1600/Screen+Shot+2012-12-18+at+15.38.43.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="78" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQD8v_963TdksloMZFv0Ch2iP1N6wDgks-L8qed71eXM4qFLSmlBt2uzxzCW5hBseWr-atvnq9kMq_6fHhVaPrtTUQ8-r4jJDZcFw_hyphenhyphenSX-cgXWDdZ8VUtzbD11jJ3ZHVAWVREJg/s640/Screen+Shot+2012-12-18+at+15.38.43.png" width="640" /></a></div>
<br />
<a href="http://www.blogger.com/"></a>
Again see the <a href="http://www.eclipse.org/virgo/download/release-notes/3.6.0.RELEASE.php">release notes</a> for more details and screenshots.<br />
<br />
The master stroke of the new web admin console is that by minimising its dependencies (shown below), it is now available in all Virgo runtimes, right down to Nano, and smoothly degrades on Nano and Nano Web by omitting panels which are only relevant to kernel and above. Where a web container is not provided by the runtime, on Nano and the kernel, we use the Equinox HTTP Service to host the admin console.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2ThArZFqIUkbYd1mLorD4Wm7uAi3fmkDepmvvT7JoTOJKLhEalbbCb1esMNBSvNqbpT8gKEieImZ3YD9TQf93bUyQcYMcSBsPY5RBVKXQ1V_A-sDzgR1IBl7725ygG5oyWTCfjw/s1600/Screen+Shot+2012-12-18+at+15.53.29.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="152" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2ThArZFqIUkbYd1mLorD4Wm7uAi3fmkDepmvvT7JoTOJKLhEalbbCb1esMNBSvNqbpT8gKEieImZ3YD9TQf93bUyQcYMcSBsPY5RBVKXQ1V_A-sDzgR1IBl7725ygG5oyWTCfjw/s640/Screen+Shot+2012-12-18+at+15.53.29.png" width="640" /></a></div>
<br />
<b>Java 7 Support</b><br />
<br />
This is boring but important since Java 6 public updates <a href="http://www.oracle.com/technetwork/java/eol-135779.html">cease</a> in February 2013. The Hudson jobs for Virgo run under Java 7, but we still do local builds and releases under Java 6 to ensure both versions work well.<br />
<br />
<b>Kernel Deployer Pathname Reduction</b><br />
<b><br /></b>
This is the smaller feature to make the lives of Windows users a bit easier and which satisfied a long-standing requirement.<br />
<br />
Firstly, the kernel deployer uses shorter paths for storing copies of deployer artefacts.<br />
<br />
Secondly, users can configure the kernel deployer to deploy bundles in packed form, which avoids the potentially long paths involved in exploded directories. Deploying in exploded form is still the default because deploying web applications packed changes their behaviour: certain servlet API methods return null when a web application is deployed packed. See "Configuring Deployment" in the section "Configuring the Kernel and User Region" in the <a href="http://www.eclipse.org/virgo/documentation/">Virgo User Guide</a> for details.<br />
<br />
<b>Conclusion</b><br />
<br />
All in all Virgo 3.6.0 is a significant release and should make Virgo attractive to a broader range of users and applications. We expect Virgo 3.7.0 to improve on 3.6.0 incrementally and we welcome bug and enhancement reports based on your experience of using 3.6.0.<br />
<br />
Can you add your project to the Virgo powered list (on the <a href="http://www.eclipse.org/virgo/">Virgo home page</a>) in 2013?<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjR7_82ucyejeaU75woZXiNm3GtqefSZQ0m2m4QNq18yYAh-nJcnsu7Iq47D7yGUTlXdH43YpgfU4zX_eKzYfa-r1qcUkXZQ53eK41lup9AfDPGbH1mIXqz29YVWSim2gHDdXyKlQ/s1600/Screen+Shot+2012-12-21+at+13.56.18.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjR7_82ucyejeaU75woZXiNm3GtqefSZQ0m2m4QNq18yYAh-nJcnsu7Iq47D7yGUTlXdH43YpgfU4zX_eKzYfa-r1qcUkXZQ53eK41lup9AfDPGbH1mIXqz29YVWSim2gHDdXyKlQ/s1600/Screen+Shot+2012-12-21+at+13.56.18.png" /></a></div>
<br />
<br />
<br />
<br />
<br />Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com2tag:blogger.com,1999:blog-34692233.post-82640296928180529372012-11-16T10:54:00.000+00:002013-01-31T17:07:35.238+00:00Virgo at the Core of SAP's CloudCongratulations to SAP on delivering <a href="http://www.sap.com/solutions/technology/cloud/netweaver/index.epx">Netweaver Cloud</a>! The SAP committers have been a great asset to the Virgo team and it's good to see their <a href="http://www.eclipse.org/projects/project.php?id=rt.virgo">hard work</a> finally paying off for SAP.<br />
<br />
Virgo is the core of SAP Netweaver Cloud or, less formally, Neo. More precisely, they are using the Virgo Nano Web deliverable which can be provisioned to their cloud VMs using p2 and which supports OSGi applications and applications written to the Java EE Web Profile. They've <a href="http://scn.sap.com/community/developer-center/cloud-platform/blog/2012/11/15/sap-netweaver-cloud-is-java-ee-6-web-profile-compatible">blogged</a> about certifying Netweaver Cloud and now appear on the <a href="http://www.oracle.com/technetwork/java/javaee/overview/compatibility-jsp-136984.html">Java EE compatibility page</a>.<br />
<br />
Here's a short video where Harald Mueller, Chief Product Owner for SAP NetWeaver Cloud, mentions the use of Virgo and some of the benefits from his point of view - the section from 4:18 to 6:55 was of most interest to me: <a href="http://www.sapvirtualevents.com/teched/embed.aspx?sid=2877">click here to watch</a>.<br />
<br />
Neo's business proposition is to encourage application developers to participate in the "SAP Store" by developing applications to run on Virgo in the cloud. They are emphasising open standards and open source to break out of SAP's proprietary image. Not only do they plan to attach a large number of application developers, they have a stretch goal of <a href="http://scn.sap.com/community/cloud/blog/2012/10/23/chewing-on-the-billion-user-goal">1 billion users</a>.<br />
<br />
Neo's <a href="http://scn.sap.com/community/developer-center/cloud-platform/blog/2012/11/07/netweaver-cloud-a-look-behind-the-curtain">current architecture</a> deploys each application to one or more virtual machines, each running one instance of Virgo Nano Web. AFAIK there is no support for multi-tenancy and popular applications will require load balancing across multiple Virgo instances, so the overall memory consumption looks pretty high. To scale up to large numbers of applications, not to speak of a billion users, multi-tenancy will be critical.<br />
<br />
There are two clear options for multi-tenancy: multiple Virgo instances per virtual machine and multiple applications per Virgo instance. The former offers better failure isolation via operating system processes whereas the latter sacrifices strict isolation in order to share the overheads of JVM plus Virgo componentry. It will be interesting to see which approach they adopt. I'm particularly intrigued to see if they exploit the Virgo kernel's scoping mechanisms, <a href="http://underlap.blogspot.co.uk/2011/02/stumbling-towards-better-design.html">regions</a>, or OSGi subsystems to achieve multi-tenancy in a Virgo instance.Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com0tag:blogger.com,1999:blog-34692233.post-17558599213035958502012-10-12T10:44:00.000+01:002012-10-12T10:44:20.024+01:00How are you using Virgo?Following on from the nice write-up about the <a href="http://www.eclipse.org/forums/index.php/mv/msg/398243/939875/#msg_939875">use of Virgo by Croatian Telecom</a>, I'd like to encourage others who have not already done so to provide a brief summary of how they are using Virgo. If you can include a company or project name, so much the better. Numbers of servers, users, and applications are helpful if you have them to hand as are peak traffic rates. A frank summary of the pro's and con's of Virgo and OSGi will help others who are assessing the technology. Also, mentioning some of the specific 3rd party components you are successfully using gives others an idea whether what they'd like to do is feasible.<br />
<br />
I can link helpful write-ups which include company or project names from the "Virgo Powered" list of the <a href="http://www.eclipse.org/virgo/">Virgo home page</a>, so you get some free marketing out of this. If you can place your write-up on a personal or company blog, I can link to that too, thus helping to increase your inbound links and traffic.
<br />
<br />
This is a great, low-cost way to contribute something back to the Virgo community, even if you never answer a forum thread, raise a bug, or send in a patch. Please take a few minutes to do a write-up if you feel you can.Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com3tag:blogger.com,1999:blog-34692233.post-91057002912114795952012-10-02T15:39:00.000+01:002012-10-02T15:39:36.776+01:00Virgo Ships in VMware vSphere 5.1Apart from all those v's, what do VMware vSphere and Virgo have in common? Well you won't be surprised to learn that VMware, as well as contributing to the <a href="http://www.eclipse.org/virgo/">Virgo project</a>, have shipped Virgo as part of their flagship virtualisation product <a href="http://www.vmware.com/products/datacenter-virtualization/vsphere/index.html">vSphere 5.1</a>. Virgo provides the web server for the vSphere Web Client.<br />
<br />
VMware chose Virgo in order to make the Web Client UI extensible. Users can write plugins defining Web Client extension points and containing WAR files, OSGi bundles and extension point configurations in order to supply Adobe Flex UI components and custom data services used by the UI.<br />
<br />
Plugins are registered with a vCenter Server and are automatically deployed to Virgo instances which connect to the vCenter Server. The result is a customisable UI and a customisable data layer that can serve up data either from the VMware Inventory Service or some other remote data source.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWqMhq9D7iXXTmMjKFfzVxHR0Duzp8KtCAn9i6Qhm8A_6Y8PpRnUPNxcZfvSO42cgoB0Uh5hjm5uPJ5SH4NTGYxvNSwZBwi16jaGBXNOnoRqgTlscS95LnG3l1_WKZNKV9Go9TyQ/s1600/vSphereClient.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="460" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWqMhq9D7iXXTmMjKFfzVxHR0Duzp8KtCAn9i6Qhm8A_6Y8PpRnUPNxcZfvSO42cgoB0Uh5hjm5uPJ5SH4NTGYxvNSwZBwi16jaGBXNOnoRqgTlscS95LnG3l1_WKZNKV9Go9TyQ/s640/vSphereClient.jpg" width="640" /></a></div>
<br />
Users can develop WAR files and OSGi bundles using the <a href="http://wiki.eclipse.org/Virgo/Tooling">Virgo IDE tooling</a> and VMware provides the configured Virgo server (known as <i>Serenity</i>), an SDK for developing extensions, and full documentation. See the <a href="http://communities.vmware.com/community/vmtn/developer/forums/vspherewebsdk">SDK download page</a> for these.<br />
<br />
Full documentation is provided in the <a href="http://pubs.vmware.com/vsphere-51/topic/com.vmware.ICbase/PDF/vs5_1_WC_Ext_Prog_Guide.pdf">vSphere Web Client Extensions Programming Guide</a>.Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com0tag:blogger.com,1999:blog-34692233.post-36962921207301994062012-09-26T16:55:00.002+01:002012-09-26T16:59:47.218+01:00Virgo 3.6.0.M01 ships with some juicy new featuresVirgo 3.6.0.M01 is released and features upgrades to Tomcat (bringing with it web sockets support), Spring, Gemini Web, and Gemini Management.<br />
<br />
It includes a completely new admin console with an interesting, minimal architecture which enables it to run on the kernel as well as the Tomcat and Jetty based deliverable. See <a href="http://codewax.org/eclipse-rt/new-web-admin-console-in-virgo/">Christ Frost's blog</a> for more details.<br />
<br />
There is also improved Windows support by shortening some of the path names in Virgo's work directory. You can also deploy bundles and WAR files in packed form if you choose to.<br />
<br />
The Nano hot deployer also has new support for <i>initial</i> bulk deployment of multiple bundles, particularly useful when they have dependencies between them.<br />
<br />
Quite a fun set of features all in all, plus plenty of bug fixes and other improvements.<br />
<br />
Please see the <a href="http://www.eclipse.org/virgo/download/release-notes/3.6.0.M01.php">release notes</a> for more details or just head over to the <a href="http://www.eclipse.org/virgo/download/milestones.php">milestone download page</a> to get going.Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com0tag:blogger.com,1999:blog-34692233.post-42067305881750039422012-09-25T09:53:00.004+01:002012-09-25T09:55:42.374+01:00Twelve-Factor Web Apps in JavaA colleague mentioned the <a href="http://www.12factor.net/">Twelve-Factor App</a> site which gives some design guidelines for building web apps. The <a href="http://www.12factor.net/dependencies">Dependencies guideline</a> has the following intriguing paragraph:<br />
<br />
<blockquote>
"<strong style="font-family: sirba-web-1, sirba-web-2, serif; font-size: 16px; line-height: 24px; text-align: justify;">A twelve-factor app never relies on implicit existence of system-wide packages.</strong><span style="font-family: sirba-web-1, sirba-web-2, serif; font-size: 16px; line-height: 24px; text-align: justify;"> It declares all dependencies, completely and exactly, via a </span><em style="font-family: sirba-web-1, sirba-web-2, serif; font-size: 16px; line-height: 24px; text-align: justify;">dependency declaration</em><span style="font-family: sirba-web-1, sirba-web-2, serif; font-size: 16px; line-height: 24px; text-align: justify;"> manifest. Furthermore, it uses a </span><em style="font-family: sirba-web-1, sirba-web-2, serif; font-size: 16px; line-height: 24px; text-align: justify;">dependency isolation</em><span style="font-family: sirba-web-1, sirba-web-2, serif; font-size: 16px; line-height: 24px; text-align: justify;"> tool during execution to ensure that no implicit dependencies “leak in” from the surrounding system. The full and explicit dependency specification is applied uniformly to both production and development."</span><br />
</blockquote>
<br />
What would this mean for a Java app? It's more than just using a build tool such as Maven to declare dependencies using explicit coordinates since there is mention of a "dependency isolation tool" at runtime. Well, I think OSGi is the only viable way of implementing this guideline in Java (unless you fancy re-implementing something like the OSGi modularity layer using custom class loaders). The author, Adam Wiggins, makes no mention of OSGi, so presumably he has written Java off as an implementation language. Or does he have some other modularity technology in mind for Java? It would be interesting to know.Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com0tag:blogger.com,1999:blog-34692233.post-2864172344872973392012-07-12T11:40:00.000+01:002012-07-12T13:54:53.584+01:00Virgo 3.5.0 with Nano TechnologyVirgo 3.5.0.RELEASE is available for <a href="http://www.eclipse.org/virgo/download">download</a>.<br />
<br />
Although the deliverables are mostly backward compatible with Virgo 3.0.x, we decided to indicate the importance of the release by bumping the version from 3.0 to 3.5.<br />
<br />
All Virgo deliverables are now launched using the Equinox launcher and have a corresponding directory structure which is slightly different from that of 3.0.x. All the deliverables can all be initially provisioned using p2, which is useful, for instance, when setting up remote nodes or automating installation.<br />
<br />
There are two new deliverables which the team is pretty excited about. Virgo nano is a cut-down subset of the kernel. It supports hot deployment of bundles via the <span style="font-family: 'Courier New', Courier, monospace;">pickup</span> directory, the Gogo shell, a single (kernel) region, and the medic component which provides Virgo's basic diagnostic capabilities.<br />
<br />
The second new deliverable is Nano Web which is essentially Gemini Web -- the Apache Tomcat based reference implementation of the OSGi Web Applications specification -- running on Nano. In addition to Nano's features, Nano Web supports hot deployment of Web Application Bundles and WAR files.<br />
<br />
Compared to the kernel-based deliverables, which are already pretty small and fast, Nano starts much faster and has a much smaller footprint as the following charts indicate.<sup>1</sup><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhq-6Cx1jsRfW_KhyphenhyphenG3nY4nXH0MujM7gGLI45jtZRffm0rdxx36CFdLJr6AchD0QfQ2kvCXK67cRPbfzDKX57UAVVm45Ptu6-ml3N23_Toq41k04ErMW9zy9_sBsaTk6x0NeGtvpw/s1600/startup.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="355" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhq-6Cx1jsRfW_KhyphenhyphenG3nY4nXH0MujM7gGLI45jtZRffm0rdxx36CFdLJr6AchD0QfQ2kvCXK67cRPbfzDKX57UAVVm45Ptu6-ml3N23_Toq41k04ErMW9zy9_sBsaTk6x0NeGtvpw/s640/startup.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6QYWbkGIHNvsMmhsXDa66ki0Y2y6y3tvtF0bGiWAM6OvojZyEbF5MUPfBaBlJVb1Pcy1yuIBS8yrjvUR_yWbBVlbYJaanrUmz4Ai45EpTbdbgiqbK-h5mK42DTbRctn9iozclag/s1600/zipsize.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="345" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6QYWbkGIHNvsMmhsXDa66ki0Y2y6y3tvtF0bGiWAM6OvojZyEbF5MUPfBaBlJVb1Pcy1yuIBS8yrjvUR_yWbBVlbYJaanrUmz4Ai45EpTbdbgiqbK-h5mK42DTbRctn9iozclag/s640/zipsize.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMhJDMgvgrfKMFWyWhz9JA1mxle4SyU3cJmi9pBQRQnb1k9xLRUCBLmuxdC9-zBezNiafJDZ22Wle3USLRYsS_SGVQmkFaQfUObCIWPtXug4anhgGcz_RKMBSPuIk6ghT_GZaFPw/s1600/heap.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="352" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMhJDMgvgrfKMFWyWhz9JA1mxle4SyU3cJmi9pBQRQnb1k9xLRUCBLmuxdC9-zBezNiafJDZ22Wle3USLRYsS_SGVQmkFaQfUObCIWPtXug4anhgGcz_RKMBSPuIk6ghT_GZaFPw/s640/heap.png" width="640" /></a></div>
<br />
Both Nano and Nano Web can be provisioned with application bundles at runtime using p2. The kernel based deliverables can only be initially provisioned using p2. Applications are then deployed into the user region in the usual way. This provisioning limitation in p2 triggered the initial development of Nano, although Nano soon appeared to have other benefits such as smaller runtime footprint (particularly important in the cloud) and the possibility of using PDE for bundle development. Also, Nano Web is likely to integrate additional enterprise Java components in the future to support applications written for the Java EE web profile.<br />
<br />
There are a number of other interesting features in 3.5.0 including integrated support for the OSGi Blueprint Service, several significant kernel enhancements, and various dependency upgrades - all described in the release notes.<br />
<br />
Additionally, Virgo Bundlor 1.1.1 and the Greenpages 2.5.0 <a href="http://www.eclipse.org/virgo/samples/">sample application</a> are released in association with Virgo 3.5.0.<br />
<br />
In parallel with the development of Virgo 3.5.0, the Virgo IDE tooling has been overhauled and will be released in the next few weeks. Meanwhile you can use milestones and snapshots of the new tooling as described on the <a href="http://wiki.eclipse.org/Virgo/Tooling">tooling wiki page</a>.<br />
<br />
Please see the <a href="http://www.eclipse.org/virgo/download/release-notes/3.5.0.RELEASE.php">release notes</a> for further details of Virgo 3.5.0 and the <a href="http://wiki.eclipse.org/Virgo/Community/Migrating_from_3.0.x_to_3.5.0">migration notes</a> for information about upgrading from Virgo 3.0.x.<br />
<br />
Footnote:<br />
<ol>
<li>Startup times are approximate. Measurements were taken using Mac OS X 10.7.4, and Java 1.6.0_33 on a Mac Pro with 12 GB RAM and 2 x 2.66 GHz dual core Intel Xeon processors.</li>
</ol>Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com0tag:blogger.com,1999:blog-34692233.post-76754036685545706962012-06-12T15:21:00.000+01:002012-06-12T15:21:57.918+01:00OSGi case study: a modular vert.x<title></title>
OSGi enables Java code to be divided cleanly into modules known
as <i>bundles</i> with access to code and resources controlled by a class loader for each bundle. OSGi <i>services</i> provide an
additional separation mechanism: the users of an interface need
have no dependency on implementation classes, factories, and so
forth.<br />
<br />
The following case study aims to make the above advantages of OSGi
bundles and services concrete. It takes an interesting Java
project, vert.x, and shows how it can be embedded in OSGi and take
advantage of OSGi's facilities.<br />
<br />
<b>Disclaimer:</b> I am not proposing to replace the vert.x
container or its module system. This is primarily a case study in
the use of OSGi although some of the findings should motivate
improvements to vert.x, especially when it is embedded in
applications with custom class loaders.<br />
<h3>
vert.x</h3>
<div>
The <a href="http://vertx.io/">vert.x</a> open source
project provides a JVM alternative to <a href="http://nodejs.org/">node.js</a>: an asynchronous,
event-driven programming model for writing web applications in a
number of languages including Java, Groovy, JavaScript, and
Ruby.</div>
<div>
<br /></div>
<div>
vert.x supports HTTP as well as modern protocols such as
<a href="https://en.wikipedia.org/wiki/WebSockets">WebSockets</a>
and <a href="https://github.com/sockjs/sockjs-client">sockjs</a> (which
works in more browsers than WebSockets and can traverse firewalls
more easily).</div>
<div>
<br /></div>
<div>
vert.x has a distributed event bus which allows JSON messages
to be propagated between vert.x applications known as
<i>verticles</i> and shared code libraries known as
<i>busmods</i>. A busmod is a special kind of verticle which
handles events from the event bus. vert.x ships some busmods,
such as a <a href="http://www.mongodb.org/">MongoDB</a> 'persistor', and users
can write their own.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUYoDE0hWOiuP-Ici_Ajo_lU-aL9BKaEmB9xSThTRaJ2055Yw85nZRFfurXXB5crl6bf4o9ZS8R5bjPwqo25y3Knmd134uA08IhViCDQff53voj0eTf_Cl9qVlF4FUmsYbrZP4tQ/s1600/vertx.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="vert.x components" border="0" height="342" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUYoDE0hWOiuP-Ici_Ajo_lU-aL9BKaEmB9xSThTRaJ2055Yw85nZRFfurXXB5crl6bf4o9ZS8R5bjPwqo25y3Knmd134uA08IhViCDQff53voj0eTf_Cl9qVlF4FUmsYbrZP4tQ/s400/vertx.jpg" title="" width="400" /></a></div>
<br />
vert.x's threading model is interesting as each verticle (or
busmod) is bound to a particular thread for its lifetime and so the
code of a verticle needn't be concerned about thread safety. A pool
of threads is used for dispatching work on verticles and each
verticle must avoid blocking or long-running operations so as not
to impact server throughput (vert.x provides separate mechanisms
for implementing long-running operations efficiently). This is
similar to the <a href="http://cicswiki.org/cicswiki1/index.php?title=Quasi-reentrant">quasi-reentrant</a>
threading model in the CICS transaction processor.<sup>1</sup><br />
<br />
Of particular interest here is the vert.x module system which has a
class loader per verticle and code libraries, known as
<i>modules</i>, which are loaded into the class loader of each
verticle which uses them. So there is no way to share code between
verticles except via the event bus.<br />
<br />
vert.x has excellent documentation including a <a href="http://vertx.io/manual.html">main manual</a>, a <a href="http://vertx.io/core_manual_java.html">java manual</a> (as well as
manuals for other language), <a href="http://vertx.io/tutorials.html">tutorials</a>, and runnable
<a href="http://vertx.io/examples.html">code examples</a>.<br />
<h3>
OSGi</h3>
<div>
If you're not already familiar with OSGi, read
my <a href="http://underlap.blogspot.co.uk/2006/11/osgi-introduction.html">OSGi
introduction</a> post, but don't bother following the links in that post right now -
you can always go back and do that later.</div>
<h3>
Embedding vert.x in OSGi</h3>
<div>
I did this in several small steps which are presented in
turn below: converting vert.x JARs to OSGi bundles and then
modularising verticles, busmods, and event bus clients.</div>
<h4>
Converting vert.x JARs to OSGi Bundles</h4>
<div>
The <a href="http://vertx.io/manual.html#vertx-embedded">vert.x manual</a>
encourages users to embed vert.x in their own applications by using
the vert.x core JAR, so the first step in embedding vert.x in OSGi
was to convert the vert.x core JAR into an OSGi bundle so it could be
loaded into an OSGi runtime.</div>
<br />
I used the <a href="http://www.eclipse.org/virgo/download/milestones.php">bundlor</a>
tool, although other tools such as <a href="http://www.aqute.biz/Bnd/Bnd">bnd</a> would work equally well.
Bundlor takes a template and then analyses the bytecode of the JAR
to produce a new JAR with appropriate OSGi manifest headers. Please
refer to the <a href="http://static.springsource.org/s2-bundlor/1.0.x/user-guide/htmlsingle/user-guide.html">
SpringSource Bundlor documentation</a> for further information
about bundlor for now as the Eclipse Virgo Bundlor documentation is
not published at the time of writing even though the bundlor
project has transferred to Eclipse.org.<br />
<br />
The template for the vert.x core JAR is as follows:<br />
<br />
<pre><b>Bundle-ManifestVersion: 2
Bundle-SymbolicName: org.vertx.core
Bundle-Version: 1.0.0.final
Bundle-Name: vert.x Core
Import-Template:
org.jboss.netty.*;version="[3.4.2.Final,4.0)",
org.codehaus.jackson.*;version="[1.9.4,2.0)",
com.hazelcast.*;version="[2.0.2,3.0)";resolution:=optional,
groovy.*;resolution:=optional;version=0,
org.codehaus.groovy.*;resolution:=optional;version=0,
javax.net.ssl;resolution:=optional;version=0,
org.apache.log4j;resolution:=optional;version=0,
org.slf4j;resolution:=optional;version=0
Export-Template: *;version="1.0.0.final"</b>
</pre>
<br />
(The <a href="https://github.com/glyn/vert.x.osgi/blob/master/bundling/vertx.core.mf">
template</a> and all the other parts of this case study are
available on <a href="https://github.com/glyn/vert.x.osgi">github</a>.)<br />
<br />
What this does is define the valid range of versions for packages
that the JAR depends on (the range "0" represents the version range
of 0 or greater), whether those packages are optional or mandatory,
and what version the JARs own packages should be exported at. It
also gives the bundle a <i>symbolic name </i>(used to identify
the bundle), a version, and a (descriptive) name. Armed with this
information, OSGi then wires together the dependencies of bundles
by delegating class loads and resource lookups between bundle class
loaders.<br />
<br />
Thankfully the <a href="http://netty.io/">netty</a> networking
JAR and <a href="http://jackson.codehaus.org/">jackson</a>
JSON JARs which the vert.x core JAR depends on ship with valid OSGi
manifests.<br />
<br />
As a sniff test that the manifest was valid, I tried deploying the
vert.x core bundle in the <a href="http://www.eclipse.org/virgo/">Virgo</a> kernel. This was simply a
matter of placing the vert.x core bundle in the <span style="font-family: 'Courier New', Courier, monospace;">
pickup</span> directory and its dependencies in the <span style="font-family: 'Courier New', Courier, monospace;">
repository/usr</span> directory and then starting the kernel. The
following console messages showed the vert.x core bundle was
installed and resolved successfully:<br />
<pre>
<b><hd0001i> Hot deployer processing 'INITIAL' event for file 'vert.x-core-1.0.0.final.jar'.
<de0000i> Installing bundle 'org.vertx.core' version '1.0.0.final'.
<de0001i> Installed bundle 'org.vertx.core' version '1.0.0.final'.
<de0004i> Starting bundle 'org.vertx.core' version '1.0.0.final'.
<de0005i> Started bundle 'org.vertx.core' version '1.0.0.final'.</b>
</pre>
Using the Virgo shell, I then checked the wiring of the
bundles:<br />
<br />
<pre><b>osgi> ss
"Framework is launched."
id State Bundle
0 ACTIVE org.eclipse.osgi_3.7.1.R37x_v20110808-1106
...
89 ACTIVE org.vertx.core_1.0.0.final
90 ACTIVE jackson-core-asl_1.9.4
91 ACTIVE jackson-mapper-asl_1.9.4
92 ACTIVE org.jboss.netty_3.4.2.Final
osgi> bundle 89
org.vertx.core_1.0.0.final [89]
...
Exported packages
...
org.vertx.java.core; version="1.0.0.final"[exported]
org.vertx.java.core.buffer; version="1.0.0.final"[exported]
...
Imported packages
org.jboss.netty.util; version="3.4.2.Final"<org.jboss.netty_3.4.2.final [92]>
...
org.codehaus.jackson.map; version="1.9.4"<jackson-mapper-asl_1.9.4 [91]>
...</b>
</pre>
I also converted the vert.x platform JAR to an OSGi bundle in
similar fashion as it was needed later.<br />
<h4>
Modularising Verticles</h4>
<div>
A typical verticle looks like this:</div>
<pre>public class ServerExample extends Verticle {
public void start() {
vertx.createHttpServer().requestHandler(new Handler<httpserverrequest>() {
public void handle(HttpServerRequest req) {
...
}
}).listen(8080);
}
}
</pre>
<div>
When the <span style="font-family: 'Courier New', Courier, monospace;">
start</span> method is called it creates a HTTP server, registers
a handler with the server, and sets the server listening on a port.
Apart from the body of the handler, the remainder of this code is
boilerplate. So I decided to factor out the boilerplate into a
common OSGi bundle (org.vertx.osgi) and replace the verticle with a
<i>modular verticle</i> bundle containing the handler and some
declarative metadata equivalent to the boilerplate. The common OSGi
bundle uses the <a href="http://www.osgi.org/wiki/uploads/Links/whiteboard.pdf">whiteboard
pattern</a> to listen for specific kinds of services in the OSGi
service registry, create boilerplate based on the metadata, and
register the handler with the resultant HTTP server.<br />
<br />
Let's look at the modular verticle bundle. Its code consists of a
single <a href="https://github.com/glyn/vert.x.osgi/blob/master/org.vertx.osgi.sample.basic/src/org/vertx/osgi/sample/basic/HttpServerRequestHandler.java">
HttpServerRequestHandler</a> class:<sup>2</sup>
</div>
<pre>public final class HttpServerRequestHandler implements Handler<httpserverrequest> {
public void handle(HttpServerRequest req) {
...
}
}
</pre>
<div>
It also has declarative metadata in the form of service properties
which are registered along with the handler in the OSGi service
registry. I used the OSGi Blueprint service to do this, although I
could have used OSGi Declarative Services or even registered the
service programmatically using the OSGi API. The blueprint metadata
is a file <a href="https://github.com/glyn/vert.x.osgi/blob/master/org.vertx.osgi.sample.basic/src/OSGI-INF/blueprint/blueprint.xml">
blueprint.xml</a> in the bundle that looks like this:
<pre><?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0">
<service interface="org.vertx.java.core.Handler" ref="handler">
<service-properties>
<entry key="type" value="HttpServerRequestHandler">
<entry key="port" value="8090">
</service-properties>
</service>
<bean class="org.vertx.osgi.sample.basic.HttpServerRequestHandler"
id="handler"/>
</blueprint>
</pre>
This metadata declares that a HTTP server should be created (via
the <span style="font-family: 'Courier New', Courier, monospace;">
type</span> service property), the handler registered with it, and
the server set listening on port 8090 (via the <span style="font-family: 'Courier New', Courier, monospace;">
port</span> service property). This all happens courtesy of
the whiteboard pattern when the org.vertx.osgi bundle is running as
we'll see below.<br />
<br />
Notice that the modular verticle depends only on the Handler and
HttpServerRequest classes whereas the original verticle also
depends on the Vertx, HttpServer, and Verticle classes. This also
makes things quite a bit simpler for those of us who like unit
testing (in addition to in-container testing) as fewer mocks or
stubs are required.<br />
<br />
So what do we now have? Two bundles to add to the bundles we
installed earlier: an org.vertx.osgi bundle which encapsulates the
boilerplate code and an application bundle representing a modular
verticle. We also need a Blueprint service implementation -- as of
Virgo 3.5, a Blueprint implementation is built in to the Virgo
kernel. The following interaction diagram shows one possible
sequence of events:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhA_YJdmq1YFCIGFTASBGzw_zLpw4Ik-rWuMI1XoJxMBUxmfkP28ZJPkPaz2HGcoySIpobueWJeu9NrO4iLFr1uujZ6aUUC_WdAApDIMnOZYHpBi2VBhyKnNMAl95uRWnXFXst7iw/s1600/modular-verticle.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="468" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhA_YJdmq1YFCIGFTASBGzw_zLpw4Ik-rWuMI1XoJxMBUxmfkP28ZJPkPaz2HGcoySIpobueWJeu9NrO4iLFr1uujZ6aUUC_WdAApDIMnOZYHpBi2VBhyKnNMAl95uRWnXFXst7iw/s640/modular-verticle.jpg" width="640" /></a></div>
<br />
In OSGi, each bundle has its own lifecycle and in general bundles
are designed so that they will function correctly regardless of the
order in which they is started relative to other bundles. In the
above example the assumed start order is: blueprint service,
org.vertx.osgi bundle, modular verticle bundle. However, the
org.vertx.osgi bundle could start after the modular verticle
bundle and the end result will be the same: a server will
be created and the modular verticle bundle's handler registered
with the server and the server set listening. If the blueprint
service is started after the org.vertx.osgi and modular verticle
bundles, then the org.vertx.osgi bundle won't detect the modular
verticle bundle's handler service appear in the service registry
until the blueprint service has started, but then the end result
will again be the same.<br />
<br />
The github project contains the source for some sample modular
verticles: a <a href="https://github.com/glyn/vert.x.osgi/tree/master/org.vertx.osgi.sample.basic">
basic HTTP vertical</a> (which runs on port 8090) and a
<a href="https://github.com/glyn/vert.x.osgi/tree/master/org.vertx.osgi.sample.sockjs">
sockjs verticle</a> (which runs on port 8091). The
org.vertx.osgi bundle needed more code to support sockjs and the
modular sockjs verticle needed to provide a sockjs handler in
addition to a HTTP handler.<br />
<h4>
Modularising BusMods</h4>
</div>
<div>
The MongoDB persistor is a typical example of a busmod which processes messages from the event bus:</div>
<pre>public class MongoPersistor extends BusModBase implements Handler<message<jsonobject>> {
private String address;
private String host;
private int port;
private String dbName;
private Mongo mongo;
private DB db;
public void start() {
super.start();
address = getOptionalStringConfig("address", "vertx.mongopersistor");
host = getOptionalStringConfig("host", "localhost");
port = getOptionalIntConfig("port", 27017);
dbName = getOptionalStringConfig("db_name", "default_db");
try {
mongo = new Mongo(host, port);
db = mongo.getDB(dbName);
eb.registerHandler(address, this);
} catch (UnknownHostException e) {
logger.error("Failed to connect to mongo server", e);
}
}
public void stop() {
mongo.close();
}
public void handle(Message<jsonobject> message) {
...
}
}
</pre>
<div>
Again there is a mixture of boilerplate code (to register the event
bus handler), start/stop logic, configuration handling, and the
event bus handler itself. I applied a similar approach to the other
verticles and separated out the boilerplate code into the
org.vertx.osgi bundle leaving the handler and metadata
(including configuration) in a modular busmod. The persistor's
dependency on the MongoDB client JAR (<span style="font-family: 'Courier New', Courier, monospace;">mongo.jar</span>)
is convenient because this JAR ships with a valid OSGi
manifest.<br />
<br />
Here's the <a href="https://github.com/glyn/vert.x.osgi/blob/master/org.vertx.osgi.mod.mongo/src/OSGI-INF/blueprint/blueprint.xml">
blueprint.xml</a>:</div>
<pre><?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0">
<service ref="handler" interface="org.vertx.java.core.Handler">
<service-properties>
<entry key="type" value="EventBusHandler"/>
<entry key="address" value="vertx.mongopersistor"/>
</service-properties>
</service>
<bean id="handler" class="org.vertx.osgi.mod.mongo.MongoPersistor"
destroy-method="stop">
<argument type="java.lang.String"><value>localhost</value></argument>
<argument type="int"><value>27017</value></argument>
<argument type="java.lang.String"><value>default_db</value></argument>
</bean>
</blueprint>
</pre>
<span style="font-family: Times, 'Times New Roman', serif;">Notice
that the boilerplate configuration consists of the handler type and
event bus address. The other configuration (host, port, and
database name) is specific to the MongoDB persistor.</span><br />
<br />
<span style="font-family: Times, 'Times New Roman', serif;">Here's
the <a href="https://github.com/glyn/vert.x.osgi/blob/master/org.vertx.osgi.mod.mongo/src/org/vertx/osgi/mod/mongo/MongoPersistor.java">
modular MongoDB busmod code</a>:</span>
<pre>public class MongoPersistor extends BusModBase
implements Handler<Message<JsonObject>> {
private final String host;
private final int port;
private final String dbName;
private final Mongo mongo;
private final DB db;
public MongoPersistor(String host, int port, String dbName)
throws UnknownHostException, MongoException {
this.host = host;
this.port = port;
this.dbName = dbName;
this.mongo = new Mongo(host, port);
this.db = this.mongo.getDB(dbName);
}
public void stop() {
mongo.close();
}
public void handle(Message<JsonObject> message) {
...
}
}
</pre>
<span style="font-family: Times, 'Times New Roman', serif;">The
code still extends</span> <span style="font-family: 'Courier New', Courier, monospace;">
BusModBase</span> <span style="font-family: Times, 'Times New Roman', serif;">simply
because</span> <span style="font-family: 'Courier New', Courier, monospace;">
BusModBase</span> <span style="font-family: Times, 'Times New Roman', serif;">provides several
convenient helper methods. Again the resultant code is simpler and
easier to unit test than the non-modular equivalent.</span><br />
<h4>
Modularising Event Bus Clients</h4>
<div>
Finally, I needed a modular verticle to test the modular
MongoDB persistor. All this verticle needs to do is to post an
appropriate message to the event bus. Normal vert.x verticles
obtain the event bus using the <span style="font-family: 'Courier New', Courier, monospace;">
Vertx</span> class, but I used the
Blueprint service again, this time to look up the event bus service
in the service registry and inject it into the modular verticle. I
also extended the org.vertx.osgi bundle to publish the event bus
service in the service registry.<br />
<br />
The <a href="https://github.com/glyn/vert.x.osgi/blob/master/org.vertx.osgi.sample.mongo/src/OSGI-INF/blueprint/blueprint.xml">
blueprint.xml</a> for the modular event bus client is as
follows:
<pre><?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0">
<reference id="eventBus" interface="org.vertx.java.core.eventbus.EventBus"/>
<bean class="org.vertx.osgi.sample.mongo.MongoClient">
<argument ref="eventBus"/>
<argument type="java.lang.String">
<value>vertx.mongopersistor</value>
</argument>
</bean>
</blueprint>
</pre>
Then the <a href="https://github.com/glyn/vert.x.osgi/blob/master/org.vertx.osgi.sample.mongo/src/org/vertx/osgi/sample/mongo/MongoClient.java">
modular event bus client code</a> is straightforward:<br />
<br />
<pre>public final class MongoClient {
public MongoClient(EventBus eventBus, String address) {
JsonObject msg = ...
eventBus.send(address, msg,
new Handler<Message<JsonObject>>(){...});
}
}
</pre>
<h4>
Taking it for a Spin</h4>
1. I've made all the necessary OSGi bundles available in the
<a href="https://github.com/glyn/vert.x.osgi/tree/master/bundles">bundles</a>
directory in git. You can grab them either by cloning the git
repository:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">
<b>git
clone git://github.com/glyn/vert.x.osgi.git</b></span><br />
<br />
or by downloading a <a href="https://github.com/glyn/vert.x.osgi/zipball/master">zip of the git
repo</a>.<br />
<br />
2. <b>vert.x requires Java 7</b>, so set up a terminal shell to use
Java 7. Ensure the JAVA_HOME environment variable is set correctly.
(If you can't get Java 7 right now, you'll see some errors when the
bundles are deployed to OSGi and you won't be able to run the
samples in steps 8 and 9.)<br />
<br />
3. If you are an OSGi user, simply install and start the bundles in
your favourite OSGi framework or container and skip to step 8. If
not, then use the copy of the Virgo kernel in the git repository as
follows.<br />
<br />
4. Change directory to the virgo-kernel-... directory in your local
copy of the git repo.<br />
<br />
5. On UNIX, issue:<br />
<br />
<b><span style="font-family: 'Courier New', Courier, monospace;">
bin/startup.sh -clean</span></b><br />
<br />
or on Windows, issue:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">
<b>bin\startup.bat -clean</b></span><br />
<br />
6. The Virgo kernel should start and deploy the various bundles in
its <span style="font-family: 'Courier New', Courier, monospace;">
pickup</span> directory:<br />
<ul>
<li>org.vertx.osgi bundle (<span style="font-family: 'Courier New', Courier, monospace;">org.vertx.osgi-0.0.1.jar</span>)</li>
<li>HTTP sample modular verticle (<span style="font-family: 'Courier New', Courier, monospace;">org.vertx.osgi.sample.basic-1.0.0.jar</span>)</li>
<li>SockJS sample modular verticle (<span style="font-family: 'Courier New', Courier, monospace;">org.vertx.osgi.sample.sockjs-1.0.0.jar</span>)</li>
<li>MongoDB persistor sample modular busmod (<span style="font-family: 'Courier New', Courier, monospace;">org.vertx.osgi.mods.mongo-1.0.0.jar</span>)</li>
</ul>
<div>
7. If you want to see which bundles are now running, start the
Virgo shell from another terminal:</div>
<div>
<br /></div>
<div>
<b><span style="font-family: 'Courier New', Courier, monospace;">
telnet localhost 2501</span></b></div>
<div>
<br /></div>
<div>
and use the <b><span style="font-family: 'Courier New', Courier, monospace;">
ss</span></b> or <b><span style="font-family: 'Courier New', Courier, monospace;">
lb</span></b> commands to summarise the installed bundles. The
<b><span style="font-family: 'Courier New', Courier, monospace;">
help</span></b> command will list the other commands
available and <b><span style="font-family: 'Courier New', Courier, monospace;">
disconnect</span></b> will get you out of the Virgo shell. Here's
typical output of the <b><span style="font-family: 'Courier New', Courier, monospace;">ss</span></b> command:</div>
<pre><b>...
89 ACTIVE org.vertx.osgi_0.0.1
90 ACTIVE jackson-core-asl_1.9.4
91 ACTIVE jackson-mapper-asl_1.9.4
92 ACTIVE org.jboss.netty_3.4.2.Final
93 ACTIVE org.vertx.core_1.0.0.final
94 ACTIVE org.vertx.osgi.mods.mongo_1.0.0
95 ACTIVE com.mongodb_2.7.2
96 ACTIVE org.vertx.platform_1.0.0.final
97 ACTIVE org.vertx.osgi.sample.basic_1.0.0
98 ACTIVE org.vertx.osgi.sample.sockjs_1.0.0</b>
</pre>
and of the <span style="font-family: 'Courier New', Courier, monospace;">
<b>lb</b></span> command (which includes the more
descriptive <span style="font-family: 'Courier New', Courier, monospace;">Bundle-Name</span>
headers):
<pre> <b>...
89|Active | 4|vert.x OSGi Integration (0.0.1)
90|Active | 4|Jackson JSON processor (1.9.4)
91|Active | 4|Data mapper for Jackson JSON processor (1.9.4)
92|Active | 4|The Netty Project (3.4.2.Final)
93|Active | 4|vert.x Core (1.0.0.final)
94|Active | 4|MongoDB BusMod (1.0.0)
95|Active | 4|MongoDB (2.7.2)
96|Active | 4|vert.x Platform (1.0.0.final)
97|Active | 4|Sample Basic HTTP Verticle (1.0.0)
98|Active | 4|Sample SockJS Verticle (1.0.0)
</b>
</pre>
<div>
8. You can now use a web browser to try out the basic HTTP
sample at <a href="http://localhost:8090/">localhost:8090</a> which should
respond "hello" or the SockJS sample at <a href="http://localhost:8091/">http://localhost:8091</a> which
should display a box into which you can type some text and a button
which, when clicked, produces a pop-up:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgU450N7Shs6rOtev0qSucaeDbCA4ItAK-i6vJp2TeEVprrrCrLIjVKtkNY_-KV0g3S4nDsFjOm6j1uNvlES5WwPtCneGo5CeRf0HcQF6c28DiHbdIbcPXMj59s_HI29Cx4u12qWg/s1600/Screen+Shot+2012-06-08+at+12.36.05.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgU450N7Shs6rOtev0qSucaeDbCA4ItAK-i6vJp2TeEVprrrCrLIjVKtkNY_-KV0g3S4nDsFjOm6j1uNvlES5WwPtCneGo5CeRf0HcQF6c28DiHbdIbcPXMj59s_HI29Cx4u12qWg/s400/Screen+Shot+2012-06-08+at+12.36.05.png" width="400" /></a></div>
<br />
9. If you want to try the (headless) MongoDB event bus client,
<a href="http://www.mongodb.org/downloads">download MondoDB</a> and
<a href="http://www.mongodb.org/display/DOCS/Quickstart">start it
locally</a> on its default port and then copy <span style="font-family: 'Courier New', Courier, monospace;">org.vertx.osgi.sample.mongo-1.0.0.jar</span>
from the <span style="font-family: 'Courier New', Courier, monospace;">
bundles</span> directory to Virgo's <span style="font-family: 'Courier New', Courier, monospace;">
pickup</span> directory. As soon as this bundle starts, it will
send a message to the event bus and drive the MongoDB persistor to
update the database. If you don't want to use MongoDB to
check that an update was made, take a look in Virgo's logs (in
<span style="font-family: 'Courier New', Courier, monospace;">
serviceability/logs/log.log</span>) to see some System.out lines
like the following that confirmed something happened:</div>
</div>
<pre><b>System.out Sending message: {action=save, document={x=y}, collection=vertx.osgi}
...
System.out Message sent
...
System.out Message response {_id=95..., status=ok}</b>
</pre>
<h3>
OSGi and vert.x Modularity</h3>
<div>
In this case study the various sample OSGi bundles all depend
on, and share, the vert.x core bundle. Each bundle is loaded in its
own class loader and OSGi controls the delegation of class loading
and resource lookups according to how the OSGi bundles are wired
together. In the same way, verticles written as OSGi bundles are
free to depend on, and share, other OSGi bundles.<br />
<br />
This is quite different from the vert.x module system in which any
module (other than a busmod) which a verticle depends on is loaded
into the same class loader as the verticle.<br />
<br />
The advantages of the OSGi module system are that a single copy of
each module is installed in the system and is visible to and may be
managed by tools such as the Virgo shell. It also minimises
footprint.<br />
<br />
The advantages of the vert.x module system are that there is no
sharing of modules between verticles so a badly-written module
could not inadvertently or deliberately leak information between
independent verticles. Also, there is a separate copy of each
(non-busmod) module for each verticle that uses it and so the
module can be written without worrying about thread safety as each
copy will only be executed on its verticle's thread. OSGi users
may, however, be happy to require reusable modules to be
thread-safe and manage any mutable static data carefully to avoid
leakage between threads.</div>
<h3>
Replacing the Container?</h3>
<div>
When I raised the topic of <a href="https://groups.google.com/d/msg/vertx/snSD38weSCw/viw_L1bXBbMJ">embedding
vert.x in OSGi</a>, the leader of vert.x, Tim Fox, asked me whether
I was writing a replacement for the current container, to which I
replied "not really". I said this because I liked vert.x's event
driven programming model and its threading model, which seem to be
part of "the container". But I <i>was</i> trying to replace a
couple of aspects of the vert.x container: the module system and
the way verticles register handlers.</div>
<div>
<br /></div>
<div>
Later it struck me that perhaps the notion of "the container"
as a monolithic entity is a little odd in a modular system and it
might be better to think of multiple, separate notions of
containment which could then be combined in different ways to suit
different users. However, the subtle interaction between the class
loading and threading models seen above shows that the different
notions of containment can depend on each other. I wonder what
others think about the notion of "the container"?<br />
<h3>
Conclusions</h3>
</div>
<div>
vert.x's claim that it can be embedded in other applications
is essentially validated since the OSGi framework is a fairly
exacting application.<br />
<br />
The vert.x module system, although not providing isolation between
modules, does neatly provide isolation between applications
(comprising verticles and their modules) and it enables modules to
be written without paying attention to thread safety.</div>
<div>
<br /></div>
<div>
One vert.x issue was raised<sup>2</sup> which should make
vert.x easier to embed in other environments with custom class
loaders.<br />
<br />
vert.x could follow the example of netty, jackson, and MongoDB JARs
and include OSGi manifests in its core and platform JARs to avoid
OSGi users having to convert these JARs to OSGi bundles. I will
leave this to someone else to propose as I cannot gauge the demand
for using vert.x inside OSGi.<br />
<br />
Running vert.x in OSGi addresses some outstanding vert.x
requirements such as how to automate in-container tests (OSGi has a
number of solutions including Pax Exam while Virgo has a
integration test framework) and how to develop verticles and deploy
them to vert.x under control of the IDE (see the <a href="http://www.eclipse.org/virgo/documentation/virgo-documentation-3.5.0.M04/docs/virgo-tooling-guide/htmlsingle/virgo-tooling-guide.html">
Virgo IDE tooling guide</a>). Virgo also provides
numerous ancillary benefits including the admin shell for
inspecting and managing bundles and verticles, sophisticated
diagnostics, and much more (see the <a href="http://git.eclipse.org/c/virgo/org.eclipse.virgo.documentation.git/plain/white-paper/virgo-white-paper.pdf">
Virgo white paper</a> for details).<br />
<br /></div>
<div>
The exercise also had some nice spin-offs for Virgo. <a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=370253">Bug
370253</a> was fixed which was the only known issue in running
Virgo under Java 7. Virgo 3.5 depends on Gemini Blueprint which
broke in this environment and so <a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=379384">bug
379384</a> was raised and fixed. I used the new Eclipse-based Virgo
tooling to develop the various bundles and run them in
Virgo. As a consequence, I found a few small issues in the tooling
which will be addressed in due course.<br />
<br />
Finally, running vert.x on the Virgo kernel is a further validation
that the kernel is suitable for building custom server runtimes
since now we have vert.x in addition to Tomcat, Jetty, and one or
two custom servers running on the kernel.</div>
<div>
<br /></div>
<b>Footnotes:</b><br />
<ol>
<li>I worked in the CICS development team in my IBM days. A
colleague at SpringSource gave me a "CICS Does That!" T-shirt soon
after we'd started working together. Old habits die hard.</li>
<li>The modular vertical currently needs to intercept vert.x's
resource lookup logic so that files in the bundle can easily be
served. It would be much better for this common code to move to the
org.vertx.osgi bundle, but this requires <a href="https://github.com/purplefox/vert.x/issues/161">vert.x issue
161</a> to be implemented first.</li>
</ol>Glynhttp://www.blogger.com/profile/08741529390385812080noreply@blogger.com10