Friday, June 01, 2007

Module system interoperation kernel

Interoperation is currently a hot topic on the JSR 277 mailing list -- see the "Relationship to JSR 291" thread in the archive.

One of the approaches to interoperation is a 'kernel'. What follows is an extract from the README of the prototype plus the related class diagrams.

Goal

The goal is to enable JSR 277 to interoperate in a first class way with JSR 291 (OSGi). JSR 277 modules must be able to use JSR 291 modules and vice versa without significant restrictions.

The framework can probably enable other types of module systems to interoperate, but that isn't the primary goal.

Strawman

The code [of the prototype] is only a strawman. It is not integrated with Felix or any other OSGi implementation. Nor is it integrated with JSR 277.

However, it outlines the interfaces that are likely to be needed to enable such module systems to interoperate well and provides several JUnit 4.1 testcases which drive the framework to illustrate the framework's features.

The code is licensed under the Apache License v2.0. See LICENSE.txt.

Module systems and wiring

The lack of a standard method of building delegation networks between class loaders requires each module system to interoperate specifically with each other module system. The framework provides a standard way for module systems to create delegation paths, or 'wires', to modules in other module systems.

We need a module system registry or 'kernel' to keep track of the available ModuleSystems. Something like the following.


public interface Kernel {
boolean registerModuleSystem(String moduleSystem, ModuleSystem ms);
boolean deregisterModuleSystem(String moduleSystem);
ModuleSystem getModuleSystem(String moduleSystem);
ModuleSystem[] getModuleSystems();
...
}

Module systems need to co-operate to resolve references to modules in other module systems. So each module system has an associated wire factory.

public interface WireFactory extends ModuleSystemRelated {
Wire resolve(Dependency dep);
Wire create(ResolutionContext ctx, Dependency dep);
Wire createLocal(ResolutionContext ctx, Dependency dep);
void complete(ResolutionContext ctx);
}

The most basic kind of dependency consists of a module name and a module version range. More sophisticated dependencies, such as package dependencies, are conceivable but omitted since they introduce much extra complexity.

A wire is a delegation path to the classloader of another module.

That leaves resolution contexts...

Resolution context

A resolution context is needed so that module systems can co-operate to resolve cyclic dependencies without ending up in an infinite loop. A resolution context also provides somewhere to anchor the state for sharing partially resolve modules and for back-tracking.

Constraint checking and back-tracking

A module system can anchor state off a resolution context so that, when asked more than once to create a wire for a given dependency, it can return each possibility in turn. To avoid interference between similar dependencies, new Dependency instances must be created for each module. If two modules have what appear to be the same dependency, then there need to be two Dependency instances.

Diagrams

The kernel class hierarchy shows the kernel with which module systems are
registered. Module systems and resolution contexts are each related to a
particular kernel (of which there is usually one per VM).
The module system class hierarchy shows wire factories, dependencies, and
wires which are each related to a particular module system.

Wednesday, May 30, 2007

Modular Swing Apps

Thanks to Colm Smyth for pointing out Jasper Potts' presentation from JavaOne on modularising Swing applications using a Spring-OSGi derivative known as SPAR. Nice to see OSGi being used by Sun.

Friday, May 25, 2007

OSGi the most important technology of the decade?

It may have been for me, but read the SD Times' view in OSGi Alliance Reaching Universal Middleware Goal.

Tuesday, May 22, 2007

JSR 291 Final Approval

JSR 291 just passed the Final Approval Ballot with twelve "yes" votes and two "no" votes.

Fourteen out of the sixteen Executive Committee members voted. The majority increased to ten compared to eight at the earlier ballots.

The positive comments praised the open approach of JSR 291, the suitability of the licenses, and the effectiveness of JSR 291 in influencing the other modularity JSRs. One comment urged JSR 277 and JSR 291 to become integrated, which will be achieved if JSR 277 delivers truly first class interoperation with JSR 291.

Voting comments from the detractors predictably concerned the work of the Expert Group rather than the underlying OSGi technology. They don't seem to like the JCP building on the work of other Java standards bodies. The concerns are discussed on the Expert Group mailing list and in an earlier blog.

I would like to express my thanks to many people who have contributed to the success of this JSR. They are too numerous to mention by name but include the JSR 291 Expert Group, the OSGi Alliance's Core Platform Expert Group, the Eclipse Equinox project, and numerous colleagues in IBM and elsewhere.

Monday, May 21, 2007

Designing module system interoperation

The various approaches to interoperation between JSR 277 and JSR 291 (OSGi) raise some interesting design considerations. The crucial distinction between these JSRs is that JSR 277 aims to be part of the Java 7 platform whereas JSR 291 is implemented as pure Java that will run on more or less any Java platform.

The JSR 277 Expert Group, or more accurately the spec. lead, is exploring the option of making some components such as the repository, module definition, and module instance pluggable and then having JSR 291 provide alternative implementations. This requires JSR 291 to surface its modules as JSR 277 modules, which will either require some generalisation of the notion of a JSR 277 module or a force fit. A force fit would probably result in unacceptable restrictions on interoperation and should be avoided.

A more ambitious approach, favoured by several Expert Group members including myself, is to rework JSR 277 to provide a 'custom module system' API analogous in concept to the way custom class loaders can be built on top of the existing class loader API. In this scheme, the JSR 277 module system would simply be the default module system packaged with the Java 7 platform and JSR 291 could implement its module system on top of the API. The challenge in this approach would be to design the API so that multiple module systems implemented to the API would naturally interoperate. A benefit of this design is that it would allow module system implementations to evolve faster than the underlying platform - something that JSR 277 doesn't yet enable its own module system to do.

A third approach, which I prototyped to prove that reasonable interoperation is achievable, involves a module system interoperation 'kernel' with which multiple module systems may register. Each module system provides a class, known as a 'wire factory' , for creating delegation paths ('wires') to its modules' class loaders.

One benefit of both the custom module system API and of the interoperation kernel is that they do not complicate the API that the module developer has to deal with. The 'plugin' approach requires generalisations of the concepts of repository, module, and module definition to be exposed on the module developer API. Although this kind of generalisation may actually improve the API, it could also end up making it more obscure. We'll have to wait and see.

Tuesday, May 15, 2007

Open Source Java at JavaOne

Much was made of OpenJDK at last week's JavaOne conference. On Tuesday morning, Rich Green delighted in telling the assembled masses "We are now, as regards the open-sourcing of Java, done" and then posted the following:


From: Rich Green
To: announce@openjdk.java.net
Subject: Open JDK is here!

Today begins the next phase for Java. I am pleased to announce that
we have completed the open source release of Open JDK.
It's a great day for innovation - and remember, compatibility matters!
Rich

It turns out there is a bit more to do, like replacing some of the 'encumbered' code with a GPL version. Currently, the 6.5 MLOCs of OpenJDK source code must be built with some binaries which provide the encumbered code. Also OpenJDK doesn't yet build on Windows.

Sun have set up an Interim Governing Board to create a constituion for OpenJDK and to set up a proper Governing Board. The IGB has the following initial membership: Doug Lea, Fabiane Nardon, Dalibor Topić, and Sun's own Mark Reinhold and Simon Phipps.

The Apache Harmony technical session was well attended with more focus on the technology than on licensing issues and the JCK open letter.

Some anomalies still need working out. Sun said at JavaOne that it would release the JCK to the open source community, but later clarified that this meant the OpenJDK community. I guess that might even exclude forks of OpenJDK, but we'll have to wait and see.

Also, there is some strangeness in the way copyright assignment will be handled for eventual non-Sun committers to OpenJDK. The GNU definition of free software describes four freedoms including "The freedom to improve the program, and release your improvements to the public, so that the whole community benefits". So, in normal GPL projects, no-one has the right the make a fork without publishing their changes under the GPL.

Contrast this with the Apache license which allows anyone to create a fork and essentially do what they like with their changes, either publishing them or keeping them private.

OpenJDK is different from both these models. Sun, and only Sun, has the right to fork the code, make changes, and keep the changes private, although it seems unlikely that Sun would ever want to do that.

At this point when the source code is Sun's donation and when Sun is obliged to provide the source code to its licensees under the TLDA license, this is quite reasonable. But I wonder how many potential contributors will be happy to sign the Sun Contributor Agreement and give Sun these extended rights over their contributions.

Monday, May 14, 2007

Modularity at JavaOne

Modularity had a lot of focus at last week's JavaOne conference. There were multiple references to JSRs 277 and 294 by Sun speakers, specific technical sessions on these JSRs, and a combined JSR 277 and 294 BOF.

There were a number of questions and comments at the JSR 294 session on the strawman's structuring of superpackage membership information in a separate source file rather than distributed across the source files of the member classes. This is something we should discuss on the mailing list (which you can read via the observer list).

Apart from an OSGi Best Practices technical session and the JSR 291 BOF, there were several mentions of OSGi and JSR 291 in other sessions. The goal of interoperating well with JSR 291 was presented in the JSR 277 session and even mentioned in one of Sun's general sessions1. Stanley Ho will float a proposal on the JSR 277 mailing list in due course, so subscribe to the observer list if you are interested (Expert Group discussions should begin to be mirrored there soon).

Ethan Nicholas gave an update on the browser edition work to minimise the download and initial install size of the JRE. This seems to be making good progress, although it is not based on any of the above JSRs, so there may be some 'shake-out' in due course.

In one of the "meet the team" sessions, I asked Sun's class library team what their plan was for exploiting JSR 277 and JSR 294 in the JRE. It seems they don't yet have one. I expect this to change, particularly after the JSR 277 and JSR 294 prototypes become available in a subproject of OpenJDK.

In contrast, the Apache Harmony session described how OSGi had been used from the start to modularise the class libraries. I gave a demo. of prototype JVM support for modular Harmony in the JSR 291 BOF.

GlassFish v3 is developing a module system known as HK2 ("Hundred Kilobytes Kernel") which is loosely based on JSR 277. It also has a dependency injection framework similar to Guice. (Unlike GlassFish which is mostly licensed under the CDDL, HK2 currently has no license defined, so you may want to avoid looking at the source code until there is a proper license.)

I had face to face meetings with Andreas Sterbenz to discuss aspects of JSR 294 and with Stanley Ho and Michał Cierniak to discuss interoperation of JSR 277 with JSR 291. There's a lot of work ahead and we won't know for quite a while how good the interoperation with JSR 291 will really be, but everyone concerned seems to be well aware of the importance of this capability.

1. See a couple of minutes from Intro. and Technical Deep Dive starting at 18 minutes and 45 seconds from the start when Danny Coward talks about modularity and refers to the goal of interoperation with OSGi.

Wednesday, May 09, 2007

JSR 291 Final Approval Ballot starts

The FAB for JSR 291 began today (Tuesday 8 May) - just in time for tomorrow's JSR 291 BOF session at JavaOne.

Tuesday, May 08, 2007

The Eve of JavaOne 2007


This week's JavaOne should be an interesting experience as it's my first.

After checking in this afternoon, I thought I'd sample James Gosling's session at the end of the Netbeans day. The content/time ratio was rather low, so I went for a walk in the daylight instead.

Perusing the pocket schedule in a Mexican restaurant later, I was pleased to see the JSR 291 BOF had made it into print, although it was too late for the IBM flyer. It seems like the BOF is competing with the tail end of the Eclipse party and Oracle's free screening of Spiderman 3. I'll be lucky if my co-presenter turns up (only kidding Richard).

The online session scheduler (JSP) seems to be struggling to stay up. How many thousands of people are hammering it? It finally let me switch a session.

Well, even blogging can't keep me awake. It's 9pm and 22.5 hours since I woke up. Better get to bed - tomorrow looks like a long day with sessions more or less continuously from 8.30am to 10.50pm. Boy do these American's take work seriously...

Wednesday, May 02, 2007

Harmony, Guice, and OSGi

I've recently been tinkering with Google's dependency injection framework, Guice, and getting a simple example to run under the Equinox OSGi framework. It was straightforward except that there are a couple of extra dependencies (i.e. that you couldn't infer from the source code of the example) that need to be expressed as imported/exported packages because of a code generation library, cglib, that Guice uses.

This morning, I thought I'd try running the example using the Apache Harmony JDK 5.0M1 and it ran cleanly first time. I'm very impressed since Guice makes extensive use of generics and annotations and cglib is hardly typical application code.

OSGi for application modularity

InfoQ published an article on why one company has decided to use OSGi, in conjunction with Spring, for application modularity.

They are not starting from scratch, so it will be interesting to see how pleased they are with the result. On the face of it, OSGi is a good match to their requirements, particularly for versioning and for deploying application updates without restarting the server.

Tuesday, April 24, 2007

Monday, April 23, 2007

JSR 291 BOF at JavaOne 2007

After reconsideration, a JSR 291 BOF1 has been accepted for JavaOne 2007. The plan is for Richard Hall and I to lead the session.

I hope to include some thoughts on JSR 277 interoperation as well as JSR 294 (superpackages) exploitation.

1. BOF-21240 entitled "JSR 291: Dynamic Component Support For The Java SE Platform", Wednesday 9 May, 9:55pm-10:45pm

More OSGi assistance for Maven users

A former colleague, Stuart McCulloch, just released some tools for building an OSGi application using Maven.

Apparently it uses bnd under the covers, although the example provided seems to need quite a bit spelling out by hand. Since I'm only a neophite Maven user, it's hard to tell...

Monday, April 16, 2007

Jeff McAffer's blog

Jeff is a colleague from Eclipse who often has an interesting perspective. You may be interested in his new blog or the first post on provisioning changes in Eclipse.

Welcome to the blogosphere, Jeff!

Wednesday, April 11, 2007

JavaOne, OSGi, and Emacs

According to the advance publicity, there seems to be one session1 at this year's JavaOne conference devoted to OSGi and only a couple of others that refer to OSGi. Not a single mention of JSR 291, although I did try putting in a submission.

I noticed an interesting tidbit in the conference guide: "[James Gosling] has also built a WYSIWYG text editor; a constraint-based drawing editor; and a text editor called Emacs, for UNIX systems."

So did Gosling invent Emacs as well as Java? Well, no, he didn't. The truth is rather more interesting.

Richard Stallman wrote the first version of Emacs and Gosling produced a C version to run on UNIX. A twist is that a legal tussle over the Emacs source code provoked Stallman to write the first precursor of the GPL. Stallman has since endorsed Sun's use of the GPL for their OpenJDK project, so I guess he and Gosling are friends again.

1. TS-1419 "Best OSGi Practices" on Thursday May 10 1:30 PM - 2:30 PM

Tuesday, April 10, 2007

Spring-OSGi Milestone 1

Thanks to Hal for the link to Costin's blog on Spring-OSGi which recently shipped the first milestone build.

Now I'm on the look out for the first application to take advantage of Spring-OSGi. Any takers?

Monday, April 09, 2007

JSR 294 Expert Group

The JSR 294 Expert Group now has a publicly visible mailing list which is good timing as the Expert Group only kicked off a couple of weeks ago, so there'll be a complete archive of its deliberations.

It's a little disappointing that the newly revamped jcp.org site doesn't appear to be capable of hosting openly readable mailing lists as I would like that to be the default for all JSR Expert Groups.

Thursday, March 29, 2007

SIP Communicator and OSGi

Nice to see another project successfully using OSGi in the form of SIP Communicator over on java.net. It was good to see they upgraded from Oscar to Felix painlessly.

They are even participating in the Google "Summer of Code™", so more developers will see how useful OSGi is. Thanks to Greg Wilson for the link.

Tuesday, March 27, 2007

JSR 291 Status

There is some confusion about the effects on JSR 291 of jcp.org being in a mess. In particular the Final Approval Ballot won't be happening for at least a week or two.

The Proposed Final Draft spec. has been posted, but is inaccessible as far as I can tell. Fortunately, this is unlikely to matter. It's a pretty mature spec. which has had a few careful modifications due to JSR 291. The spec. is already undergoing a 45 day review by OSGi members. Also JSR 291 received no comments during the Public Review draft of the specification and only non-technical comments during the PR ballot.

So what's happening with the Final Approval Ballot? I submitted the FAB materials, but the email bounced after a couple of days so I just resent it. The spec. license needs a review by Sun lawyers because we used an IBM license rather than a standard Sun license, which will cause an additional unknown delay. After that, the FAB will begin. Hopefully jcp.org will be stable in time...

GlassFish Modularisation

Let's keep an eye on Glassfish and see what we can learn as they select a modularisation technology.

Monday, March 26, 2007

JSR 291 Proposed Final Draft

The PFD of JSR 291 has appeared on jcp.org, but the whole JCP site is broken at the moment due to some substantial changes that went in recently.

Thursday, March 22, 2007

Further interest in OSGi and SCA

Yesterday the Open SOA Collaboration announced they are planning to standardise the SCA and SDO specifications via OASIS.

A couple of the participants, IONA and Siemens AG, mentioned their interest in using OSGi with SCA. This is consistent with the OSGi Alliance's Enterprise Expert Group track on SCA and OSGi which IONA's Eric Newcomer mentioned back in January.

Monday, March 19, 2007

Background to Java modularity

Should Java have included modularity features from the start or is the kind of modularity really needed in Java rather different from that baked in to some other languages?

In 2003, IBM Research published a paper known internally as "modjava" but named "MJ" externally. The authors' approach was to create a static module system not dissimilar to the strawman for JSR 294 except that MJ was built from class loaders for ease of prototyping and didn't encompass versioning (although a couple of us refactored the internal MJ prototype to enable versioning).

Vernon Green and I then led a working group in IBM's Software Group Architecture Board on Java versioning and modularity which started from the MJ paper. Later we discovered OSGi and how that matched in many respects what the working group was looking for. I was given the task of taking our additional requirements into the OSGi R4 standards activity where I struck up a collaboration with Richard Hall who had a similar perspective based on his research work on the Oscar OSGi framework.

The real world requirements of versioning and dynamic modification of applications and supporting software without restarting the JVM have effectively forced Java to develop the new, and highly successful, OSGi technology.

It's interesting to speculate what would have happened if a static module system had been put into Java much earlier. My guess is that it wouldn't have bothered to address versioning or dynamicity requirements which only become crucial in the context of relatively large systems with continuous operation.

The JSR 294 Expert Group is just starting up, so this background suddenly becomes relevant...

JSR 291 Final Approval Ballot materials

As I mentioned to the JSR 291 Expert Group, I just submitted the specification, reference implementation, and technology compatibility kit for JSR 291 for the Final Approval Ballot by the Java SE Executive Committee.

Tuesday, March 13, 2007

Modularity and the Java platform

Neil questioned the point of putting modularity support into the Java platform.

Of course, OSGi does a pretty good job of modularity support based on java.lang.ClassLoader. So why isn't that sufficient?

At one level, OSGi is quite sufficient. There is already massive adoption of OSGi and this is continuing to spread into new and interesting areas.

At a more theoretical level, there are some things that custom class loader networks will never achieve.

Most importantly is VM enforcement of module boundaries. It is possible to obtain unexported Class instances from a class loader. OSGi bundles are implemented as class loaders. Similarly, JSR 277 modules will boil down to specialised class loaders. So this is where JSR 294 comes in with its objective of enforcing module boundaries in the VM.

This extra level of enforcement is irrelevant if the aim of using modules is to divide a piece of software into manageable units with well-defined interfaces.

However, VM enforcement starts to become interesting if modules need to protect some parts of an application or the system from inappropriate access. There is also some scope for additional, or at least simplified, performance optimisations if certain classes can be guaranteed never to be accessed outside their defining module.

I would dearly like JSR 294 to provide mechanisms that will be equally useful to OSGi as to JSR 277. I think OSGi would benefit from VM support, although it would be an exaggeration to say that OSGi really needs VM support.

JSR 277 and class loaders

Neil wondered what changes are likely to be made to java.lang.ClassLoader in support of JSR 277.

The Early Draft spec. lists some requirements for class loader behaviour in section 8.3.4. These boil down to class loaders being able to cope with more concurrency, especially in the presence of cyclic dependencies between JSR 277 modules.

Currently the deadlock exhibited by Sun's reference implementation has an experimental fix activated by a switch. There is some work in the pipeline to improve this situation for Java 7 which should also benefit OSGi at least when running on Sun's RI. Keep an eye on the Hotspot VM on OpenJDK if you are interested.

Monday, March 12, 2007

The state of Java modularity: JSRs 277, 291, and 294

People are pretty perceptive. They have picked up that there's a struggle to work out the right model for Java modularity. However, it's not a simple choice between JSR 277 and JSR 291 as many seem to think. Let's look at what's going on in a bit more depth.

Background

Firstly, a brief recap for the uninitiated.

JSR 277 is trying to create a static module system for Java 7. JSR 291 is referencing and extending the already mature OSGi dynamic component system while maintaining compatibility with JSR 232 which did a similar thing for Java ME. JSR 294, an offshoot of JSR 277, is focused on the VM and language support for modularity.

My personal involvement is that I belong to the JSR 277 Expert Group (but cannot disclose the discussions which are private to the group) and am the spec. lead of JSR 291 (which has an openly readable mailing list). I am also in the JSR 294 Expert Group, which hasn't started work yet.

Secondly, I want to be up front about my potential bias.

I have been involved in the OSGi Core Platform Expert Group for a couple of years during which time I co-authored some of the modularity improvements for OSGi R4 with Richard Hall (the leader of the Apache Felix OSGi project).

I work for IBM which belongs to the OSGi Alliance and which has based some of its products on OSGi. IBM and Sun co-operate on Java, but compete in related areas such as Java EE, not to mention UNIX-style operating systems and hardware. So IBM and Sun don't necessarily agree on everything that happens in Java. In fact, I think it would be pretty unhealthy if that were the case.

I should also re-iterate that my opinions are not necessarily shared by IBM.

That said, I'll try to lay out what's going on as objectively as I can.

Design points

The crucial thing to notice is that the goals of these JSRs are rather different even though the underlying technology is similar. JSRs 277 and 294 are building basic support into the Java SE platform whereas JSR 291 is pure Java built on top of the Java platform.

Looking at it another way, JSRs 277 and 294 are attempting to produce an evolution of java.lang.ClassLoader with a new distribution and packaging format, a new repository, plus new language features supported by the VM. JSR 291 on the other hand is implemented in pure Java on top of java.lang.ClassLoader and uses standard JAR files with standard manifests as a distribution and packaging format.

The design point of OSGi has always been to support a broad set of Java ME environments, such as profiles based on the Connected Device Configuration, and Java SE versions, from Java 1.2 onwards, in a compatible way. It has achieved this by building on commonly available Java concepts, notably class loaders. I think of OSGi as being the pinnacle of custom class loader systems. The way it manages to install, uninstall, start, stop, and update modules without restarting the VM is quite an achievement. Of course, some other systems have provided some of this capability, but I am not aware of any that have done it in such a generally reusable way.

So you can package OSGi with your application, or list it as a prerequisite, and then run on a wide range of Java platforms. JSR 277 on the other hand has the advantage that if you are happy to pre-req. Java 7 or later, then JSR 277 will be built in to the platform.

Not invented here?

So why can't the JCP simply adopt JSR 291 as a standard module system for Java? That's what many of the critics of JSR 277 suggest should happen.

One problem is that the OSGi specification, although mature, is still evolving. A number of new requirements were surfaced by the JSR 291 Expert Group. OSGi is rapidly spreading into new areas of application which are bound to result in additional requirements for OSGi R5 and beyond. To lock a particular Java SE version, such as Java 7, to a particular version of OSGi would be limiting as there wouldn't be an opportunity to upgrade the OSGi support significantly before Java 8. For example, an application that required some new features of OSGi wouldn't necessarily want to limit its deployment to Java 8 environments. So the ability to mix and match Java and OSGi versions as required is essential.

You may be wondering whether JSR 277 has the same problem, but since it is deliberately positioned to be the module system packaged with the Java 7 platform, its subsequent rate of evolution is, by definition, the same as that of Java. So the design point is that it has to be fairly basic in its support and not aim for the level of sophistication of OSGi, particularly in its support for legacy applications, which is one of OSGi's strong points.

The challenge for JSR 277 is to address its requirements in a sufficiently general way that it does not need to evolve quickly. The classic way of achieving this would be to divide the function into a lower level and relatively stable API and a user overridable piece implemented in terms of the lower level API. Like a module version of custom class loaders. JSR 277, as you can see from the Early Draft, is not adopting this approach but is attempting to produce one module system which can cope without being overridden. This will be quite an achievement if the Expert Group can pull it off.

The road ahead

So it seems there is a place for both JSR 277 and JSR 291. Of course, JSR 291 should ideally benefit from the features introduced by JSR 277 and JSR 294. But the immediate requirement is that JSR 277 and JSR 291 interoperate well so that modules written in terms of one JSR are able to make effective use of modules written in terms of the other JSR.

To achieve this, the Expert Groups of JSR 277 and JSR 291 will need to work together. I expect this to take the form of the JSR 277 Expert Group proposing a means of interoperating with other module systems and then the JSR 291 Expert Group reviewing this proposal and providing feedback.

Interoperation is certainly an interesting challenge to keep us occupied, but it's just part of the challenge of working out the right model for modularity in the Java platform.

Friday, March 09, 2007

BEA's experience of OSGi in building mSA

Another excellent talk BJ pointed out was BEA's "The Good, Bad, and Ugly of OSGi: What we learned building the mSA Backplane".

Not only is this a great OSGi adoption story, but it points out several requirements that could feed in to a future version of OSGi. :-)

See Alex Blewitt's summary of the talk if you don't have time to download the slides.

Wednesday, March 07, 2007

IBM, Cisco, and OSGi

Quite a week for OSGi! IBM and Cisco just announced a platform based on OSGi known as the "unified communications and collaboration Client Platform ", although UC2 (or should that be UC²?) is less of a mouthful.

There were some hints inside IBM earlier this week that some interesting OSGi news was about to break, but I had no idea what it was.

UC2 is based on a subset of the Lotus Sametime 7.5 product that is popular for instant messaging (at least inside IBM). It aims to handle IM, voice, and video.

BEA's use of OSGi

Greg Brail gave some insights into how BEA are using OSGi. Seems like they are decomposing their products into OSGi bundles which can then be composed into new products.

The WebLogic Server has provided some of these bundles, but whether it will be structured as a collection of OSGi bundles is not yet publicly known. But given BEA's commitment to OSGi, I'd be very surprised if WebLogic didn't eventually end up being componentised in terms of OSGi.

eWeek article on OSGi

I recommend reading the eWeek article mentioned by BJ. Apart from giving a useful survey of the history of OSGi, it lists a number of projects that are exploiting OSGi, including one I personally hadn't heard of before.

The article takes the popular, if rather simplistic, position on JSR 277's relationship to OSGi, which makes me a little uncomfortable since I'm trying to get the two to interoperate.

Part of the success of OSGi is due to the fact that it isn't bolted into a particular Java platform but provides compatible behaviour across a broad range of ME and SE platforms. I should expand on this point some time.

Oracle TopLink being donated to Eclipse

BJ posted a really interesting development from EclipseCon. Oracle is donating a sophisticated persistence framework to Eclipse.

I can't tell from the FAQ whether the framework consists of 'pure' OSGi bundles or whether they make use of any Eclipse-specific features, but clearly the former would be ideal.

Tuesday, March 06, 2007

OSGi universal middleware

The buzz around OSGi is increasing. Not only is there an exciting OSGi track at EclipseCon this week, but more Java developers are slowly realising that OSGi isn't just a buzz-word - it can actually help solve their versioning and deployment issues (and a whole lot more).

Take Marc Loggeman's thoughts on how he's thinking of using OSGi as one tangible example.

How many technologies have a mature spec. and three open source implementations, with friendly licensing, to choose from? There is no barrier to entry other than getting up to speed with the fairly basic principles of creating and managing OSGi bundles.

I like the EclipseCon term for OSGi: "universal middleware".

JSR 291 Proposed Final Draft submitted

I just sent the Proposed Final Draft specification for JSR 291 to the JCP Program Management Office. It should emerge for download/review in a couple of weeks, so watch this space.

The technical work is essentially done and now there is a wealth of administrivia to attend to before the JSR can go to the Final Approval Ballot and then, all being well, into Final Release.

Glad I've got the module system interoperation discussion going on in JSR 277 to keep me sane...

Monday, March 05, 2007

JSR 291 rubber-stamp?

My colleague Jim Colson was interviewed by SD Times for the article "OSGi, JCP Tussle Over Component Support in Java: Critics decry effort as a rubber-stamp move".

The piece notes that JSR 291 is an additional source of requirements for OSGi, but doesn't go into specifics. It ends by quoting Hani Suleiman as saying "I was told that this is in fact a rubber-stamping effort and that the OSGi spec will not change".

So is this just a matter of hearsay and opinion or are there some relevant facts? Did the OSGi specification pop out of JSR 291 without any input from the Expert Group?

Well a number of specific requirements were raised by the JSR 291 Expert Group and resulted in changes in the underlying OSGi specification.

Take a specific example of a non-rubber-stamp change: the addition of an API to enable declarative service support and other similar features to obtain the Bundle context of a specific bundle.

This requirement was initially raised by BEA on the JSR 291 Expert Group mailing list as part of a thread relating to SPIs. (I'm glad we decided to run the Expert Group mailing list in the open!)

Since then the new Bundle.getBundleContext method has appeared in the early draft and public review draft specifications and in the Equinox open source reference implementation.

Does that seem like rubber-stamping to you?

Thursday, March 01, 2007

Visualising dependencies between OSGi bundles

You may be interested in a tool for graphing the dependencies between OSGi bundles.

I haven't tried it, but I'd like to see what it makes of the dependencies of the Apache Harmony bundles. In the past we've run similar tools against class library code, but the resultant dependency graphs are pretty messy.

Focussing on OSGi will raise the level to package and bundle dependencies.

I like to structure large systems into layered collections of subsystems and sometimes the only way to achieve that is to see what a system's dependencies look like and then get refactoring.

OSGi gathering at JavaOne

Robert Varttinen is planning an OSGi gathering at this year's JavaOne. My plans are currently up in the air, but it sounds like a great idea for those that can make it.

Monday, February 26, 2007

JSR 291 Reference Implementation candidate

Alex's mention of the release of Eclipse driver 3.3M5eh (Canadian for "3.3M5a") reminds me to mention that the 3.3M5 version of Equinox is the current candidate for a Reference Implementation for JSR 291.

Download it here if you want to try out any of the new features mentioned in the PR draft.

Note that Equinox hasn't changed between 3.3M5 and 3.3M5eh.

Thanks to Tom Watson for another interpretation of "eh".

JSR 291 nears Proposed Final Draft

As I posted on the JSR 291 Expert Group mailing list, I hope to send the Proposed Final Draft specification for JSR 291 to the JCP Program Management Office on 5 March in which case it should pop up for review around 19 March.

It'd be nice if JSR 291 was pretty much done by this year's JavaOne.

Thursday, February 22, 2007

Decimal syntax and operator overloading

As mentioned previously, I'm keen for Java to provide nice syntax for BigDecimal types. Someone asked me whether I was trying to sneak full operator overloading into Java through the back door. It's a reasonable concern.

Much as I like some other features of C++, I was very comfortable that Java omitted general operator overloading. So I am definitely not suggesting that BigDecimal syntax be introduced along with general operator overloading.

Of course, if these syntax improvements go ahead, Sun are free to add them via a limited form of operator overloading mechanism, but that's an implementation choice from my perspective and certainly not part of the requirement.

Wednesday, February 21, 2007

Geronimo's use of OSGi

I've noticed a couple of references which have hinted at Geronimo being based on OSGi, so I wanted to clarify the current situation for the benefit of those not closely involved.

While we were extending the Equinox OSGi framework in parallel with the spec. work for the modularity improvements in OSGi R4, my colleague Simon Burns had a crack at modularising Geronimo to make sure there were no obvious gotchas. This was way before WebSphere Application Server had exploited OSGi, so I was keen for Simon to try this out to smooth the path for WAS.

Simon posted a brief summary on the Geronimo dev list, but we didn't follow up as the Geronimo team was undecided about whether they wanted to exploit OSGi or extend the Geronimo GBean technology.

Of course, I would love eventually to add Geronimo to the list of application servers based on OSGi...

JSIG material on OSGi and JSR 291

The presentation and demo's from last week's excellent JSIG meeting on OSGi and JSR 291 are now available from EclipseZone.

Friday, February 16, 2007

JSR 277 and JSR 291 Interoperation Strawman

For anyone who's interested, I contributed a strawman module system interoperation framework to Apache Felix for discussion there and in the Expert Groups of JSR 277 and JSR 291. I hope we can make these module systems work well together...

See a post to the JSR 291 EG mailing list and docs/README in the JAR attached to the JIRA issue for some technical background.

Thursday, February 15, 2007

OSGi and JSR 291 at JSIG

Today's JSIG meeting on OSGi and JSR 291 in London was well worth the commute.

I met beforehand for coffee with three guys from Paremus: Richard Nicholson (CEO), Mike Francis (Sales and Marketing Director), and Robert Dunne. They described their 'Infiniflow' product built, you guessed it, on top of OSGi. It runs on Knopflerfish and Equinox, although they hope to support Felix in due course. They ship the product's base layer, Newton, as open source. Newton provides basic services and is responsible for propagating SCA components, known as composites, around a network according to defined declarative specification documents.

I wondered if they might have to overcome IBM's perennial problem of creating software "by PhD's for PhD's", but they assured me there were simple use cases which made it easy for application programmers to get started.

The JSIG meeting was well attended with Alex Blewitt, the EclipseZone editor (on the left below), presenting and Neil Bartlett (on the right), running live demo's to illustrate the theory.


Alex explained the problem that OSGi solves and gave details of the key framework features. (I'll forgive Alex for the stunning slide transitions courtesy of Neil's Mac. Is he on commission?)

Neil's demo's were coded and built using NetBeans and included a Swing application to emphasise the point that OSGi is far from just 'an Eclipse thing'. He used Knopflerfish to run the demo's as it has an excellent GUI for installing bundles and starting and stopping them, etc.

They promised to dump the presentation and code on the JSIG website.

The audience, of around 30-40, seemed fairly lively and there were lots of questions, some of which were quite searching. It was good to bump into a former colleague of mine from Hursley. Sunny Chan worked on the initial version of the shared classes support we put into IBM's Java 5. He now seems to be enjoying working for an investment bank maintaining Java environments.

Afterwards I had lunch with Neil and Benoît Xhenseval of ObjectLab. The conversation centred on OSGi, although the need for decent decimal syntax in Java financial software came up again without my prompting. Must follow that up.

There was some mention of the various modularity JSRs throughout the day. It's amazing to see how different things look to complete outsiders (to the JCP). There seems to be a general bewilderment over JSR 277. Also, since some of these guys have only recently stopped supporting Java 1.3, Java 7 seems eons away from being the default platform for deployment.

The lasting impression of the day was how incredibly useful OSGi is for underpinning various kinds of commercial software. It seems that the OSGi Alliance has created the start of a potentially enormous value network which provides business opportunities for framework and middleware vendors, such as Paremus and ObjectLab, as well as to 'real' customers.

Wednesday, February 14, 2007

OSGi Alliance expands

In spite of the recent debate on SOA and OSGi, the SOA digest just published news that the OSGi Alliance has some new members: BEA Systems, IONA Technologies, Jayway AB, Eclipse, and Interface21 (the Spring guys).

The more the merrier!

OSGi at EclipseCon

Next month's EclipseCon features an OSGi track for the first time.

One of my close colleagues, Tim Ellison, will be talking about Apache Harmony, including its use of OSGi.

Another close colleague, BJ Hargrave, is co-leading a seminar on Spring-OSGi integration.

BEA are presenting their experiences of using OSGi to underpin the microService Architecture, while JBoss are presenting on their thoughts for how they could use OSGi.

All in all, sounds fascinating...

Monday, February 12, 2007

Is OSGi a SOA?

There's a heated debate in Markus Voelter's blog on the relationship of OSGi to Service Oriented Architecture (SOA). I must confess to having thought of OSGi as an example of a SOA until now.

One of the big benefits of SOA is a standardised way of lashing together myriad legacy (and new) systems. You may have seen the 'circuit diagrams' some organisations draw to illustrate the problems they face.

Wikipedia discusses SOA and quotes an OASIS definition:

A paradigm for organizing and utilizing distributed capabilities that may be under the control of different ownership domains. It provides a uniform means to offer, discover, interact with and use capabilities to produce desired effects consistent with measurable preconditions and expectations.

While OSGi can provide a very useful single process component model, it doesn't directly address distributed systems. Perhaps Markus should add 'distributed' to his list of SOA attributes.

Subsystems in OSGi

Neil Bartlett's excellent article on Eclipse Extensions versus OSGi Services touches on an interesting issue. OSGi services are always potentially dynamic - they can come and go when you least expect it.

Some developers may prefer always to use services to interface between bundles, but that is probably not a winning strategy for larger systems.

An ideal structure for such systems, IMO, is to group bundles into subsystems and have subsystems interact with other subsystems via services. Subsystems are not a formal OSGi concept, but they are an important concept in the mind of the designer.

Within a subsystem, bundles can depend on each other more tightly, e.g. by sharing classes. They could even use require-bundle for really tight coupling, although importing and exporting packages with a suitable attribute is sufficient to prevent subsystems from interfering with each other while maintaining the benefits of a somewhat looser coupling.
The benefit of tighter coupling in a subsystem is that there are fewer failure modes (types of error that can arise) due to bits of a subsystem coming and going when other bits of the subsystem least expect it.

In the example above, bundle a1 will not resolve unless its package import can be resolved. So by the time any code in bundle a1 runs, it can be sure that the dependency on a2 has been met and will not break for the lifetime of a1.

Friday, February 09, 2007

OSGi in the Enterprise

A new site for OSGi enterprise users (not just members of the OSGi Alliance) appeared the other day. More background on EclipseZone.

Thursday, February 08, 2007

The art of OSGi programming

Neil Bartlett recently published the first of a series of EclipseZone articles on OSGi programming.

Since most OSGi users I've talked to are implementers of the bowels of Eclipse, I'll enjoy seeing how Neil uses OSGi.

If you don't use Eclipse, don't be put off. Implementing and running the bundles in these articles will require only an editor and a command line shell (or two).

Wednesday, February 07, 2007

Nuxeo EP based on OSGi

The recent announcement of Nuxeo Enterprise Platform v5 on Freshmeat hinted that they had adopted an Eclipse-like extension mechanism based on OSGi. The Nuxeo homepage made me wonder if this was just buzzword compliance. But it seems not.

Eric Barroca's blog describes the use of OSGi in a little more detail. Nuxeo exploits OSGi when it is available and mimics it otherwise. I wonder why they didn't bundle an existing OSGi framework?

It'll be interesting to see whether JBoss decides to bundle an existing framework...

Tuesday, February 06, 2007

Java 7 Content

There's increasing interest in what might go into Java 7. The spec. lead has given some hints in his blog and a recent presentation.

However, Alex Miller's Java 7 page seems to be the definitive collection of candidate items.

I'm particularly interested in:

Knopflerfish OSGi tutorial

Thanks to some Australian OSGi users for providing a nice link to a Knopflerfish OSGi tutorial.

My example of how to create an OSGi bundle in under 10 minutes still takes some beating for simplicity (or triviality, if you prefer), but I do believe in giving beginners a really easy way in. If they're smart, they'll soon catch on.

Monday, February 05, 2007

Osxa OSGi

Pierro's excellent summary of open source OSGi implementations includes the three long-standing projects, but also mentions Osxa.

Osxa seems to have started recently given its relative incompleteness and also the state of its web site. If you know about the objectives of Osxa, I'd love to hear from you.

Friday, February 02, 2007

BigDecimal operator support in Java 7

One of the candidate items for Java 7 helpfully listed by Alex Miller is BigDecimal operator support. I'm keen to see this go in as it would really help customers who use decimals.

Mike Cowlishaw's General Decimal Arithmetic page is a rich source of information on this subject. It contains an example 'telco' benchmark coded in Java and C# which provide a helpful comparison when thinking about syntax support for BigDecimal.

Take the following sequence (massaged a little for display here) from the Java version:


b = p.multiply(basetax);
b = b.setScale(2, BigDecimal.ROUND_DOWN);
sumB = sumB.add(b);
t = p.add(b);

if (calltype != 0) {
d = p.multiply(disttax);
d = d.setScale(2, BigDecimal.ROUND_DOWN);
sumD = sumD.add(d);
t = t.add(d);
}

and compare the C# version for readability:

b = p * basetax * 100;
b = Decimal.Truncate(b) / 100;
sumB = sumB + b;
t = p + b;

if (calltype != 0) {
d = p * disttax * 100;
d = Decimal.Truncate(d) / 100;
sumD = sumD + d;
t = t + d;
}

Enough said?

Thursday, February 01, 2007

OSGi Enterprise Expert Group Kick-Off

I'm grateful to Eric Newcomer for an encouraging summary of the recent OSGi Enterprise Expert Group kick-off meeting. I toyed with going and decided my attendance wasn't essential. But it certainly sounded like a lot of fun.

Wednesday, January 31, 2007

JRockit-powered Harmony

Excellent news! BEA have made available a binary version of the JRockit VM for use with the Apache Harmony class libraries. See their post on harmony-dev.

As Geir notes, Harmony now has two decent VMs to choose from. Anyone planning to hack Hotspot to run with Harmony?

Tuesday, January 30, 2007

Vista - the "Pain" starts now

Good to know Microsoft are ahead of the game. They shipped the first, and hopefully the last, critical security update a good two weeks ahead of yesterday's worldwide release of Vista.

Good job they rewrote the code from the ground up...

Writing partial designs

I recently kicked off a piece of development work and then handed over leadership of the small team to someone else. The new leader is keen and thorough, but I don't think I've really persuaded the rest of the team of the value of writing down partial, incomplete designs.

The developers in the team seem to have the impression that doing design involves solving all the problems and then writing up a solution. But it's much better, as I've mentioned before, to use a design document as a work in progress which records the solution so far and has placeholders for all the known problems. Such a 'live' document can be shared with other people and used to stimulate new ideas and solutions.

Rock climbing provides a good analogy. A team can climb up a rock face without pausing to bang in pegs. But then one person's achievement in reaching a particular point of the climb can't be shared by those following behind. Also the effects of a fall can be catastrophic.



In a similar way, writing down partial designs enables a team to build on each other's progress. If a major design problem turns up, it is often possible to reconstruct a solution from the remains.

This sharing of design information can be done verbally, but this lacks the precision of a written document and doesn't force out so many issues.

Monday, January 29, 2007

What's the point of JSR 291?

It seemed time to re-iterate the reasoning behind JSR 291. For example, Sun's recent voting comment showed they apparently still don't understand what the JSR's there for.

Adoption
There's a ton1 of software already built on top of OSGi and it keeps increasing. Since this is mostly Java code, the JCP ought to take notice, and sooner rather than later.

Familiarity
As Sun point out, the specification and its expert group already exist thanks to the OSGi Alliance, so what could standardisation in the JCP possibly add?

Well, there are benefits to the JCP and to OSGi. The Java community gets access to the specification, RI, and TCK in the familiar, JCP-standard way. OSGi gets another source of requirements, continuing its tradition of engaging with current and potential users.

Compatibility with Java ME
JSR 232 does a similar job to JSR 291, but for Java ME. Having a consistent, compatible dynamic component system across Java SE and Java ME is a no-brainer.

Footnote:
1. Although good project managers use KLOCs only in joking terms these days, no better metric has gained universal acceptance. The 'ton' is a strong contender as it refers to the mass of software, rather than its volume. 'kiloton' and 'megaton' are useful for larger software projects, especially in the defence sector.

Work-life balance

In a recent IBM telecon, a senior manager mentioned that a 60 hour week felt about right for her. This was in the context of a discussion about work-life balance among the more senior techies in her organisation. There was no implication that everyone should work those kind of hours, although I guess a fair number of those on the call do.

A few days ago I heard a speaker on technical leadership say that no-one should expect to become a senior technical leader by working a 37 hour week.

But a 37 hour week feels about right for me. Maybe the promotion process is broken because it doesn't take into account working hours. I don't think so. Results are what counts rather than being physically present. And, of course, no-one should expect to be promoted unless they produce results.

I used to work longer hours and I know this makes me ineffective. I lose most of my creative spark. You know the kind of thing: you're struggling with a seemingly intractable problem and then a solution pops into your head when you're thinking about something else (or not thinking about anything in particular). The trick seems to be to load the problem up in your brain and, if you can't make progress, switch to doing something completely different, like going home or taking some exercise.

Some people can work long hours and still be creative. Others seem to enjoy spending most of their waking hours at work. Each to his or her own, but the crucial thing is to make a choice that you won't regret later, which I hope I've done.

Friday, January 26, 2007

Is full interoperation realistic?

Wouldn't it be useful if applications could communicate over a network using a standard message format and protocol? Wouldn't this solve a lot of real-world problems, especially if the whole thing could be kept simple?

Sure, but before long you'd have to address the 'full interoperation' requirements: distributed transactions (so application programmers don't spend most of their time writing error handling logic), security (for sensitive data), a directory server (so distributed applications can find each other; ideally secure and transactional), load balancing, distributed debugging, etc.

And since real world systems don't all run on a single vendor's hardware or software, most of these features would need standardising so applications running in different environments could communicate with each other.

Now if the probability of each feature interoperating correctly was, say, 99.9% and there were, say, six features, then the probability of all six features interoperating correctly would only be about 99.4% (0.9996).

You'd end up with a system that would be much more complex than was originally envisaged and the interoperation wouldn't be great.

But, hey, what if you stepped back and came up with a standard message format and protocol so that applications could communicate over a network? Wouldn't this solve a lot of real-world problems, especially if the whole thing could be kept simple? Indeed it would ... if.

Many vendors are following the full interop. model with the likes of CORBA (e.g. via Java EE) and SOAP. Others, notably Microsoft, support full interop. but would much prefer a homogeneous (software) network also viewed as 'single vendor lock in' (or world domination).

I don't think there's a perfect, multi-vendor solution, desirable as this would be. The most flexible approaches that I've seen work in practice are distributed systems consisting of homogeneous 'islands' loosely connected with relatively simple protocols (such as MQ Series messaging, IIOP, or vanilla SOAP).

Thursday, January 25, 2007

JSR 291 and weird stuff

Google's blog search is cool, but it sometimes turns up weird stuff. Take ocean's recent thinking on the curse of specifications in general and JSR 291 in particular.

Actually 'thinking' may be an exaggeration, but the blog does raise some questions:

  1. Why does it link back to the JSR review ballot of March 2006? Things have moved on.
  2. If specifications are so divisive, are there any non-divisive alternatives?
  3. Is rolling your own module system from class loaders really better than using an existing module system that others have taken the time to hone? Presumably ocean codes in assembler on principle.
But I should lighten up, as our American cousins are fond of saying, ignore the weird stuff, and enjoy the blog's link to WS-DeathStar.

Wednesday, January 24, 2007

IBM Java 6 /.'d

A post on Slashdot about the early release of IBM's Java 6 is generating some interesting discussion. But, from the wording, I don't think IndioMan has ever worn a blue suit. ;-)

Tuesday, January 23, 2007

JSR 291 passes public review ballot

In the last hour, JSR 291 passed its Public Review ballot with 9 "yes" votes, 2 "no" votes, 1 abstention, and 4 non voters (including Google and Red Hat who voted "no" in the initial JSR approval ballot). Details here.

Thursday, January 18, 2007

Is hyper-threading hot stuff?

My home machine recently exceeded the processor temperature threshold when one of my sons was thrashing the CPU with a game. I tried various things like hoovering out the dust and increasing the fan speed, to no avail.

But disabling hyper-threading in the BIOS did the trick, with no perceptable decrease in performance. I was expecting Windows startup to be slower, but I guess it's mainly gated by disk access.

An old ZDNet article claimed that hyper-threading "increases power consumption and thus heat dissipation" which fits my experience.

I could try forcing the game to use a single logical processor, but why bother as I don't seem to need hyper-threading anyway?

Wednesday, January 17, 2007

Silly Java rules?

Kevin Barnes observes that some pretty silly rules have developed around the use of Java.

I should declare my interest at this point: I work on Java standards and IBM's Java runtimes. That said, I think some rules of thumb are useful for beginners provided that skilled programmers break the rules as necessary.

I dislike classes with numerous getter and setter methods as this is a pretty strong hint that a class is badly designed, or that the programmer would have preferred a C struct ;-). Some consider these methods inherently evil, but I wouldn't go that far.

The use of final is more interesting. In C++, const is useful in defining good interfaces and final may play a similar role in Java. But the use of final on local declarations doesn't seem to be worth the syntactic clutter.

One of the proponents (thanks for the link, Kevin) of the use of final seems more concerned about controlling subclassing. That's a more interesting point to argue as it comes back to good interface design. But I don't think there's a general rule - it depends on the circumstances. Anyone designing a Java interface needs to understand the trade-off.

Since Java enshrines 'non-final' as the default (and it's too late to change that!), beginners risk coding Java interfaces that will behave unexpectedly when subclassed. But changing the language default wouldn't enable beginners to design good interfaces. As they say "there's no substitute for understanding".

Scrapping designs

When should you start writing down a design of a piece of software?

Some programmers put this off as long as possible. Once you commit yourself to paper, you start to become locked in to a particular direction. The more you write, the more you become wedded to a particular approach.

Others might advise skipping the design stage and getting on with coding. But the design lock in is even stronger once you have a swathe of code and a decent test suite. Refactoring is fun in small doses, but you'd have to be a masochist to want to spend a lot of time on it.

On the other hand, until I start writing down and organising my thoughts, I can't flush out most of the issues. Putting ideas on 'paper' forces me to define terms and helps me notice inconsistencies and omissions. Maybe I'm just a frustrated mathematician who can't solve a problem without a pencil in my hand?

No, seriously, writing a design down can be really helpful. Once you've done it, you can let the document go 'cold' for a while and then come back to it with a critical eye. It's hard to be objective when ideas are confined to your head. Also, dumping issues in a document helps me stop worrying about them - especially important when I'm trying to sleep.

The best compromise seems to be to start writing things down fairly early but to scrap a design and start again to free yourself from 'lock in'. If you use a decent version control system, you can always retrieve deleted versions.

Tuesday, January 16, 2007

JSR 291 public review ballot

The Public Review of JSR 291 lasts just a few more days and the Executive Committee ballot starts today.

The JSR review ballot, just under year ago, passed with eleven EC members in favour, four against, and one abstaining, so it will be interesting to see if any positions have shifted.

Red Hat have recently joined the Expert Group because of the interest of JBoss (a division of Red Hat) in using OSGi, so I'm hoping Red Hat are now comfortable with the rationale for the JSR and will vote in favour.

Some EC members were worried that the JSR would be a rubber-stamping exercise, but since the Expert Group have raised a number of requirements which resulted in spec. changes, these fears should have been allayed.

Monday, January 15, 2007

Harmful distractions

Matthew Stibbe discussed distraction-free text editors from an author's perspective. Geir Magnusson made a similar point about WriteRoom. Windows users have DarkRoom. But, heck, anyone can set their IM to 'busy' and maximise the window of their favourite editor.

The point is: writers are easily distracted. Anything in their field of view is likely to entice them away from the task of writing. The same is true for software developers. Coding is sufficiently addictive for this not to be an issue, but writing specifications or other design documents is like any other kind of writing, with many of the same pit-falls.

A more insidious source of distractions is multi-tasking. Supposed scientific studies that show interruptions are worse for IQ than smoking dope or coding in VB need taking with a pinch of salt, but I know the effect rapid multi-tasking has on me (and it's not good).

So, in 2007, I plan to use my time more effectively. I already block out two days per week in my calendar to avoid 'death by meetings'. But I need to set aside chunks of time for important work and try to resist the temptation to check email, etc. every few minutes. And maybe setting my IM to 'busy' and maximising the current window will help.

Friday, January 05, 2007

Creating an OSGi bundle

Newcomers to OSGi may like a simple guide to developing a bundle. Peter Kriens has provided an extensive tutorial (December 2008: currently offline pending an overhaul) which is a must for anyone serious about learning OSGi. However, it has a long introduction and assumes you are happy to run Eclipse. So I thought I would provide a trival example that people could get going using only a Java SDK and their favourite text editor.

Here are instructions to create and run a trivial bundle, hopefully in about 10 minutes.

1. Download an OSGi framework (I used Equinox v3.2.1).

2. Paste the following code into the file org/foo/example/Example.java:

package org.foo.example;

import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;

public class Example implements BundleActivator {
public void start(BundleContext bc) {
System.out.println("Hello world");
}
public void stop(BundleContext bc) {
System.out.println("Goodbye world");  
}
}


3. Compile the code:
> javac -cp org.eclipse.osgi_3.2.1.R32x_v20060919.jar org/foo/example/Example.java


4. Paste the following bundle manifest into the file MANIFEST.MF:
Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-SymbolicName: org.foo.example.Example
Bundle-Version: 1
Bundle-Activator: org.foo.example.Example
Import-Package: org.osgi.framework;version="1.3.0"
Make sure you have a carriage return and/or newline character at the end of the last line as this is required (see manifest specification in the JAR specification).

5. Package the manifest and code into a bundle:
> jar cvfm example.jar MANIFEST.MF org

Note that the order of the flags is important - the f and m options need to appear in the same order as the jar file and manifest file parameters.

6. Run the resultant bundle example.jar interactively using the Equinox console, e.g.:
> java -jar org.eclipse.osgi_3.2.1.R32x_v20060919.jar -console

osgi> install file:///osgi/blog/example/example.jar
Bundle id is 6

osgi> start 6
Hello world

osgi> stop 6
Goodbye world

osgi> close

(close exits the framework).

That's it! Anyone care to comment how long it took them?

OSGi non-assert agreements

In a comment on Niclas Hedman's blog, Dalibor Topic wrote:

What weirds me out about OSGi is the licensing of the real specs:

"You are not authorized to create any derivative work of the Specification. The OSGi Alliance also grants you a perpetual, non-exclusive, worldwide, fully paid-up, royalty free, limited license (without the right to sublicense) under any applicable copyrights, to create and/or distribute an implementation of the Specification that: (i) fully implements the Specification including all its required interfaces and functionality; (ii) does not modify, subset, superset or otherwise extend the OSGi Name Space, or include any public or protected packages, classes, Java interfaces, fields or methods within the OSGi Name Space other than those required and authorized by the Specification. An implementation that does not satisfy limitations (i)-(ii) is not considered an implementation of the Specification, does not receive the benefits of this license, and must not be described as an implementation of the Specification. An implementation of the Specification must not claim to be a compliant implementation of the Specification unless it passes the OSGi Alliance Compliance Tests for the Specification in accordance with OSGi Alliance processes."

The silly talk about subsetting in that license makes Equinox, Felix and all the other OSGi implementations developed open to patent attacks from OSGi patent holders, since they are naturally going to be subsets until they implement whatever the current full specification is.

Are those compliance tests freely available somewhere, so that re-distributors can limit their liability to being attacked via software patents on OSGi, by checking which implementations actually conform to the tests?
His fears should be addressed, at least for OSGi R4, by OSGi's non-assert agreements which provide royalty free access to patents for implementers of OSGi. Implementations, in this sense, do not need to pass the TCK, although they must be "intended to accurately implement the normative portions of [the OSGi R4] specification (subject to normal, inadvertent programming errors)" and must not "intentionally introduce incompatibilities that hinder interoperability".

Implementers that want to claim OSGi compliance may obtain the OSGi R4 TCK in three ways:

  • Pay to join the OSGi Alliance.

  • Pay for the TCK as part of JSR 232.

  • I believe the JSR 232 TCK is available at no cost to qualifying open source projects and not-for-profit organisations.


I expect the same to be true of the OSGi R4 V4.1 TCK for JSR 291.

Thursday, January 04, 2007

Application servers based on OSGi


An increasing number of Java application servers are being based on OSGi along the lines shown in the figure. An OSGi framework provides a runtime environment for the application server's modules. As the application server initialises, its modules are started and can communicate between themselves either using services or by importing classes from each other.

Applications running inside container modules provided by the application server need not be aware of the OSGi framework. This provides a low-cost migration path for existing applications.

The following application servers are based on OSGi, or are thinking about it:

I've heard of some others but couldn't find reasonable links.

Modularity in proprietary and open source projects

In successful proprietary software projects components are carefully owned and managed. A component owner’s perspective is that there is a set of responsibilities which are the remit of that component and no other. So the owner takes care to encapsulate the data and function of the component. Ideally, and to a large extent in practice, when a new requirement needs to be satisfied, component owners negotiate and divide up the responsibilities necessary to satisfy the requirement. Responsibilities are assigned according to the remit of each component, although these remits may be extended or refined over time.

Accessing another component’s implementation detail is frowned upon and known to impact reliability, serviceability, and maintainability. Since each component’s objectives are ultimately subservient to those of the project, component owners try hard to hide implementation detail from other components. In particular, component owners are motivated to reduce the cost of servicing old releases.

The motivation of developers on open source projects is markedly different. Of course, they too are concerned to deliver components with useful function which are in themselves reliable and which can be evolved in the future. However, they have limited resources and each component has a potentially large set of users who will not necessarily agree about the precise remit or objectives of a component. So it is in the interest of an open source project to keep components sufficiently open that they may be extended by other projects to produce variants.

A common experience in open source is that, as soon as some implementation detail is in any way declared ‘private’, a developer from a related project will complain and argue persuasively why they need special access to the implementation detail. Since there is no overriding set of objectives and open source projects are not responsible for each other’s code, the path of least resistance is usually taken and the ‘private’ declaration is removed. If the implementation detail later changes and projects depending on that detail are impacted, then no blame is attached to the project that made the change. The attitude is that the other project got what it deserved. The onus is therefore on ‘consumers’ from other projects to select the type of dependencies they wish to create and to manage these dependencies appropriately.

The Eclipse open source project faced similar resistance to declaring implementation details private. So they added an Eclipse-specific convention of using a ‘x-internal’ directive to plugin manifests to flag certain implementation packages as ‘internal’ to the module that exports the package. These packages may still be used by other modules, but the fact the packages are internal is explicitly acknowledged.

Saturday, December 30, 2006

The failure of string theory?

My holiday reading is a fascinating book by Peter Woit entitled "Not Even Wrong" which argues that string theory has failed. There was quite a backlash from certain outspoken string theorists, but no-one appears to have rebutted Woit's specific criticisms.

The gist of the book's argument is that string theory isn't really a theory at all as it makes no experimentally falsifiable predictions and it doesn't yet have an underlying mathematical foundation. It's really a highly fashionable piece of wishful thinking about where the next fundamental theory of physics might come from. I can think of other fashion industries closer to home, but that's a topic for the future.

I think the crucial question is how long a speculative search for a new physical theory should continue before it makes a prediction that can be checked experimentally. Perhaps string theory will ultimately deliver something useful, but, in the interim, other directions are effectively being starved of resources.

The book mentions in passing that the Massachusetts Institute of Technology (MIT) is now publishing its courses online under an initiative known as "Open Course Ware". If you want an easy example of a computing course, take a look at the lecture notes on module dependence diagrams, given by the Department of Electrical Engineering and Computer Science.

I've been investigating a mathematical model of software modules recently and some of the dependency relations look like they would benefit from a bit of graph theory, so I was delighted to find some mathematics course notes and even a link to a softcopy graduate textbook on graph theory via the MIT site.

Unfortunately, the graduate course notes are handwritten and then scanned in, so would need more effort to explore. However, since I'm a novice at graph theory, I was pleased to find some undergraduate material nicely captured in PDF files.

Tuesday, December 12, 2006

AOP meets OSGi - phase II

More news on this interesting project to fuse aspect orienting programming with OSGi: phase II is now released.

Now you can apply aspects to unmodified bundles -- the so-called "co-opt" model. This feels more natural since AOP is usually non-invasive.

Thursday, November 16, 2006

Experimental fix for sunbug 4670071

The Hotspot VM has a long-standing class loader deadlock, sunbug 4670071, which is top of Sun's bug parade. When the VM loads a class, it calls the loadClass method of java.lang.ClassLoader via a private, synchronized loadClassInternal method. The result is that multiple threads using multiple class loaders can deadlock.

This is a particular problem for networks of custom class loaders in JBoss, OSGi, JSR 277 (see section 8.3.4 of the EDR spec.), and elsewhere.

Sun have implemented an experimental fix, which I'll come to shortly. The point of describing the fix here is to make the caveats clear and to solicit feedback which can be taken into account in future releases of Hotspot. I, as a member of the JSR 277 Expert Group, am talking to the engineers in Sun who are looking at potential class loading improvements for Dolphin.

So what's the fix? Well, it turns out you need to specify some VM options:

-XX:+UnlockDiagnosticVMOptions -XX:+UnsyncloadClass

Please note that I am not suggesting that this is a robust fix to bug 4670071. You must treat it as unstable, experimental, temporary, unsupported, etc. In particular a custom class loader's loadClass method must be threadsafe and findClass should be synchronized to rule out various races in the VM.

However, it's not all bad news: the fix seems to work in some cases. For example, Equinox supports unsynchronized calling of loadClass. This is the default behaviour in the IBM JDK and can occur with Hotspot when loadClass is called directly from Java code (rather than implicitly by the VM while loading another class). Equinox's DefaultClassLoader synchronizes findClass.

Wednesday, November 15, 2006

Early release of IBM's Java 6

The Early Release Programme for IBM's Java SE 6 on a number of Linux platforms just began.

Apart from the functional enhancements inherent in Java 6 and the additional performance and RAS features we put into this release, there were significant changes in the development process.

Of particular note is the way we handled the class library code licensed from Sun. In Java 5 and other recent releases, we carried forward a base from release to release and merged in Sun's changes -- fairly labour intensive. This time we introduced a much more efficient process.

We developed a system to capture the IBM changes to Sun code such as bug fixes, performance improvements, and support for IBM platforms. Each logical group of changes is captured as a "smart patch" based on a UNIX patch file. The system automatically applies all the smart patches to each new drop of Sun code. Manual intervention is only required where the Sun code has changed significantly in an area covered by a smart patch.

The net effect is that we have been able to track the development of Sun's Java 6 much more closely than in the past. Also, the changes we make have been separated out in to logical, manageable units.

I'm hoping to contribute the smart patch system into open source as it would be useful to any project that needs to apply its own changes to a moving base.

Monday, November 06, 2006

More on JSR 277

InfoQ joined the fray. Stanley Ho's latest post drew a few comments.

Murphee pipped Mike Milinvokich and Alex Blewitt in the prize for least understated. Compare Robilad.

Friday, November 03, 2006

Public discussion of JSR 277

I'm pleased to see some public feedback.

The net of the negative feedback, e.g. on JavaLobby, seems to be "why re-invent aspects of OSGi/Maven/Ivy?". Peter Kriens' review describes some technical issues previously encountered and solved by OSGi.

A relatively positive review asks why the Expert Groups of JSR 277 and JSR 291 didn't work more closely together. BEA, IBM, Intel, Oracle, Richard (Felix) Hall, and Sun are members of both Expert Groups, so lack of working closely together isn't the reason the results are turning out rather different.

JSR 277, with the help of JSR 294, is aiming to produce a static module system integrated into the Java SE 7 platform. JSR 291 on the other hand aims to support dynamic components on existing Java SE platforms in a way which is compatible with JSR 232 on Java ME.

Looking at it another way, JSR 277/294 are attempting to produce an evolution of java.lang.ClassLoader with a distribution format, repositories, language, and VM support. JSR 291 on the other hand can be implemented in pure Java on top of java.lang.ClassLoader.

Wednesday, November 01, 2006

OSGi introduction

Some of my colleagues need to develop OSGi bundles and run them on Equinox. They are new to OSGi, so they asked me for some introductory reading. I thought I'd post it here for posterity and more general consumption.

I recommend starting with the technology overview on the OSGi web site, but skipping the sections on specific services on the first reading.

After reading this, you should have a feel for the three major layers in the OSGi framework.

The modularity layer enables you to package Java code with some metadata into a JAR known as a bundle, which is OSGi's module concept. Each bundle is loaded in its own class loader. Bundles share classes in a couple of ways. A bundle may import a package that another bundle exports or a bundle may require another bundle, which is similar to importing all the packages that the other 'required' bundle exports. Bundles may also contain native libraries and arbitrary kinds of resource files.

The life cycle layer allows you to dynamically install, uninstall, start, stop, and update bundles. This is essential when the OSGi framework needs to run for a long time but applications need to be managed, e.g. taken offline or have a bug fix applied, without restarting the framework, which would impact all the applications running in the framework.

The service layer enables bundles to provide services to other bundles. A service is a Java object that implements an agreed interface. Services are registered in a service registry so that users can locate the service they need, cast it to the agreed service interface, and then call the service.

If you would like to re-enforce these concepts or get a bit more detail, take a look at the architecture description.

After that, you could read about an example application, you could dip into the core specification, or you may prefer to try developing and running a simple bundle using one of the open source OSGi framework implementations. For example, the Equinox OSGi framework comes with a quick start guide.

Finally see this guide to writing a trivial bundle in 10 minutes with no special tools. Once you get beyond a "hello world" bundle, you'll want to refer to the javadoc on the OSGi web site.

Projects

OSGi (130) Virgo (59) Eclipse (10) Equinox (9) dm Server (8) Felix (4) WebSphere (3) Aries (2) GlassFish (2) JBoss (1) Newton (1) WebLogic (1)