netzooid

I’ve been trying out Ubuntu as I need to use Linux for some work related things. Here are my impressions so far.

Things Which Rock

• Having a real Unix Shell
• Its a bit faster than Windows for things like SVN
• Install was quick
• Wifi works better than Windows I think!

Things Which Suck

• Fonts suck big time on Linux still. Even will all the tweaks (Installing MS fonts, turning on autohinting, etc), they still just don’t compare to Mac OS X and Windows. I have a hard time looking at them all day.
• Dual monitors seem to require a lot of extra set up
• Docking/undocking my T60 doesn’t really work
• Skype on Linux isn’t as good as Skype on Windows
• Bugs: X & Java, Rhapsody in the browser
• No desktop Rhapsody client
• Verizon card doesn’t work out of the box
• UI is extremely inconsistent in Gnome
• Webex doesn’t seem to work out of the box (I installed the Java browser plugin, but still no go – it just froze up)
• OpenOffice…

Let me know if I’m wrong on any of these accounts or did something ridiculously stupid.

Verdict: still not ready for prime time.

I recently submitted a piece for CIO.com where I ranted about UDDI for a bit. Be sure to see the Microsoft counterpoint which is… interesting to say the least. More on that later. While comments are always welcome here, be sure to leave comments on the CIO.com piece as well!

Without further ado:

The registry landscape needs to change. The unholy trinity of SOAP, WSDL and UDDI needs to go. While registry vendors, such as Systinet, have pioneered a number of good ideas in the field, they have failed to meet the needs of enterprise integration projects today, because they are based on these technologies. One should be able to guess this just by seeing the limited adoption, low success rate and high price tags of these solutions. Mass adoption is just not in the cards.

If we’re to understand why this approach for registries has failed, we need to look to the bottom of the stack: SOAP/WSDL-based services. Over the last decade we’ve learned that they’re tightly coupled—making distributed, anarchic change highly difficult.

Yes, your enterprise is a bit anarchic. Business needs change rapidly. To keep up with these changes, IT scales and evolves rapidly. To do this, enterprises have been adopting a RESTful approach to building highly scalable services. RESTful architectural principles are the ones behind HTTP, and allowed the Web to scale and evolve at such a rapid pace.

Even if SOAP/WSDL was the right answer, we will never be able to adopt a single type of service. All technologies have tradeoffs associated with them, and we should never assume a homogenous enterprise environment (not to mention the massive cost of switching everything over). Because UDDI has been designed around the concepts of SOAP/WSDL, it has failed to evolve and address this problem.

There are yet more problems with UDDI, though. A lot of effort has been spent on seldom-required use cases. For instance, it’s often recommended that people use it at runtime to dynamically look up services. In reality, this is often dangerous, and hardly ever done. You can easily end up with a service you don’t want, whether it’s because the functionality is slightly different, the service is slower, or it lacks redundant backups.

On top of this, UDDI is also quite hard to use. It seems people were so confused, the technical committee had to write another long document explaining how it could be used as a registry with SOAP/WSDL services. Then Systinet had to write another long document called the Governance Interoperability Framework to make it actually possible for inter-vendor communication using UDDI.

Where does this leave us? We need to leave this whole stack behind. Registries need to be more agile, easier to use, and focus on things that matter for SOA projects today. We’ve been pioneering a new approach with Mule Galaxy. It uses a simple, extensible services model which enables you to reap the benefits of registries—service visibility, contract management, policy management, etc.—while remaining in a heterogeneous environment.

It also uses a new type of registry API based on the Atom Publishing Protocol (AtomPub). This API, which is built on Atom (the successor to RSS), enables integration using a simple, RESTful mechanism. This RESTful approach, combined with our flexible services model, reflects the way that enterprises actually do business, not some vendor-imposed theory.”

I was giving a demo to a company today. The following sequence of events happened:

• Webex froze my computer, not once, not twice, but three times – requiring full reboots. (It seems it doesn’t work well with dual monitors)
• Construction knocked out my cable – which I use for phone and Internet
• I called back in on my cell phone and my call dropped

Sometimes it is just not your day.

Folks, we managed to push out a Galaxy 1.5 release candidate yesterday. It has some pretty cool new features I think:

• Much expanded metadata capabilities. You can define new properties which are stored of various types – like links to other things in the registries, lifecycles, users, and simple values. For instance, you can define a property called “supercedes” which is a link type. Than users can link to another entry in the registry that a particular service/artifact supercedes.
• The ability to work with a much larger variety of services. Galaxy 1.0 only stored artifacts. Galaxy 1.5 allows you to just store metadata about services in the form of registry “entries.” For instance, you can define an entry for a RESTful service then start attaching metadata such as the service owner, URL, links to documentation, lifecycle information, etc.
• Event API – easily develop extensions which listener for lifecycle transitions or new entries/artifacts
• Scripting shell – great for scripting things such as notificactions or replication

And there’s a lot more as well…you can find out the rest of the details in the What’s new in Galaxy 1.5 document.

We’ll be releasing the 1.5 final version (along with the 1.5 enterprise version) in a couple weeks once we’re able to put it through some more QA/testing and once we get all the documentation updated for 1.5.

As always, we’re eager to here feedback so kick the tires and please let us know what you think.

Yesterday, I touched down in Oslo to attend the JavaZone conference (can you believe Sun still lets them use that name?). This is always a great conference as the Norwegians are always doing cool stuff with Java.

This morning I attended Jim Webber’s talk on Guerilla SOA. He was hilarious as always. He (correctly) derided the idea of a giant “Bus” which sits on your network which does intelligent routing for every application then talked about how a more webby approach can solve a lot of the problems. Unfortunately, I there was no mention of enterprise man boobs (a.k.a. “moobs”) in this talk.

I also tried to attend Arjen’s talk on RESTful services in Spring, but it was full and they wouldn’t let me in.

I’ll be giving a talk tomorrow entitled “SOA Governance – 5 common mistakes and how to avoid them.” Hope you can make it if you’re here.

Mark Little recently posted a summary of a debate on whether or not governenace for the web is different then typical SOA governance.

Stefan Tilkov left a comment* that I think is worth highlighting as I heartily agree with it:

To me, governance is about defining and enforcing rules in the interest of a better overall result. This seems a good idea regardless of any particular architectural or technical approach. What’s somewhat contrarian to WOA (hate that term) goals is the idea of having a central authority. But why is this a necessity? I think governance can be de-centralized, and if it is, it works perfectly with a more web-like style.

Also, the InfoQ article quotes Dan Foody:

How can a provider make it easier to on-board customers and keep them happy (all while changing the service frequently)?
How can a consumer establish and build trust in their service provider (that’s trust as in “trust but verify”)?

I think governance tools need to help answer these questions for all types of services/architectures. Whether things are centralized or decentralized, and wether you’re using a SOAP/WSDL approach, a RESTful approach or are doing other types of services.

With our next version of Galaxy, we’ve been at least adressing this first point. We’re trying to ensure that you can use Galaxy as a registry for any type of service – RESTful services, JMS, TCP, SOAP, etc. Inside Galaxy you can create pure metadata entries which represent services. I could write about it, but I suggest you just click on the screen shot.

You’ll see that you’re able to start capturing a whole host of metadata around services such as

• Business information
• Contacts
• Documentation
• URLs/entry points to access the service

If all this information is in a central, searchable place, it becomes much easier to on board customers.

This leaves the issue of trust. I’m a little unsure as to how a tool can build trust. Galaxy allows you to attach lifecycle information so you can see if a service is in production, is deprecated, etc. However you still need to trust the person who created this information.

Galaxy 1.1 should be out sometime this month. In the mean time, please try one of our snapshot builds – we’re pretty much just cleaning things up at this point.

* InfoQ: please give us comments feeds!!! I love your site, but I never remember to come back for discussions.

WSO2 has felt the need over the past few months to make many false claims about Mule’s performance. For instance:

Mule CE 2.0.1 couldn’t handle the cases where we used a concurrency level of 80; while other ESB’s scaled to support to over 2500 concurrent connections. This was after tuning the maximum active thread count to 100 from its default value, which limited Mule to a very few concurrent connections.

I ran their benchmarks. Sure enough, with their configuration, Mule performance was crappy. There were a couple fatal flaws with their benchmark though:

• It used the stock HTTP transport instead of the Jetty transport which is NIO based. Swtiching fixed concurrency issues.
• It turns out there is a bug/feature with Linux pre 2.6.17 that requires you to turn on tcpNoDelay switch with Mule. This affects performance on Linux based systems significantly for many of the tests—up to 200-300% differences were noted. In essence this controls whether or not the tcp message is sent before the buffer is full. Because the number of concurrent users is low in a lot of tests, the system is operating far under 100% load. This means it takes longer for a buffer to fill up and hence longer for the message to send.

Results

We released a paper with pretty graphs. Here are the relavent conclusions:

With a proper configuration Mule was able to process many more transactions per second than WSO2′s ESB in all three of their scenarios at almost every load level. Mule was on average 28% faster for [proxying HTTP endpoints], 77% faster for [XPath based] content based routing, and 286% faster for [XSLT] transformations. The only tests where Mule did not exceed WSO2 were with small XML messages and very light loads. Here the difference was less than 2% and is not statistically significant.

My hunch is that we can also beat the proprietary ESB in many scenarios as well if the system is properly tuned.

Content Based Routing

While I was looking into this I decided we might as well not just beat them, but significantly widen the lead. You may remember a while back I announced SXC, an XML parser compiler. It has a streaming XPath engine. Mule now supports it and it out performs anything else out there by a wide margin. For small messages (0.5K), we were up to 25% faster. For medium sized messages (5K) we were 200-300% faster with large loads. Take out all the HTTP overhead and I think we can safely assume that SXC is about 10x faster than anything else.

On the other hand we have AXIOM + Jaxen. Jaxen is fundamentally a DOM based. Even though AXIOM is a “streaming DOM”, Jaxen is very often going to trigger a full load of the document into memory. Not to mention SXC actually compiles the whole XPath expression down to a series of Java functions/statements to the most optimized form possible.

(Surely someone will object and say that SXC doesn’t support all of XPath. Yes, that is true. However, in that case you can just use the Jaxen routing filter and then performance is equal. But rarely do you route on such complicated expressions. If SXC is missing something, file a JIRA and I’ll try to add it.)

In addition to all this goodness, I get the added satisfaction of knowing that to equal our performance WSO2 will have to adopt code that I’ve written (SXC) or write something like it from scratch, which I would consider quite funny.

Bill de hÓra writes:

At a different layer but with passing similarities – can I suggest that OSGi and Maven port their jar/bundle metadata to Atom?

You can already do the OSGi part with Galaxy.

Step 1: Start Galaxy:

java -jar galaxy-web-standalone-1.0.jar

Step 2: Add a new OSGi bundle to an AtomPub collection [1]:

curl -v –data-binary @slf4j-api-1.5.0.jar -u admin:admin -H “Content-Type: application/octet-stream” -H “X-Artifact-Version: 1.5.1″ -H “Slug: slf4j-api-1.5.0.jar” http://localhost:8080/api/registry/Default%20Workspace

Step 3: Do either:

curl -v -u admin:admin http://localhost:8080/api/registry?\
q=select artifact where jar.osgi.Import-Package.packages = 'org.slf4j.impl'

Or:

curl -v -u admin:admin \

http://localhost:8080/api/registry/Default%20Workspace/slf4j-api-1.5.0.jar;atom

For the former command you receive (snipped for brevity):

<?xml version='1.0' encoding='UTF-8'?>
<feed xmlns="http://www.w3.org/2005/Atom">
...
<entry>
 <link href="/api/registry/test/slf4j-api-1.5.0.jar;atom" rel="edit" />
 <id>urn:galaxy:artifact:70c344d2-5bba-4686-bdd4-6a83432ef8fd</id>
 <title type="text">slf4j-api-1.5.0.jar</title>
 <updated>2008-07-07T22:44:36.959Z</updated>
 <author>
 <name>Galaxy</name>
 </author>
 <summary type="xhtml"></summary>
 <metadata xmlns="http://galaxy.mule.org/1.0">
  ....
  <property name="jar.osgi.Import-Package.packages" locked="true" visible="true">
   <value>org.slf4j.impl</value>
  </property>
  <property name="jar.osgi.Export-Package.packages" locked="true" visible="true">
   <value>org.slf4j</value>
   <value>org.slf4j.spi</value>
   <value>org.slf4j.helpers</value>
  </property>
 </metadata>
 ...
 <content type="application/java-archive" src="/api/registry/test/slf4j-api-1.5.0.jar" />
 <link href="/api/registry/test/slf4j-api-1.5.0.jar" rel="edit-media" />
 </entry>
</feed>

For the second command you receive just the individual entry.

I will also note that this is completely extensible. The OSGi headers are indexed via a simple Groovy script that comes bundled with Galaxy. You can add your own groovy scripts as well.

1. It took me the better part of an hour to figure out that one must use –data-binary not -d with curl. Argh…

Seeming that I’m on an OSGi roll here, I want to elaborate on why Maven and OSGi don’t mix well.

One of the things I like about Maven is that it has created a repository layout to store JARs/applications/etc for folks to use. There is a central repository, but you can also create your own repositories easily as well. Whether its log4j or Mule or Jetty or Spring, I can find it in the Maven repository.

The thing I want to discuss in this entry is how to create an application, with Maven, when dependencies are a mix of OSGi enabled bundles and just regular ol’ JARs. (Things such as how do I create an OSGi bundle from Maven or how to do I create a POM around an existing OSGi bundle are a separate discussion…)

Flawed Approaches

Spring Source went down the route of giving everything a new name in the Maven repository, or more specifically a new groupId which starts with com.springsource. This is a fundamentally flawed model as it breaks everything that Maven does. Now, I have some projects which depend on com.springsource.org.hibernate:hibernate and some which depend on org.hibernate:hibernate. It requires a complete clean room implementation of the Maven repository which is a time consuming and losing battle. Every time somebody adds a project, it has to be added to the SpringSource repository.

Also, what happens as projects add OSGi headers to their jars? Will everything in the repository be instantly switched back to the original groupId? If so its probably going to break all sorts of stuff in consumers’ POMs like dependency exclusions or usage of the assembly plugin. I could go on about how bad this approach is long term…

Next option: What if I used BND to modify JARs inside a repository and adding headers. Now you avoid all the issues around conflicting artifact/group ids, but you open up another can of worms. If the dependency is in your local repository, then you have the issue of having a different JAR from the regular Maven repository. Checksums and signatures: broken.

If you modify a JAR that sts in a public repository, then we have issues. First, who gives you the right to add the OSGi headers to somebody else’s JAR? Its their official distribution not yours. Then, how are you going to resign it? What are you going to do about the fact that some repositories might have the original JAR which you didn’t modify? You could end up with the non-OSGi enabled version of the JAR in your repository quite easily.

You could try to convince everybody to add OSGi headers upstream. I’m not waiting around for that to happen though.

A Way Forward?

I see a couple approaches which can be utilized.

First, you can use the mixed approach that David Savage advocates in a comment on my blog:

If you are building some new OSGi application that depends on classes from a plain old jar and those classes never need to be passed between OSGi code in different bundles. Then in this case you can embed the jar containing the classes directly into your bundle and add it to the local classpath of your bundle via the Bundle-Classpath header.

This gives you a deployable unit of code that automatically contains all code needed to function. Compared to a standard J2SE approach where you need to suppliment the local classpath with all the dependencies of a given jar.

The biggest problem with this is that you end up not gaining any of the strongly touted benefits of OSGi. If I’m going to do this, why even bother with OSGi?

The second approach that I can see is to use different version names. Instead of 2.0, you can use 2.0-osgi. This allows you to get around some of the disadvantages that I outlined above as its clear that this isn’t the original version of the JAR, its a new one. It also gets around the SpringSource problems as switching to an OSGi versio of the jar is as simple as adding the dependency (with the updated version) to your <dependencyManagement> section of the POM. Even better, as projects decide to add OSGi headers, you can remove these dependency listings from your <dependencyManagement> section, allowing your POM to shrink and become more manageable as time goes on.

(Along these lines, I wouldn’t mind seeing a Maven repository which could dynamically create bundles via a wiki like fashion. I as a user could submit an updated set of MANIFEST headers. Then the proxy would dynamically download the original dependency, add the headers, change the version and provide it to me as a user.)

Another approach that I wouldn’t mind seeing (and this is more long term) is to equip OSGi containers to be able to retrieve OSGi metadata to supplement JARs from a server. On a final note, I’l leave you with a tool that popped up on Steve’s blog to be able to do this – PAX URL wrap:

Pax URL Wrap is an OSGi URL handler that can process your legacy jar at runtime and transform it into an OSGi bundle.

Cool stuff. I have yet to try it out though.

I feel the need to rant about OSGi for a bit. Too many people are hailing it as the solution to everything and anything, and it can be a serious PITA sometimes.

Today’s topic (one of many that are on my mind) is bundles.

Bundles are a PITA.

The recommended OSGi practice is that you turn every JAR into a “bundle” so that it is OSGi compliant. I have to echo what Steve Loughran said this week about this: tool choices should not be transitive [1]. I shouldn’t have to force every upstream project to get on the OSGi bandwagon. A JAR should not have to be OSGi compliant, a JAR is something that somebody gives to me and I should not have to changed. Ideally it is signed so I wouldn’t even want to change it.

Requiring the OSGi headers in a bundle manifest also breaks the way we work with many tools [2]. Take Maven or Ivy for instance. They download JARs for my application from a common public repository. What happens when a JAR is not OSGi enabled? You end up having to do crazy things like having your own version of a Maven repository.

So why should I have to modify a JAR to be able to deploy things in an OSGi container? Or why should a softare provider have to supply this information upstream? It seems to me that a better way to go about this would be to have an OSGi supplemental metadata repository. Given JAR X we could download the missing OSGi information. An OSGi container could just assemble these two things on the fly.

Am I missing something here? Why would people develop such an invasive model?

1. While I did not investigate in depth, I do have to slightly disagree with the stance taken on his blog. ODE should make it easier for Maven users and distribute the transitive dependency information. But, that doesn’t mean Maven has to be the build system for ODE. Just means they have to distribute POMs.

2. Don’t give me crap about how OSGi has been around longer or how we should just ensure that everyone has OSGi headers. We don’t live in an OSGi only world.