gettin all zoidal on ya
I’m doing a webinar next week Tuesday (April 22) on Mule and REST. I’ll be giving an overview of the concepts of REST, how things like scalability/evolvability emerge from the constraints of REST, and how you can build RESTful applications with Mule.
If you’re interested in attending, the signup is here. Hope you can make it!
I’m sitting here at MuleCon in San Francisco today. Its pretty cool to see 200+ people here to learn about whats going on with Mule and to connect with many friends from accross the world.
One of the things that we’re launching today is a product I’ve been hacking on for the last couple months – the Mule RESTpack. What is it? Its really two components:
1. Documentation geared at helping Mule developers build RESTful architectures with Mule. (I still recommend that you go find the RESTful web services book!). This is still being expanded – if you see a missing topic, please let me know.
2. A series of connectors to help developers implement RESTful services. Included currently are connectors for Abdera, Jersey, and Restlet.
I think this is quite important offering for us. Many developers are struggling to figure out how to do RESTful integration. It is now significantly easier for people to do so via these connectors. It also says we’re committed to REST and helping people build RESTful services. I believe we’re the first ESB vendor to really do this.
We’ve pushed out the first set of milestones for Mule 2.0.0-RC3. Now that Mule 2.0.0 final is out (more on that later), we’ll be pushing out 1.0 releases of the connectors later this week.
Check it out and let us know your feedback!
This week I put the finishing touches on the first version of a Jersey connector for Mule. It makes it really easy to drop in JAX-RS services into a Mule application.
For the non-Mule users out there – you’re probably asking, why would I need Mule and JAX-RS?
Imagine this scenario: you receive messages over HTTP. You wish to process these messages asynchronously with your backend servers. With Mule its pretty easy to set up an HTTP listener and connect it to a JMS queue. You can then have pull messages off the queue and process them asynchronously within a transaction. JAX-RS makes it very easy to map your Java class to resources with it’s simple annotations.
While you could piece this together all yourself, Mule is like integration glue – it makes connecting all these things easier. </marketing>
Several people objected to the proposal posted on Stefan’s blog from the WSO2 guys. Mark Baker and Bill de hÓra argue vehemently against it saying we should move toward a universal data model such as RDF/OWL. While I agree that the solution proposed is not necessarily full proof or what should ultimately be adopted, its a good that we’re discussing this and we need *something* like it.
This looks like another long, never ending, I-won’t-listen-to-you type of debate, but I’ll chime in a cent or two.
Universal All the Way to the Data Model?
Should we just use a data format like RDF/OWN which can be molded into just about anything? Paul Fremantle does a good job of answering this:
The problem of more general media types is that they just push the problem elsewhere. For example, we could do everything with RDF triples, but then you have just pushed the problem to be one of finding a suitable ontology.
One of the best things about the web is how minimal the number of truly centralized services are – and media types just doesn’t seem – to me – to be in the same class as DNS root servers and IP address allocation.
We need more protocols like AtomPub
AtomPub does a great job in generalizing things so you can achieve useful things without knowing as much about a service. This comes at a functionality tradeoff though. For those who say we need more stuff like AtomPub I would like to see proposals. Either you’re going to make it more generic, and reduce the amount of functionality you can access without knowing anything about the service. Or you’re going to make it more specific, reduce the number of use cases it applies to, and increase the functionality you can access without knowing anything about the service. Either way, not a solution to the problem at hand as you’re always leaving problems out of the solution.
Also, even in AtomPub you need to extend it to support extensions. We have OpenSearch extensions, GData extensions, security extensions, etc etc. When I encounter one of these extensions how do I figure out what it is? Sure its easy if its some universal extension. But try inventing a real (semi-complex) service without your own extensions or your own Microformat. Nearly impossible.
I’m all for AtomPub and more things like it. But people should keep these two things in mind.
Get your head out of the Internet and into the intranet
Media type proliferation is a governance problem. On the Internet, the IANA is the governing body. In an Intranet, …. it depends on your governance model. What’s clear is that having everyone’s IT department register their own vnd media type seems both silly and untenable because those media types will not likely be general. So they’ll have their own corporate & partners registry.
Sure you can say those media types should be general. See the above though.
Cultural Issues
There is this idea that we should stay far far away from anything that even remotely reminds anyone of WSDL (even if its not an IDL) because it can be misused. I can’t believe that no one isn’t throwing up their arms against this idea. I suppose this will make me very non-cool, but just because a tool can be used in a bad way doesn’t means that you should never use it. I can use an ice pick to kill someone, but that doesn’t mean one shouldn’t have ice picks. (If you’re twisted you could even view this versatility as a strength of the ice pick 
)
Update: Just to clarify, I agree with much of what Mark/Bill say. But, to say that the *idea* is completely misplaced seems wrong to me. I still think we need a way to:
a) Discover more about a media type which I create for my intranet/business partners/etc. All I need is a way to associate a media type with a link.
b) Discover more about XML extensions or microformats embedded in things like AtomPub.
A little while back there was a lot of pontificating on about web service description languages in the blogosphere. I avoided weighing in partly because its a complex topic and partly because of lack of time. A recent thread on TSS and a commenter have prompted me to get back to actually writing this post though. So here we go…
RESTful services do not need an interface description language. They have a defined uniform interface. This is fundamental.
There are some things which you do need to know to interact with a service though:
1. Documentation of the entry points into the hypertext engine.
This is to some extent solved. This is what OpenSearch is for.
2. What types of media types the resource supports.
On the retrieval side this can be done with a HEAD request and the Content-Type header. On the POST/PUT side, you can use a HEAD plus the Accept header.
3. What methods are available to use on a resource.
Use OPTIONS.
4. A description of the media type.
That is we need a description of the data, not the interface. This item is where I feel there is a large void.
Lets say I have an XML schema for a purchase order. And then I have my media type application/vnd.acme-purchase-order. I need a way to associate these two.
For those who tell me to look in the documentation: that isn’t a good enough answer. We need a machine to machine way to do this. Also, why are the first 3 items OK to do with machines and not this one? Seems rather silly to me.
I know of two possible options for solving this:
1. Define an HTTP header which points to a document which provides a media type to description mapping. Then when you HEAD a resource, you get a link to the description. Then you get a document which associates a media type with a link to a description of that resource.
2. Shove it in the content type header a la James’ proposal (see the comments). Ideally though I’d like it to mesh better with the accept header.
3. Create a document which creates a link between the media type and it’s description, then point people to it. The document basically just needs to have something like <link rel=”application/vnd.acme-foo” href=”foo.xsd”/>. (For bonus points it could have links to the OpenSearch documents too…)
The larger question
The larger question in my mind is if this is all a little.. obtuse. While its kind of cool architecturally, there is a lot of resource scanning going on here to use your service. There is a lot to be said for having a description of your service all in one place like WADL does. Users will be less confused (have you tried teaching what REST is to anyone lately?). Who wants to go around and do a bunch of HEAD/OPTIONS requests while trying to figure out how to interact with a service? You can document your service in one place (which you need to do anyway). You can think more coherently about your service as its all defined in one place. You can give it to something like Jersey and get some service stubs generated (*gasp* who would want to do something this evil!?). And probably a bunch more stuff I’m not thinking of.
Conclusions
Does REST need an interface description language? No. That’s an oxymoron.
Do RESTful services need a way for you to determine how schemas and media types align? Yes.
Would having one document to describe a set of resources be really handy? Yes.
Is this document an IDL? No.
Should you forget about IDLs then and give resource description languages a second chance because anyone who doesn’t at least admit they are somewhat useful is clearly being one sided about this? YES.
In short, I think that having a resource description language is extremely useful from a practical standpoint and I generally support the effort to come up with one. It just simplifies my life too much for me to not want it.
Update: fixed a grammar problem or two and added a third way to discover media type descriptions.
I feel the need to repost my latest posting from The Server Side here…
James Watson writes:
Personally I would like to see a good open source framework for REST based web services. I’m not sure if it’s the way to go but WSDL 2.0 spec should theoretically support REST. I’m unaware of any tools that help with creating these types of WSDLs.
Don’t use WSDL 2.0 for REST. It’s an ugly way of doing RESTful services IMNSHO. There are plenty of great toolkits out there for helping you build RESTful web services. The ones I have my eye on are:
Please keep in mind that REST and SOAP programming models don’t mix well. There is this complete fallacy going around (I succumbed to it too at one point) that you can use the same service class or description to build a RESTful and a SOAP based service. However, the interactions and mappings should end up being completely different from one to the other! One is message based and one is resource based. You’re going to have to design your service differently as they’re two completely different beasts.
Use the best tool for the job. You’ll save yourself time and a lot of hassle.
Now that Galaxy is announced, I can unleash my fury of blogging again. Or, as we all know will probably happen, I can blog just slightly less sporadically again.
One of the recent posts I’ve wanted to comment on is Atom is the New JCR (prompted by Sam Ruby). Adrian Sutton writes:
When the Java Content Repository (JCR) standard first came out it was supposed to bring in a new era of compatibility between content repositories and put an end to the content silo. There was, and still is, a lot of talk about it and just about everyone added JCR compliance to their marketing materials. Unfortunately, that’s mostly where things stopped – the implementation work that followed was generally done was buggy or incomplete and the only viable JCR implementations that I’ve seen have come out of Day Software, who lead the JCR spec effort.
We use JCR inside Galaxy for all our storage needs at the moment. Using it has been an interesting experience to say the least. One of my favorite “features” of JCR has been the fact that you can not have two seperate threads create a child node with the same name at the same time. Which means in our activity log, we can’t add two nodes called “/activity” to “/activities” at the same time.
The main reason we decided to go with JCR is that it supports both simple database type data and file type data via the same interface transparently. I’m not 100% sure it was the right choice though. I had to write an ORM like framework as the one in Jackrabbit seemed pretty immature and I didn’t have time to delve into the rabbit hole of the jcr-mapping module in Jackrabbit. Mine is limited but as least I could figure out how to use it. We’ll be reevaluating this for the future though now that Jackrabbit 1.4 is nearly out.
Adrian continues:
Then along came Atom which is all about remote access and manipulation of data and missing probably 90% of the functionality that JCR offers. It really isn’t a competitor to JCR at all and yet it’s doing more to break down content silos than JCR ever has. Atom support isn’t just being added to the marketing materials, it’s actually shipping and is usable in a lot of places – IBM’s Lotus Connections has Atom APIs to everything and, as best I can tell, only Atom APIs to it’s repository.
I completely agree that JCR isn’t very worthwhile as something that will break down content silos. It does have value as an API to work with data though. Atom is quite limited in the granularity it can work with data (which coincidentally is one of the reasons Web3S exists as well). And you still need to store your data somewhere.
Atom has not replaced JCR it has supplemented it. We use JCR as a content store for Galaxy. I’ve also written a generic JCR content store for AtomPub inside of Abdera. Sure, the vendor promises are probably wrong about the content-silos, but it sure simplifies some things when writing applications – which is what really counts!
Update: David Nuescheler has written a great follow up to this whole discussion.
Also, Jackrabbit 1.4 is out.
What does one do after going to QCon and Oredev? ApacheCon of course. I barely made it on my plane out of Sweden. I arrived at the airport checkin thinking the arrival time of the flight was the departure time. Oops.
Here’s my presentation that I did on building RESTful HTTP services. Really its just a bunch of practical stuff that you should know when you get in bed with HTTP.
I was jazzed before the talk as I had somehow managed to teach Sam Ruby something about HTTP that he didn’t know. Expect: 100-continue. You can use the Expect header to ensure that the server will be willing to process the request. Sam brought up an example where you might be uploading a large image from a cellphone and you need to be authenticated to do so. You certainly don’t want upload the image and then get an unauthorized response! The solution to this is to have a client send an Expect: 100-continue header. After the client sends the headers it waits for the server to respond with either a “100 Continue” message or some other status like “401 Unauthorized”. Provided that the client gets a 100, it can continue sending the message. If it gets some other message, the connection is immediately closed and then the client can decide what to do next – like authenticate itself.
James Snell: Wrong. All of this is complex. Technology is complex. It does nobody any good to argue that one particular way of building applications is less complicated than any other way of building applications. WS-* is complex. REST is complex. It’s all complex. Complexity is not the issue. The real issues are things like utility, consistency, accessibility, etc.
Not much I can add to that.
Bill (where is that Ó key) de hÓra: “The fundamental problem here is looking for a RESTful way to do RPC/RMI, instead of redesigning the application to expose resources.”
This gets at one of my main problems with getting people to build good RESTful services: it’s a lot of work.
While OO in theory can map pretty well to REST, it takes a lot of metadata and careful thought to turn those objects into resources. Making an RPC application is much easier. This is one of the killer features of SOAP/WSDL. I can take my business service and build a web service out of it with very little effort (I assert that there are non-evil ways to do this, but thats another story). I can then be interoperating with a .NET application in just a minute or two. Or I can take a WSDL, generate a set of objects, and just write some glue code between my internal objects and the web service objects.
Building the RESTful equivalent isn’t nearly as easy IMO. (At least for Java).
I suppose there are two ways to build RESTful services. First, you can turn your interfaces/POJOs into a RESTful service. This requires you to know:
Given a business interface and a set of POJOs, this isn’t quite so easy. Sure, people are working on JSR 311, annotations, etc. But I would like to assert that it will still be harder than building the equivalent RESTful service. At the very least you need to add a bunch more annotations as well as design your URI structure. (JPA annotations may give us some help here in the future – @Id, @OneToMany, etc would give some of the necessary metadata – but not everyone has these annotations on their application. It also seems a little icky to retarget these for RESTful services.).
The other option would be to design your application in a RESTful manner to begin with. I’ll be honest, I don’t have a lot of experience with doing it from the start. Most applications that I encounter have some OO/DAO backend which is exposed through various frontends.
Am I wrong about all this? Should building a RESTful service be easy/easier? (Just for the record – even if it is more work, that doesn’t mean I don’t think there aren’t other more important benefits like scalability)