New Mule Performance Benchmark: Yup, we come out on top.
July 21st, 2008WSO2 has felt the need over the past few months to make many false claims about Mule’s performance. For instance:
Mule CE 2.0.1 couldn’t handle the cases where we used a concurrency level of 80; while other ESB’s scaled to support to over 2500 concurrent connections. This was after tuning the maximum active thread count to 100 from its default value, which limited Mule to a very few concurrent connections.
I ran their benchmarks. Sure enough, with their configuration, Mule performance was crappy. There were a couple fatal flaws with their benchmark though:
- It used the stock HTTP transport instead of the Jetty transport which is NIO based. Swtiching fixed concurrency issues.
- It turns out there is a bug/feature with Linux pre 2.6.17 that requires you to turn on tcpNoDelay switch with Mule. This affects performance on Linux based systems significantly for many of the tests—up to 200-300% differences were noted. In essence this controls whether or not the tcp message is sent before the buffer is full. Because the number of concurrent users is low in a lot of tests, the system is operating far under 100% load. This means it takes longer for a buffer to fill up and hence longer for the message to send.
Results
We released a paper with pretty graphs. Here are the relavent conclusions:
With a proper configuration Mule was able to process many more transactions per second than WSO2’s ESB in all three of their scenarios at almost every load level. Mule was on average 28% faster for [proxying HTTP endpoints], 77% faster for [XPath based] content based routing, and 286% faster for [XSLT] transformations. The only tests where Mule did not exceed WSO2 were with small XML messages and very light loads. Here the difference was less than 2% and is not statistically significant.
My hunch is that we can also beat the proprietary ESB in many scenarios as well if the system is properly tuned.
Content Based Routing
While I was looking into this I decided we might as well not just beat them, but significantly widen the lead. You may remember a while back I announced SXC, an XML parser compiler. It has a streaming XPath engine. Mule now supports it and it out performs anything else out there by a wide margin. For small messages (0.5K), we were up to 25% faster. For medium sized messages (5K) we were 200-300% faster with large loads. Take out all the HTTP overhead and I think we can safely assume that SXC is about 10x faster than anything else.
On the other hand we have AXIOM + Jaxen. Jaxen is fundamentally a DOM based. Even though AXIOM is a “streaming DOM”, Jaxen is very often going to trigger a full load of the document into memory. Not to mention SXC actually compiles the whole XPath expression down to a series of Java functions/statements to the most optimized form possible.
(Surely someone will object and say that SXC doesn’t support all of XPath. Yes, that is true. However, in that case you can just use the Jaxen routing filter and then performance is equal. But rarely do you route on such complicated expressions. If SXC is missing something, file a JIRA and I’ll try to add it.)
In addition to all this goodness, I get the added satisfaction of knowing that to equal our performance WSO2 will have to adopt code that I’ve written (SXC) or write something like it from scratch, which I would consider quite funny.
July 21st, 2008 at 7:32 pm
False claims? The data we published was the best available data, given that (i) we repeated asked for help tuning Mule, and (ii) there is no documentation on how to do it.
You’ve done a good job tuning Mule and its nice to see you improving the product. I hope the docs catch up now too.
Paul
July 22nd, 2008 at 8:22 pm
Where can I find the tuned configurations of Mule? and have you used tcpNoDelay switch with WSO2 ESB as well?
Any way, good job. Nice to see other products trying to beat us, which will obviously bring the advantage to the open source community !!
Thanks,
Ruwan
July 22nd, 2008 at 8:24 pm
I got those configurations from here [http://www.mulesource.com/solutions/mule-benchmark.tar.gz]
Thanks,
Ruwan
July 22nd, 2008 at 10:20 pm
Ruwan: Well if you looked at the httpcore source code you would see that by default tcp no delay is set to true. Whereas for HttpClient it defaults to false.
Also “nice to see other products trying to beat us” ? You are the ones trying to beat us and who seem to be doing incessant benchmarking.
The whole benchmark is pretty meaningless IMO anyway. Its just measuring the speed of the libraries and the overhead of the ESB. And ESB overhead is going to be stupendously low compared to something like XSLT.
July 24th, 2008 at 9:04 am
Agree.
ESB performance benchmark are irrelevant. How can you measure performance of the wrapper around some real library?
Real performance benchmark would be JMS broker versus JMS broker with non-JDBC based transaction journal (a.k.a. custom journal implementation).
This is just silly.
July 24th, 2008 at 10:12 am
The only problem with Mule is its licensing…
Our organization prefer flexible licensing for adoption of software from open source. So we would go for WSO2 over Mule, since WSO2 performance is acceptable. I am sure WSO2 would make improvements.
August 11th, 2008 at 12:06 pm
Does ESB performance matter?
August 11th, 2008 at 12:08 pm
Pay attention to VTD-XML as it will blow away the competition (5X~15x), nothing (AXIOM, or whatever) will not be even close.
http://vtd-xml.sf.net
August 11th, 2008 at 12:33 pm
Jimmy:
First, AXIOM isn’t an XML parser, so I don’t know how you can compare it to VTD-XML.
Second, these tests are really comparing XPath and XSLT performance. XML parsing is really just a fraction of what is going on here. There are no XSLT or XPath engines on to of VTD-XML.
Finally, if you want to compare it to anything compare it to Aalto. I guarantee you its much closer than 5x, if not equal. Not to mention it does well formedness checks and implements the standard StAX API.
August 11th, 2008 at 12:36 pm
I take back my second comment - VTD does have XPath support.
I looked at your benchmarks on your site. I would encourage you to retry your benchmarks with current versions of Aalto/Woodstox + SXC. It may not be quite as fast given that you skip so much checking inside VTD, but it’ll be close I bet.
August 11th, 2008 at 4:56 pm
Not true, VTD-xml is a conforming XML parser, it doesn’t support DTD, but skips nothing else…
August 11th, 2008 at 4:57 pm
Also Axiom is the data model like a DOM tree, so it is a kind of XML parsing model…
VTD-XML’s comparison is fair in that VTD is comparable to VTD-XML