Dodgy Benchmarks Indeed
February 8th, 2007I’ve been meaning to post a follow up to the WSO2 benchmarks for some time. I was hoping to have my own ready, but since InfoQ is carrying them today I would like to respond and with an important point (this is slightly modified from my InfoQ comment).
After looking at their benchmarks I think the latest claim about Axis2 being 2.5% faster than XFire with JiBX is not really true. I noticed that they added in a <soap:Header/> to every message. XFire is optimized for the non header case as that is what most clients will send. Sending a Header will trigger the creation of a JDOM Document to hold the headers in. This is why XFire looks slower in their benchmarks for small messages. For at least 90% of the people out there, XFire will in fact be faster. This also applies to many of the benchmarks in their first posting comparing ADB & JAXB as well.
If WSO2 wants to look at scenarios that use headers, like WS-Addressing or user headers, thats great, and Axis2 might win out there, but this benchmark seems to be about the performance of moving data around without the bells and whistles.
Update: I removed my snippy comments about Axis2 as I don’t know that they were called for. Also, I want to elaborate on the case of headers since Paul Downey raised it in the comments. When there are actual headers in the document, and you are executing logic based on these headers, the document creation time will probably not matter as much. Regardless though, this is a difference of a few percent either way. 5% simply doesn’t matter for web services as your service logic will most likely take more time than that. Now if someone can show me something that doubles performance, that would be noteworthy. However that will be quite tricky to achieve because now the limiting factor in web service performance is probably not XFire or Axis2, its going to be your XML parser, HTTP provider, and/or your data-binding implementation.
February 8th, 2007 at 7:15 pm
I guess, you don’t know much about your own code do you? See the profiler stats on the soap:Header processing in XFire
http://ws.zones.apache.org/~dims/xfire-jibx-headers/
thanks,
dims
February 8th, 2007 at 10:15 pm
For me, header building takes 7% of the invocation time for the case of a simple invocation.
Maybe you are testing a large request?
February 9th, 2007 at 2:10 am
get it running, make it right, make it faster. Axis2 and XFire are still at first base in my experience, especially when it comes to databinding.
You are also so wrong about the headers. That might have been true a few years ago, but WS-Addressing has been required by most .NET services for some time and WS-Security is something most people require now. if a service doesn’t need headers, it doesn’t need SOAP.
February 9th, 2007 at 1:25 pm
I agree that a lot of people use WS-Addressing, but the overhead in creating a document becomes less once you actually start doing header processing. My gripe is that the benchmark wasn’t representative of either scenario.
Not that this all matters that much, we’re talking about a couple percent here, and I don’t know of any web service where that will matter that much.
February 9th, 2007 at 4:48 pm
Thanks Dan!
February 12th, 2007 at 4:19 pm
Dan sums it up quite well. However, it’s not only header processing that consumes time but the time spent in the web service operation itself. With many customers we are finding the SOAP stack doesn’t even show up in the performance analysis when the operations being called are doing identity processing, security and (usually the biggest part of any service) numerous database accesses.
Performance is important but only up to the point where it isn’t showing up as a major part of exchanges. These tests, although interesting, do not represent any kind of real use which will include security, addressing and database access.
These are the kinds of benchmarks we used to produce to show off our performance but more and more people are interested in scalability with more realistic loads. This involves real requests, real database accesses, real client behaviour (time between requests, etc.), etc. and the key here is the number of concurrent clients that can be supported while providing acceptable response times. Firing back-to-back requests at services that do nothing except return the data generally leads to maximum throughput being at about “CPUs+1″ which is meaningless (I’m guessing going to a big company and saying you can support 3 clients on a 2 CPU box before performance degrades won’t be too well received
).
November 12th, 2008 at 4:11 pm
mwkubzhf2buflhzc