Comments on: Dodgy Benchmarks Indeed http://netzooid.com/blog/2007/02/08/dodgy-benchmarks-indeed/ gettin all zoidal on ya Sat, 24 Jul 2010 09:48:52 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Pete Hendry http://netzooid.com/blog/2007/02/08/dodgy-benchmarks-indeed/comment-page-1/#comment-1070 Pete Hendry Mon, 12 Feb 2007 23:19:11 +0000 http://netzooid.com/blog/2007/02/08/dodgy-benchmarks-indeed/#comment-1070 Dan sums it up quite well. However, it's not only header processing that consumes time but the time spent in the web service operation itself. With many customers we are finding the SOAP stack doesn't even show up in the performance analysis when the operations being called are doing identity processing, security and (usually the biggest part of any service) numerous database accesses. Performance is important but only up to the point where it isn't showing up as a major part of exchanges. These tests, although interesting, do not represent any kind of real use which will include security, addressing and database access. These are the kinds of benchmarks we used to produce to show off our performance but more and more people are interested in scalability with more realistic loads. This involves real requests, real database accesses, real client behaviour (time between requests, etc.), etc. and the key here is the number of concurrent clients that can be supported while providing acceptable response times. Firing back-to-back requests at services that do nothing except return the data generally leads to maximum throughput being at about "CPUs+1" which is meaningless (I'm guessing going to a big company and saying you can support 3 clients on a 2 CPU box before performance degrades won't be too well received ;) ). Dan sums it up quite well. However, it’s not only header processing that consumes time but the time spent in the web service operation itself. With many customers we are finding the SOAP stack doesn’t even show up in the performance analysis when the operations being called are doing identity processing, security and (usually the biggest part of any service) numerous database accesses.

Performance is important but only up to the point where it isn’t showing up as a major part of exchanges. These tests, although interesting, do not represent any kind of real use which will include security, addressing and database access.

These are the kinds of benchmarks we used to produce to show off our performance but more and more people are interested in scalability with more realistic loads. This involves real requests, real database accesses, real client behaviour (time between requests, etc.), etc. and the key here is the number of concurrent clients that can be supported while providing acceptable response times. Firing back-to-back requests at services that do nothing except return the data generally leads to maximum throughput being at about “CPUs+1″ which is meaningless (I’m guessing going to a big company and saying you can support 3 clients on a 2 CPU box before performance degrades won’t be too well received ;) ).

]]> By: Davanum Srinivas http://netzooid.com/blog/2007/02/08/dodgy-benchmarks-indeed/comment-page-1/#comment-1042 Davanum Srinivas Fri, 09 Feb 2007 23:48:00 +0000 http://netzooid.com/blog/2007/02/08/dodgy-benchmarks-indeed/#comment-1042 Thanks Dan! Thanks Dan!

]]> By: Dan Diephouse http://netzooid.com/blog/2007/02/08/dodgy-benchmarks-indeed/comment-page-1/#comment-1040 Dan Diephouse Fri, 09 Feb 2007 20:25:49 +0000 http://netzooid.com/blog/2007/02/08/dodgy-benchmarks-indeed/#comment-1040 I agree that a lot of people use WS-Addressing, but the overhead in creating a document becomes less once you actually start doing header processing. My gripe is that the benchmark wasn't representative of either scenario. Not that this all matters that much, we're talking about a couple percent here, and I don't know of any web service where that will matter that much. I agree that a lot of people use WS-Addressing, but the overhead in creating a document becomes less once you actually start doing header processing. My gripe is that the benchmark wasn’t representative of either scenario.

Not that this all matters that much, we’re talking about a couple percent here, and I don’t know of any web service where that will matter that much.

]]> By: Paul Downey http://netzooid.com/blog/2007/02/08/dodgy-benchmarks-indeed/comment-page-1/#comment-1037 Paul Downey Fri, 09 Feb 2007 09:10:40 +0000 http://netzooid.com/blog/2007/02/08/dodgy-benchmarks-indeed/#comment-1037 get it running, make it right, make it faster. Axis2 and XFire are still at first base in my experience, especially when it comes to databinding. You are also so wrong about the headers. That might have been true a few years ago, but WS-Addressing has been required by most .NET services for some time and WS-Security is something most people require now. if a service doesn't need headers, it doesn't need SOAP. get it running, make it right, make it faster. Axis2 and XFire are still at first base in my experience, especially when it comes to databinding.

You are also so wrong about the headers. That might have been true a few years ago, but WS-Addressing has been required by most .NET services for some time and WS-Security is something most people require now. if a service doesn’t need headers, it doesn’t need SOAP.

]]> By: Dan Diephouse http://netzooid.com/blog/2007/02/08/dodgy-benchmarks-indeed/comment-page-1/#comment-1036 Dan Diephouse Fri, 09 Feb 2007 05:15:14 +0000 http://netzooid.com/blog/2007/02/08/dodgy-benchmarks-indeed/#comment-1036 For me, header building takes <a href="http://envoisolutions.com/~dan/headerbuilding.png"/ rel="nofollow">7% of the invocation time</a> for the case of a simple invocation. Maybe you are testing a large request? For me, header building takes 7% of the invocation time for the case of a simple invocation.

Maybe you are testing a large request?

]]> By: Davanum Srinivas http://netzooid.com/blog/2007/02/08/dodgy-benchmarks-indeed/comment-page-1/#comment-1035 Davanum Srinivas Fri, 09 Feb 2007 02:15:29 +0000 http://netzooid.com/blog/2007/02/08/dodgy-benchmarks-indeed/#comment-1035 I guess, you don't know much about your own code do you? See the profiler stats on the soap:Header processing in XFire http://ws.zones.apache.org/~dims/xfire-jibx-headers/ thanks, dims I guess, you don’t know much about your own code do you? See the profiler stats on the soap:Header processing in XFire

http://ws.zones.apache.org/~dims/xfire-jibx-headers/

thanks,
dims

]]>