I doubt Joshua Bloch suffers from performance anxiety but it was slightly ironic that there were some technical problems before he started his talk, titled Performance Anxiety, to an absolutely packed room at Devoxx today. I knew it would be a popular session so I got there early and so managed to find a seat, unlike the others who sat on the steps or crowded the doorway.

His talk was about microbenchmarking – measuring the performance of a small piece of code as opposed to profiling an application.

In the good old days performance could be estimated simply by counting program instructions. Now it is simply impossible to do this because of the ‘abstraction gap’. As code has become more and more complex it has become harder and harder to estimate and measure the performance. Complexity and predictability are inversely related.

Even benchmarking code doesn’t really work with consecutive runs often showing a variance of 20%. The code and systems are so complex nobody really knows where this variance comes from.

The solution is to run a bunch of JVMs in sequence and then statistically analyse the data (mean, median, standard dev, etc).

Basically, benchmarking is really, really hard and most benchmarks are seriously broken. Either the results are unrelated to the test or the error bars exceed the measured values.

Caliper is microbenchmarking framework from Google which can help.