A microbenchmark case study and lessons learned

TitleA microbenchmark case study and lessons learned
Publication TypeConference Paper
Year of Publication2011
AuthorsGil, J Y, Lenz K, Shimron Y
Conference NameProceedings of the compilation of the co-located workshops on DSM'11, TMC'11, AGERE!'11, AOOPES'11, NEAT'11, &\#38; VMIL'11
Conference LocationNew York, NY, USA
ISBN Number978-1-4503-1183-0
Keywordsbenchmark, measurements, steady-state

The extra abstraction layer posed by the virtual machine, the JIT compilation cycles and the asynchronous garbage collection are the main reasons that make the benchmarking of Java code a delicate task. The primary weapon in battling these is replication: "billions and billions of runs", is phrase sometimes used by practitioners. This paper describes a case study, which consumed hundreds of hours of CPU time, and tries to characterize the inconsistencies in the results we encountered.