Producing wrong data without doing anything obviously wrong!

TitleProducing wrong data without doing anything obviously wrong!
Publication TypeConference Paper
Year of Publication2009
AuthorsMytkowicz, T, Diwan A, Hauswirth M, Sweeney PF
Conference NameASPLOS '09: Proceeding of the 14th international conference on Architectural support for programming languages and operating systems
Conference LocationNew York, NY, USA
ISBN Number978-1-60558-406-5
KeywordsExperimentation, Measurement, Performance

This paper presents a surprising result: changing a seemingly innocuous aspect of an experimental setup can cause a systems researcher to draw wrong conclusions from an experiment. What appears to be an innocuous aspect in the experimental setup may in fact introduce a significant bias in an evaluation. This phenomenon is called measurement bias in the natural and social sciences.

Our results demonstrate that measurement bias is significant and commonplace in computer system evaluation. By significant we mean that measurement bias can lead to a performance analysis that either over-states an effect or even yields an incorrect conclusion. By commonplace we mean that measurement bias occurs in all architectures that we tried (Pentium 4, Core 2, and m5 O3CPU), both compilers that we tried (gcc and Intel's C compiler), and most of the SPEC CPU2006 C programs. Thus, we cannot ignore measurement bias. Nevertheless, in a literature survey of 133 recent papers from ASPLOS, PACT, PLDI, and CGO, we determined that none of the papers with experimental results adequately consider measurement bias.

Inspired by similar problems and their solutions in other sciences, we describe and demonstrate two methods, one for detecting (causal analysis) and one for avoiding (setup randomization) measurement bias.

Refereed DesignationRefereed