Experimental Evaluation of Software and Systems in Computer Science

A measurement context is inappropriate when it is flawed or does not reflect the measurement context that is implicit in the claim. This may become manifest as an error or as a distraction (a "red herring").

Example: You claim that your smartphone battery lasts 10 hours when running your newspaper application. Your evaluation ran the application while the phone was in airplane mode, which caused your application to use data cached locally on the phone instead of fetching data from a server over the wireless network.

Real-World Examples

• Quote only 32-bit performance results, not 64-bit results. We all know that it is hard to obtain impressive performance using 64-bit floating point arithmetic. Some research systems do not even have 64-bit hardware. Thus always quote 32-bit results, and avoid mentioning this fact if at all possible. [...] 32-bit arithmetic may or may not be appropriate for your application, but the audience doesn't need to be bothered with such details.
  -- Point #1 in David H. Bailey. Twelve ways to fool the masses when giving performance results on parallel computers in Supercomputing Review, August 1991
• Quietly employ assembly code and other low-level language constructs. It is often hard to obtain good performance from straightforward Fortran or C code that employs the usual parallel programming constructs, due to compiler weaknesses on many highly parallel computer systems. Thus you should feel free to employ assembly-coded computation kernels, customized communication routines and other low-level code in your parallel implementation. Don't mention such usage, though, since it might alarm the audience to learn that assembly-level coding is necessary to obtain respectable performance.
  -- Point #3 in David H. Bailey. Twelve ways to fool the masses when giving performance results on parallel computers in Supercomputing Review, August 1991

...