Experimental Evaluation of Software and Systems in Computer Science

Inconsistent data analysis occurs when different data analysis techniques are applied to data pertaining to different objects of an experimental study.

Example: You claim that your new feature does not increase the response time. Your evaluation used the mean to compute the response time of the new system. You compared that mean to the median response time you found in a report about the performance of the prior system.

Real-World Examples

• Publish best vs. worst execution times. There are a number of ways to further boost the performance of your parallel code relative to the conventional code. One way is to make many runs on both systems, and then publish the best time for the parallel system and the worst time for the conventional system.
  -- Point #11 in David H. Bailey. Twelve ways to fool the masses when giving performance results on parallel computers in Supercomputing Review, August 1991

...