Irreproducible Data Analysis
For a data analysis to be reproducible, the experiment must carefully document the analysis it uses and identify and report any biases in the analysis. If an experimenter does not document their analysis and explicitly state their biases, other researchers have little hope of reproducing the analyses and arriving at the same conclusion.
Example: You claim that the increase in response time is not statistically significant. Your evaluation did not report the confidence level, the corresponding confidence interval, and the number of samples taken.
Real-World Examples
...
