Experimental Evaluation: The Canon
The "Experimental Evaluation in Software and Systems Canon" is a collection of readings on experimental evaluation and "good science" that have influenced us and that have the potential to influence the researchers coming after us.
Contents of the Canon
The Canon is available in the form of the bibliography of this site. Its focus is on papers discussing experimental evaluation and scientific methodology, not on papers containing experimental evaluations.
Besides the bibliography (which is also accessible by keyword), we provide quotes from the readings in the canon, organized by the various issues, problems, and pitfalls in experimental evaluation. For example, check out the page on reproducibility, with quotes from Feynman's Cargo Cult Science paper.
Expanding the Canon
We gladly accept proposals for readings to add to the canon.
A Canon?
The idea of a canon originated at Evaluate 2010. We wanted to collect resources that we and others can use for educating ourselves and our students about proper experimental evaluation. The use of the term "canon" is inspired by Michael Eisenberg's SIGCSE'03 paper "Creating a computer science canon: a course of "classic" readings in computer science".
