Experimental Evaluation of Software and Systems in Computer Science

Dear Program Chairs and Program Committee Members:

We are writing to you out of concern for the health of empirical science within Computer Science. We believe that as those who evaluate papers and form programs, you have a crucial role to play in addressing this problem.

Classical empirical sciences (such as biology and medicine) have a deeply ingrained culture of the scientific method and rigorous evaluation. In these sciences, scientists observe phenomena to gain insight and understanding, and scientists reproduce prior work to validate results.

Observation studies allow the research community to generate new ideas: Classical empirical sciences rely on observation studies to increase understanding through observation and measurement. This shared understanding is the foundation for new ideas because a credible solution is predicated on a correct understanding of the problem. A lack of rigorous observation studies derails scientific progress, because scientists pursue solutions based on inadequate observation of the problem space.

Reproduction studies allow the research community to validate ideas: Classical empirical sciences depend on reproduction studies to validate and refute prior findings. This validation is key to the integrity of new ideas because empirical studies are inherently subject to (unintended) bias. A lack of rigorous reproduction studies derails scientific progress, because scientists base their work on invalid findings.

Despite the importance of observation and reproduction, it is rare in our community for either of these to be the primary focus of a paper. Instead most papers are primarily “idea” papers, where evaluation becomes secondary to the presentation of new algorithms, architectures or methods. An absence of observation and reproduction papers inherently requires that “idea” papers alone carry the burden of unbiased, definitive evaluation. This situation stands in contrast to other empirical sciences and suggests a cultural bias which may be unhelpful to the long term health of our discipline. We argue that active encouragement of observation and reproduction papers is necessary to combat this cultural bias.

Over the years, we have heard two main reasons why observation and reproduction papers are perceived as less important than idea papers and should not be accepted into our top conferences and journals; we argue that these reasons are not justified.

Reason 1: "Observing computer systems and evaluating systems from prior work is an uncreative task and thus a waste of time for a researcher." We disagree. An insightful evaluation often requires much creativity which requires us to (i) design an experiment that identifies and avoids sources of bias; (ii) collect data without perturbing the system; (iii) confirm that the data is actually telling us what we think it is telling us (e.g., "is the speedup really due to our optimization or due to some unintended interaction with the hardware"?). All of these steps require creativity and a deep understanding of not just the underlying system but also knowledge of statistical and visualization methods.

Reason 2: "Observing computer systems and evaluating systems from prior work leads to incremental research where we keep refining prior work rather than jump into new territory." We disagree. A careful evaluation not only leads to a deeper understanding of the original ideas, but also often to new research opportunities. When we conduct a careful evaluation of an existing approach we not only discover when the approach works well but also when it fails. These failure conditions often lead to interesting research problems and novel algorithms.

We believe it is time for Computer Science to become a better empirical science by promoting both observation and reproduction studies.

We hope you agree with us. If you disagree, please let us know. If you agree, we request that you actively encourage observation and reproduction papers in your "call for papers".

Thank you!

Signatories

If you would like to sign this letter, please fill in this form.

(by role*, in alphabetical order)

*) If your name does not show up under the correct role, or if your name, homepage, or affiliation needs to be updated, please edit this page (if you are an Evaluate Collaboratory member), or email Matthias.Hauswirth@usi.ch.