Experimental Evaluation of Software and Systems in Computer Science

Welcome to the collaboratory on Experimental Evaluation of Software and Systems in Computer Science.

This web site serves as a resource and a hub for everybody interested in understanding and improving the state of practice in experimental evaluation.

Currently, we provide the Canon (a bibliography on experimental evaluation), and a list of venues focusing on experimental evaluation.

History

The Evaluate Collaboratory originated at Evaluate 2010, the first workshop on Experimental Evaluation of Software and Systems in Computer Science, at SPLASH/OOPSLA 2010. The main outcome of that workshop was our open Letter to PC Chairs, which continues to gain support from a growing group of strong researchers.

Many of the Collaboratory members met again at Evaluate 2011 (co-located with PLDI 2011, which this year was part of FCRC 2011) on Sunday, June 5, in San Jose. The goal of this second instance of the Evaluate workshop was the development of a coherent set of experimental evaluation anti-patterns, with the aim of allowing the community to learn from our past mistakes. The resulting paper is available as a technical report.

Evaluate 2012 (co-located with PLDI, ECOOP, ISMM, and LCTES) was held on June 15, in Beijing, China. The theme of this workshop was "Improving Experimental Evaluation through Education". A key outcome of that workshop was our feedback on the Computer Science Curricula 2013 Strawman Draft, with the goal of strengthening the aspect of experimental evaluation of computer systems.