Home

Evaluate 2012 - Call for Contributions

Third Workshop on Experimental Evaluation of Software and Systems in Computer Science

Theme:
Improving Experimental Evaluation through Education

Location

Evaluate 2012 is co-located with PLDI 2012 in Beijing, China.

Important Dates

  • Early submission deadline: Friday, April 6, 2012
  • Early submission notification deadline: Thursday, April 19, 2012 (before the PLDI early registration deadline)
  • Late submission deadline: Tuesday, May 8, 2012
  • Workshop date: Friday, June 15, 2012

Join us on LinkedIn or Facebook.

Context

Our effort originated at Evaluate 2010, the first workshop on Experimental Evaluation of Software and Systems in Computer Science, at SPLASH/OOPSLA 2010. The main outcome of that workshop was our open Letter to PC Chairs, which continues to gain support from a growing group of strong researchers.

Many of the Collaboratory members met again at Evaluate 2011 (co-located with PLDI 2011, which that year was part of FCRC 2011) on Sunday, June 5, in San Jose. The goal of this second instance of the Evaluate workshop was the development of a coherent set of experimental evaluation anti-patterns, with the aim of allowing the community to learn from our past mistakes. The resulting paper is available as Evaluate Collaboratory Technical Report #1 and under review for publication.

Call for Contributions

We are organizing this workshop out of concern for the health of empirical science within the software and systems area of computer science.

Classic empirical sciences (such as biology) have a deeply ingrained culture of the scientific method and rigorous evaluation. On the other hand, computer science struggles with the role of evaluation. This ambivalence or confusion is most likely rooted in the reality that our discipline encompasses both formal and empirical science. The former is based on formal systems, and has no need for empirical observation.

Evaluation is fundamental to empirical science. Weak evaluation retards science: it limits our creativity by failing to reveal opportunities, and it exposes us to derailment by incorrect findings due to invalid results.

The goal of this workshop is to raise the quality of empirical evaluation in the area of computer systems and software through education. We will create a set of effective teaching resources for strengthening the education on experimental evaluation in existing courses of the CS curriculum. The workshop will bring together expert researchers in our field, collect their best ideas and experiences for how to educate students about evaluation, and structure and organize the resulting teaching material into a coherent set of educational resources.

Workshop Format

The workshop will not be a mini conference, but a true "work" shop with problem solving and brainstorming sessions. We solicit submissions consisting of brief statements. The workshop organizers will evaluate the submissions. Like we have done at Evaluate 2011, we plan to ask each participant to present a 1-slide 90-second summary of their statement, and we will use the remainder of the day for structured discussions and collaboration sessions with the goal of organizing the contributions into a coherent whole.

Publication/Dissemination of Contributions

We do not intend to publish workshop proceedings. The goal of our workshop is to put together a coherent set of effective teaching materials on experimental evaluation, and to make that material publicly available on the Evaluate Collaboratory. Moreover, to inform the community about our resources, we will discuss that material in a submission to a venue such as the SIGCSE conference or the Communications of the ACM.

We will make the submitted statements available on the Evaluate Collaboratory for all participants before the workshop. We will ask the participants to read all statements before the workshop. During the workshop we will collaborate on reorganizing and integrating ideas and material. We will continue that collaboration after the workshop to make the resources publicly available and to write and submit the common paper.

Submission Information

We solicit submissions consisting of statements addressing one or more of the following three questions. We grouped these questions into two perspectives:

Consumer perspective:

(1) You are a PhD advisor and get a new PhD student. What would you like that PhD student to know about experimental evaluation?
(2) You are an employer and hire a new graduate (BS, MS, or PhD in CS). What would you like that graduate to know about experimental evaluation?

Producer perspective:

(3) You are a teacher and are using creative means for teaching students about experimental evaluation. Could you share your ideas?

Please submit your statement on easychair in the form of an abstract-only submission (just paste the text into the abstract field).

Organizers

Past Workshops

Evaluate 2012 is the third workshop of the Evaluate workshop series. The first Evaluate workshop, Evaluate 2010, was held at SPLASH/OOPSLA 2010 in Reno/Tahoe, Nevada, USA, and in 2011 we held Evaluate 2011 at FCRC/PLDI in San Jose, California.