Artifact Evaluation Artifact - AEC Invitation Letter - First Wave
As a result of this email, all 16 invitees accepted. The latest response came in April 22.
Date: April 16, 2013
To: (16 nominees)
Subject: Invitation to OOPSLA’13 AEC
Dear ...,
It is our pleasure as co-chairs to invite you to serve on the OOPSLA’13 Artifact Evaluation Committee (AEC), which we are creating [1]. We are inviting you because of your background and because you have been nominated for this role by a senior member of our community. The purpose of artifact evaluation is to slowly move our discipline towards repeatability and reproducibility of scientific results [2]. We say more about how it works below.
Committee work will take place in the window from June 1 to July 29, 2013, with the work trailing off after July 20. We expect that each AEC member will be responsible for reviewing about three artifacts. The workload depends on how many artifacts are submitted, and of what nature. We will work hard to keep the work manageable and will consider enlarging the committee if it appears necessary. We expect most committee work to be carried out via correspondence. We will not be having a face-to-face meeting and hope to avoid having a centralized voice/video meeting, although during the discussion period we encourage members to use voice and/or video to discuss their positions.
We would greatly appreciate it if you could respond to us fairly quickly and let us know whether you accept, decline, or need more time to think about it.
However, if you are planning to accept this invitation to serve, please read the rest of this message carefully. We would like to explain in some detail what the OOPSLA AEC is and what the duties and expectations of its members are. This is particularly important since this is the first time an AEC has been used at OOPSLA.
The idea of the AEC was developed by Shriram Krishnamurthi and Carlo Ghezzi for ESEC/FSE'11 [3], and it was recently adopted by ECOOP'13. We have adapted their idea to OOPSLA at the instigation of the OOPSLA steering committee. The AEC has been designed to be as lightweight and unthreatening as possible, while still making an essential step in the right direction. Basically, we are going to let authors of *accepted* papers submit artifacts (code, data, proofs,...) to the AEC, a committee whose role is to evaluate the artifacts. We will report the result of the that evaluation effort at the conference. Very importantly: submission of artifacts is voluntary, it does not affect acceptance of the paper, and does not require sharing the artifact broadly (details of the rules are here [1]).
[1] http://splashcon.org/2013/cfp/due-june-01-2013/665-oopsla-artifacts
[2] http://www.cs.purdue.edu/homes/jv/pubs/r3.pdf
[3] http://www.cs.brown.edu/~sk/Memos/Conference-Artifact-Evaluation/
For each submitted artifact, the committee will decide whether the corresponding OOPSLA paper will receive the AEC's seal of approval. To receive the AEC seal, artifacts should be consistent with the paper, as complete as possible, well documented, and easy to reuse, facilitating further research. Artifacts may in principle take many forms, including source code, empirical data with analysis, proofs, etc. We will assign artifacts to AEC members according to the reviewer’s expertise.
This is a new process, so the mechanics will not be familiar to you. Below we outline key elements of AEC members’ roles.
(1) AEC members are expected to observe strict confidentiality, just as is expected of a PC member. The deliberations of the AEC are strictly confidential. The submissions under review are also confidential. Of course an AEC member may consult with a colleague when conducting a review, but: (a) they must inform the colleague of the confidentiality of the material, and (b) must have full intellectual ownership of the outcome of any such discussion, so as to be able to represent the review well to the AEC.
(2) After the artifact submission deadline, we will ask AEC members to bid for the artifacts they would like to evaluate most. We will then consider their bids when assigning artifact reviews to AEC members, and we will ensure that each artifact will receive at least three reviews.
(3) We anticipate committee members spending in the order of one full day on the evaluation of each of their assigned artifacts (we expect there to be about three). This time includes reading the submitted OOPSLA paper (this is essential to be able to assess whether the artifact is consistent with the paper).
(4) We expect AEC reviews to be in the order of one typewritten page (much like a paper review). The reviews have to assess the artifact along four dimensions: consistency with the paper, completeness, documentation, and ease of reuse.
(5) Each review will grade an artifact on a three-point scale: exceeded expectations, met expectations, did not meet expectations. After all reviews are in, the AEC will try to reach a consensus on the overall grade of each artifact. In case a consensus cannot be reached, the AEC chairs will break the tie. Authors of artifacts that "exceeded expectations" or "met expectations" will be eligible to place our seal of approval on their OOPSLA paper. Moreover, authors with artifacts that "exceeded expectations" will also be recognized at the conference (possibly with an "distinguished artifact award").
Of course if you have any questions about the process that are not answered here or on the web page [1], please don’t hesitate to contact us. We will provide more details about the process as the artifact submission deadline comes closer.
Thanks very much for reading through all of these details. Even if you are unsure whether you can serve, we would appreciate a yes/no/maybe response as quickly as possible so we can complete the composition of the committee. Feel free to let us know if you have any questions and we will respond quickly.
We very much hope you will be willing to serve on the committee, and we look forward to working with you.
Best regards,
Matthias Hauswirth and Steve Blackburn
OOPSLA’13 AEC Co-Chairs
