Artifact Evaluation Artifact - Timeline
What we would do differently:
- Allocate a little bit more time (2 weeks?) between stage-2 paper notification and artifact submission
- More clearly define what "kicking the tires" means (and what happens to the result)
- Maybe schedule an explicit deadline for getting author responses on "kicking the tires" feedback?
At OOPSLA'13, artifact evaluation is interleaved with the research paper evaluation.
| Friday, March 22 | Research paper abstracts due |
| Thursday, March 28 | Research papers due |
| Friday, May 17 | PC meeting finished |
| AEC chairs grow AEC if needed (based of number and kind of expected artifacts, estimated from the research papers accepted in the first stage). | |
| Saturday, May 25 | Research papers first-phase notification |
| Authors package and submit artifacts | |
| Saturday, June 1 - extended to June 9 | Artifacts submitted |
| AEC members bid for artifacts, identifying their preferences and expertise | |
| Monday, June 3 - extended to June 12 | Bids due |
| AEC members ‘kick the tires’ of each of their assigned submissions by working through the “Getting Started” guide provided by the authors. Reviewers evaluate this phase with a boolean ‘satisfactory’ / ‘unsatisfactory’ and provide short feedback and questions. | |
| Monday, June 17 - extended to June 23 | Kicking-the-tires due |
| AEC members review each of their assigned submissions, reading the associated paper. They submit their reviews via CyberChair. The reviews will later (at the notification deadline) be sent to the authors, together with a one-paragraph statement produced during the subsequent discussion phase, and the acceptance decision reached during the later virtual AEC meeting. | |
| Monday, July 15 | Reviews due |
| AEC members engage in online discussion (via the discussion feature on the artifact page on CyberChair) with other members who reviewed the same artifacts, calibrating their scores and identifying and resolving points of disagreement where possible. The chairs will identify ‘discussion leaders’ for each submission who will prepare (and submit via the discussion feature) a one-paragraph statement on the submission for consumption by the rest of the AEC (and the artifact's authors). | |
| Monday, July 22 | Deadline of one-paragraph statement |
| AEC members will hold a virtual asynchronous AEC "meeting" where final decisions will be made on each submission. All AEC members will read the summary paragraphs for each submission. Note that the decisions are up to the reviewers and the entire AEC. | |
| Monday, July 29 | Notification (for artifacts & research papers) |
| Authors of accepted artifacts integrate AEC badge into the PDF of their research paper | |
| Monday, August 5 | Research paper camera-ready copy due |
