Artifact Evaluation Artifact - Timeline

What we would do differently:
  • Allocate a little bit more time (2 weeks?) between stage-2 paper notification and artifact submission
  • More clearly define what "kicking the tires" means (and what happens to the result)
  • Maybe schedule an explicit deadline for getting author responses on "kicking the tires" feedback?

At OOPSLA'13, artifact evaluation is interleaved with the research paper evaluation.

Friday, March 22 Research paper abstracts due
Thursday, March 28 Research papers due
Friday, May 17 PC meeting finished
AEC chairs grow AEC if needed (based of number and kind of expected artifacts, estimated from the research papers accepted in the first stage).
Saturday, May 25 Research papers first-phase notification
Authors package and submit artifacts
Saturday, June 1 - extended to June 9 Artifacts submitted
AEC members bid for artifacts, identifying their preferences and expertise
Monday, June 3 - extended to June 12 Bids due
AEC members ‘kick the tires’ of each of their assigned submissions by working through the “Getting Started” guide provided by the authors. Reviewers evaluate this phase with a boolean ‘satisfactory’ / ‘unsatisfactory’ and provide short feedback and questions.
Monday, June 17 - extended to June 23 Kicking-the-tires due
AEC members review each of their assigned submissions, reading the associated paper. They submit their reviews via CyberChair. The reviews will later (at the notification deadline) be sent to the authors, together with a one-paragraph statement produced during the subsequent discussion phase, and the acceptance decision reached during the later virtual AEC meeting.
Monday, July 15 Reviews due
AEC members engage in online discussion (via the discussion feature on the artifact page on CyberChair) with other members who reviewed the same artifacts, calibrating their scores and identifying and resolving points of disagreement where possible. The chairs will identify ‘discussion leaders’ for each submission who will prepare (and submit via the discussion feature) a one-paragraph statement on the submission for consumption by the rest of the AEC (and the artifact's authors).
Monday, July 22 Deadline of one-paragraph statement
AEC members will hold a virtual asynchronous AEC "meeting" where final decisions will be made on each submission. All AEC members will read the summary paragraphs for each submission. Note that the decisions are up to the reviewers and the entire AEC.
Monday, July 29 Notification (for artifacts & research papers)
Authors of accepted artifacts integrate AEC badge into the PDF of their research paper
Monday, August 5 Research paper camera-ready copy due