Artifact Evaluation Artifact - Announcements - Kicking-the-Tires Guidance
Date: June 20, 2013
To: (all 24 AEC members)
Subject: Re: OOPSLA Artifact Reviewing Started
Dear AEC Member,
The "kicking-the-tires" parts of your reviews are arriving in growing numbers. Thanks a lot for all your effort!
After looking at the incoming reviews and receiving some good questions from you, we'd like to bring up a few key points to all of you:
Note that the Artifact Evaluation process is new to the authors as well as you (and us!). To get the perspective of the authors, you may want to quickly reread what we wrote to the authors in the call for artifacts (http://splashcon.org/2013/cfp/665), especially the part on the "Getting Started Guide" (the last paragraph in the "Packaging Guidelines").
If authors requested in their "Getting Started Guide" that you rerun their entire experiment (especially if that takes days), this is not really in the spirit of the "Getting Started Guide". Feel free to ignore such steps.
In case a submitted artifact is unsatisfactory to you (say, it's a pain to download a large VM image for such a small piece of software), but according to our call it would be ok, please harass us, but don't punish the authors.
In general, please try to focus your questions to the authors (field "# 2 - Questions for authors, based on working through the Getting Started Guide" in the review form) on cases where you get stuck, i.e., where you cannot evaluate the artifact without further help. Move other comments, questions, or feedback to other fields in your review.
Again, thanks a lot for your hard work, and feel free to contact us with questions.
Best regards,
Steve and Matthias
