Artifact Evaluation Artifact - Notification
Date: July 31, 2013
To: (all artifact authors, sent via Cyberchair by Richard van de Stadt)
We first present the general text shared between acceptance and rejection notifications, and then we show the snippets specific to acceptance and rejection.
Text Shared Between Acceptance and Rejection Notification
Dear *AUTHORS*,
*ACCEPT/REJECT_INTRO*
*PAPER_TITLE*
(ID: *PAPER_ID*)
*ACCEPT/REJECT_TEXT*
Below you find a description of our evaluation process, followed by the detailed
reviews, and a request to please provide us with feedback on our process.
*ACCEPT_BADGE_INFO/REJECT_AVAILABLE*
Each artifact was evaluated by at least three AEC members.
In the first phase, each reviewer "kicked the tires" of the artifact to catch
cases where e.g., an artifact's archive file was corrupted or a VM image could
not be loaded. To do that, the reviewers tried to work through the "Getting
Started Guide" you provided as part of your artifact. The results of that phase
are included in the reviews as "Quality of Getting Started Guide" and,
optionally, "Questions for the authors, based on working through the Getting
Started Guide". If there were fundamental problems during this first phase, the
AEC chairs notified you and forwarded your responses to the AEC. In case there
were no fundamental problems during this first phase, you received a
corresponding notification from the AEC chairs.
In the second phase, the reviewers evaluated the artifact, starting with the
"Step by Step Instructions" you provided as part of your artifact. Each reviewer
read your paper, wrote a short summary of what they perceived as the key
contributions of the paper ("Summary and contributions of the paper"), and wrote
a short summary of the artifact ("Artifact summary"). In their artifact
evaluation, the reviewers focused on four key questions: (1) "Is the artifact
consistent with the paper?" (2) "Is the artifact complete?" (3) "Is the artifact
well documented?" and (4) "Is the artifact easy to reuse?" They provided an
overall grade ("What is the artifact grade?" -- "did not meet expectations",
"met expectations", or "exceeded expectations") a brief "Overall assessment",
and a more detailed "Rationale for assessment", followed by "Suggestions" for
improving the artifact.
In the third phase, the reviewers who reviewed your artifact discussed their
reviews and tried to converge on common position. They summarized their common
position in a summary statement that the discussion lead reviewer included in
their review (you can find it clearly marked in one of the reviews).
In the fourth phase, all AEC members not conflicted with your artifact could see
the reviews, summaries, and discussions, and they could contribute to the
discussion. This allowed a calibration of the reviews across different
artifacts.
In the fifth phase, the AEC chairs announced the tentative grades for each
artifact (based on the summaries and results of AEC-wide discussion), and the
AEC members approved or disapproved of the tentative grades, ultimately reaching
a final grade for each artifact.
We would like to thank you for submitting your artifact, and we very much hope
that you will continue to submit your artifacts in the future.
PLEASE TELL US WHAT YOU THINK: This was the first time OOPSLA ran an artifact
evaluation, and we are interested in hearing your opinion on this new process.
We would very much appreciate if you could provide your feedback through our
questionnaire at
*GOOGLE_DOC_FORM*
-- it would be great if every author could submit a separate form. This will
help shape and improve future artifact evaluation processes.
Best regards,
Matthias Hauswirth and Steve Blackburn, AEC Co-Chairs
*REVIEWS*
Acceptance
ACCEPT_INTRO
Congratulations! We are pleased to inform you that the artifact of your paper
ACCEPT_TEXT
was accepted by the OOPSLA artifact evaluation committee.
ACCEPT_BADGE_INFO
You are encouraged to include the AEC.s stamp of approval on your camera ready
paper (approved by the OOPSLA PC chair). The following zip file includes the
pdf for the badge and instructions for adding it to your paper:
http://users.cecs.anu.edu.au/~steveb/aec-badge.zip
Rejection
REJECT_INTRO
We are sorry to inform you that the artifact of your paper
REJECT_TEXT
could not be accepted by the OOPSLA artifact evaluation committee (AEC).
REJECT_AVAILABLE
We regret that we were unable to accept your artifact. We hope that you will
make the artifact publicly available nevertheless, and that the reviews,
enclosed below, will be helpful in revising the artifact for that purpose.
