Artifact Evaluation Artifact - Artifact Review Form

The artifact review form consists of two major parts: the fields to be completed by the early kicking-the-tires deadline, and the fields to be completed by the later review submission deadline.

Kicking-the-tires fields

AEC members are supposed to work through the artifact's "Getting Started" guide very early after submission of the artifact. We call this phase of reviewing the "kicking-the-tires" phase. The goal is to ensure that artifacts actually are reviewable, and to be able to fix accidental issues with submitted artifacts (e.g., truncated or wrongly configured VM image files, missing passwords, ...).

Complete review fields (scores)

During the main artifact reviewing phase, the AEC members have to come up with an overall score for the artifact. That score reflects the decision for whether or not to accept the artifact. An artifact is accepted if it met or exceeded the expectations of the AEC. If it is accepted, the authors can place the AEC badge on the PDF of their research paper. If it exceeded the expectations, the authors will be especially acknowledged.

The overall score is just a high-level view of the artifact's quality. The AEC decomposes this quality into four aspects (which are also represented in the AEC badge): consistent with the paper, complete, well documented, easy to reuse.

Complete review fields (text)

The probably most important part of the review form are the following text fields.

We expected the AEC members to read the paper (otherwise they wouldn't be able to assess the consistency of the artifact with the paper). Thus, we asked for a summary of the paper in field 8.

In field 9 we asked for a non-judgemental summary of the artifact.

Field 10 asked for a short high-level summary of the assessment (possibly in the form of a couple of bullet items), while Field 11 asked for a more extensive explanation/substantiation (expanding on those bullet items and discussing other, more secondary, issues). The main purpose of field 10 is to support the discussion and decision making phases, to simplify and speed up the comparison of different reviews and artifacts.

Field 12 asked for suggestions to the authors on how to improve their artifact.

Field 13 was for passing on information within the AEC, without sharing it with the authors.

Complete review fields (co-reviewers)

While we did not intend to use co-reviewing, this field showed up in the review form.