Artifact Evaluation

Recently, major conferences have started also evaluating the artifacts underlying the papers by having Artifact Evaluation Committees (AECs) in addition to their PCs.

Criteria

The AEC reviews artifacts with respect to the following evaluation criteria (if applicable):

  • Consistent with the paper. Does the artifact substantiate and help to reproduce the claims in the paper?
  • Complete. What is the fraction of the results that can be reproduced?
  • Well documented. Does the artifact describe and demonstrate how to apply the presented method to a new input?
  • Easy to reuse. How easy is it to reuse the provided artifact?

Conferences using Artifact Evaluation

Here is a list of conferences that use artifact evaluation:

POPL'17 - Jean Yang and Steve Chong
POPL'16 - Steve Chong and Arjun Guha
ECOOP'16 - Tijs van der Storm and Matthew Flatt
OOPSLA'16 - Michael Hind and Michael Bond
CGO'16 - Grigori Fursin and Bruce Childers
VISSOFT'16 - Alexandre Bergel and Santiago Vidal
PLDI'16 - John Regehr and Zachary Tatlock
CGO'15 - Grigory Fursin and Bruce Childers
PPoPP'15 - Grigory Fursin and Bruce R. Childers
VISSOFT'15 - Alexandre Bergel and Anne Etien
ISSTA'15 - Milos Gligoric and Sai Zhang
OOPSLA'15 - Robby Findler and Michael Hind
PLDI'15 - John Regehr and Eric Eide
POPL'15 - Arjun Guha and Jan Vitek
CAV'15 - Arie Gurfinkel, Temesghen Kahsai, Michael Tautschnig
ECOOP'15 - Matthew Flatt and Camil Demetrescu
FSE'15 - Giuliano Casale and Martin Nordio - Replication Packages Evaluation Committee, similar to a normal AEC
ISSTA'14 - Milos Gligoric and Falk Howar
OOPSLA'14 - Matthias Hauswirth and Robby Findler
PLDI'14 - Eric Eide, Shriram Krishnamurthi, and Jan Vitek
ECOOP'14 - Camil Demetrescu and Erik Ernst
FSE'14 - Antonio Carzaniga and Prem Devanbu
SAS'13 - Francesco Logozzo and Manuel Fahndrich - PC Chairs, PC performing artifact evaluation
OOPSLA'13 - Steve Blackburn and Matthias Hauswirth - see the corresponding Artifact Evaluation Artifact
ECOOP'13 - Jan Vitek, Erik Ernst, and Shriram Krishnamurthi
FSE'13 - Alexandre Bergel and Lorenzo Bettini
FSE'11 - Carlo Ghezzi and Shriram Krishnamurthi

Resources

Experienced AEC members and award-winning artifact submitters Dan Barowy, Charlie Curtsinger, Emma Tosch, and John Vilk, wrote up this HOWTO for AEC Submitters.

Steve Blackburn and Matthias Hauswirth, who ran the first OOPSLA AEC, documented their implementation in their Artifact Evaluation Artifact in the hope that this will serve future AEC co-chairs.

Authoritative Description of Artifact Evaluation Process

Shriram Krishnamurthi maintains the authoritative "Artifact Evaluation for Software Conferences" web site at artifact-eval.org.