Artifact Evaluation
Recently, major conferences have started also evaluating the artifacts underlying the papers by having Artifact Evaluation Committees (AECs) in addition to their PCs.
Criteria
The AEC reviews artifacts with respect to the following evaluation criteria (if applicable):
- Consistent with the paper. Does the artifact substantiate and help to reproduce the claims in the paper?
- Complete. What is the fraction of the results that can be reproduced?
- Well documented. Does the artifact describe and demonstrate how to apply the presented method to a new input?
- Easy to reuse. How easy is it to reuse the provided artifact?
Conferences using Artifact Evaluation
Here is a list of conferences that use artifact evaluation:
OOPSLA'20 - Colin Gordon and Anders Møller
PLDI'20 - Jonathan Bell and Luís Pina
POPL'20 - Benjamin Delaware and Jeehoon Kang
ICFP'20 - Ben Lippmeier and Brent Yorgey
ECOOP'20 - Lisa Nguyen Quang Do and Manuel Rigger
CGO'20 - Bastian Hagedorn, Michel Steuwer, and Michael Laurenzano
CAV'20 - Xinyu Wang and He Zhu
ISSTA'20 - Rody Kersten and Kasper Luckow
OOPSLA'19 - Colin Gordon and Jan Vitek
ECOOP'19 - Maria Christakis and Manuel Rigger
PLDI'19 - Ravi Chugh and Roopsha Samanta
POPL'19 - Benjamin Delaware and Cătălin Hriţcu
CGO'19 - Bruce Childers, Luis Oliveria, and Michel Steuwer
CAV'19 - Yu Feng and Ruben Martins
VISSOFT'19 - Leonel Merino and Dirk Zeckzer
POPL'18 - Cătălin Hriţcu and Jean Yang
ECOOP'18 - Maria Christakis, Philipp Haller, and Marianna Rapoport
ISSTA'18 - Ben Hermann and Nicolás Rosner
OOPSLA'18 - Sam Tobin-Hochstadt and Jan Vitek
CGO'18 - Michel Steuwer and Grigori Fursin
PPoPP'18 - Grigori Fursin and Malavika Samak
PLDI'18 - Ravi Chugh and Brandon Lucia
POPL'17 - Jean Yang and Steve Chong
ECOOP'17 - Philipp Haller, Michael Pradel, and Tijs van der Storm
OOPSLA'17 - Michael Bond and Sam Tobin-Hochstadt
CGO'17 - Joseph Devietti and Wonsun Ahn
PPoPP'17 - Wonsun Ahn and Joseph Devietti
VISSOFT'17 - Anne Etien and Omar Benomar
PLDI'17 - Murali Krishna Ramanathan and Brandon Lucia
POPL'16 - Steve Chong and Arjun Guha
ECOOP'16 - Tijs van der Storm and Matthew Flatt
OOPSLA'16 - Michael Hind and Michael Bond
CGO'16 - Grigori Fursin and Bruce Childers
PPoPP'16 - Bruce Childers and Grigori Fursin
VISSOFT'16 - Alexandre Bergel and Santiago Vidal
PLDI'16 - John Regehr and Zachary Tatlock
CGO'15 - Grigory Fursin and Bruce Childers
PPoPP'15 - Grigory Fursin and Bruce R. Childers
VISSOFT'15 - Alexandre Bergel and Anne Etien
ISSTA'15 - Milos Gligoric and Sai Zhang
OOPSLA'15 - Robby Findler and Michael Hind
PLDI'15 - John Regehr and Eric Eide
POPL'15 - Arjun Guha and Jan Vitek
CAV'15 - Arie Gurfinkel, Temesghen Kahsai, Michael Tautschnig
ECOOP'15 - Matthew Flatt and Camil Demetrescu
FSE'15 - Giuliano Casale and Martin Nordio - Replication Packages Evaluation Committee, similar to a normal AEC
ISSTA'14 - Milos Gligoric and Falk Howar
OOPSLA'14 - Matthias Hauswirth and Robby Findler
PLDI'14 - Eric Eide, Shriram Krishnamurthi, and Jan Vitek
ECOOP'14 - Camil Demetrescu and Erik Ernst
FSE'14 - Antonio Carzaniga and Prem Devanbu
SAS'13 - Francesco Logozzo and Manuel Fahndrich - PC Chairs, PC performing artifact evaluation
OOPSLA'13 - Steve Blackburn and Matthias Hauswirth - see the corresponding Artifact Evaluation Artifact
ECOOP'13 - Jan Vitek, Erik Ernst, and Shriram Krishnamurthi
FSE'13 - Alexandre Bergel and Lorenzo Bettini
FSE'11 - Carlo Ghezzi and Shriram Krishnamurthi
Resources
Experienced AEC members and award-winning artifact submitters Dan Barowy, Charlie Curtsinger, Emma Tosch, and John Vilk, wrote up this HOWTO for AEC Submitters.
Steve Blackburn and Matthias Hauswirth, who ran the first OOPSLA AEC, documented their implementation in their Artifact Evaluation Artifact in the hope that this will serve future AEC co-chairs.
Authoritative Description of Artifact Evaluation Process
Shriram Krishnamurthi maintains the authoritative "Artifact Evaluation for Software Conferences" web site at artifact-eval.org.
