Replicability in not Reproducibility: Nor is it Good Science

TitleReplicability in not Reproducibility: Nor is it Good Science
Publication TypeConference Paper
Year of Publication2009
AuthorsDrummond, C
Conference NameThe 4th workshop on Evaluation Methods for Machine Learning
Date Published06/2009
Conference LocationMontreal, Canada

At various machine learning conferences, at various times, there have been discussions arising from the inability to replicate the experimental result published in a paper. There seems to be a wide spread view that we need to do something to address this problem, as it is essential to the advancement of our field. The most compelling argument would seem to be that reproducibility of experimental results is the hallmark of science. Therefore, given that most of us regard machine learning as a scientific discipline, being able to replicate experiments is paramount. I want to challenge this view by separating the notion of reproducibility, a generally desirable property, from replicability, its poor cousin. I claim there are important differences between the two. Reproducibility requires changes; replicability avoids them. Although reproducibility is desirable, I contend that the impoverished version, replicability, is one not worth having.