|Title||Replicability in not Reproducibility: Nor is it Good Science|
|Publication Type||Conference Paper|
|Year of Publication||2009|
|Conference Name||The 4th workshop on Evaluation Methods for Machine Learning|
|Conference Location||Montreal, Canada|
At various machine learning conferences, at various times, there have been discussions arising from the inability to replicate the experimental result published in a paper. There seems to be a wide spread view that we need to do something to address this problem, as it is essential to the advancement of our field. The most compelling argument would seem to be that reproducibility of experimental results is the hallmark of science. Therefore, given that most of us regard machine learning as a scientific discipline, being able to replicate experiments is paramount. I want to challenge this view by separating the notion of reproducibility, a generally desirable property, from replicability, its poor cousin. I claim there are important differences between the two. Reproducibility requires changes; replicability avoids them. Although reproducibility is desirable, I contend that the impoverished version, replicability, is one not worth having.