Home › Issues

Negative Results

Wieringa et al.'s "How to Write and Read a Scientific Evaluation Paper" provides a nice and crisp definition of evaluation papers:

Scientific evaluation papers evaluate existing problem situations or validate or refute proposed solutions by means of scientific research, ranging from formal, mathematical analysis to empirical research.

Negative results papers are in the second category: they refute proposed solutions.

Prior work on negative results

  • Feynman 1974 Cargo Cult Science
    • "If you’ve made up your mind to test a theory, or you want to explain some idea, you should always decide to publish it whichever way it comes out. If we only publish results of a certain kind, we can make the argument look good. We must publish both kinds of result. For example—let’s take advertising again—suppose some particular cigarette has some particular property, like low nicotine. It’s published widely by the company that this means it is good for you—they don’t say, for instance, that the tars are a different proportion, or that something else is the matter with the cigarette. In other words, publication probability depends upon the answer. That should not be done."
  • Basili 1996 The role of experimentation in software engineering: past, current, and future
  • Prechelt 1997 Why We Need an Explicit Forum for Negative Results
    • "If a failure is not published at all, obviously nobody else can learn from it"
    • "[...] failures are published in a clouded, hidden form: Either the problem is redefined appropriately as to fit the solution obtained, or the 'solution' is evaluated using only selected examples where it works well. Sometimes the solution is hardly evaluated at all. In all of these cases, the reasons for the failure are not analyzed and our understanding is improved less than it could have been."
    • "One reason why we find such insufficient evaluations is that they would often produce negative results which would be hard to publish."
    • "As a consequence of both lost or spurious insights, other researchers might pursue investigations that are ineffective or at least inefficient and that they would not have pursued, had the respective negative results been published with an appropriate analysis."
    • "Even if a meant-to-be problem-solving contribution fails and thus represents no direct engineering progress, it can be a useful research contribution: Quite often an analysis of the reasons why a particular approach to a problem failed could contribute to understanding, thus promoting further engineering advances and avoiding unfruitful research efforts."
    • "The Forum for Negative Results (FNR) is a permanent special section of the Journal of Universal Computer Science (J.UCS) and exclusively publishes negative results"
  • Tichy 1998 Should Computer Scientists Experiment More?
    • "Experimentation can accelerate progress by quickly eliminating fruitless approaches, erroneous assumptions, and fads."
  • Kaiser 2010 (transcript of a 1986 speech by Hamming) Richard Hamming - You and Your Research
    • "For example, many scientists when they found they couldn't do a problem finally began to study why not. They then turned it around the other way and said, ``But of course, this is what it is'' and got an important result."
    • "Darwin writes in his autobiography that he found it necessary to write down every piece of evidence which appeared to contradict his beliefs because otherwise they would disappear from his mind. When you find apparent flaws you've got to be sensitive and keep track of those things, and keep an eye out for how they can be explained or how the theory can be changed to fit them. Those are often the great contributions."

Check out all papers in the bibliography classified under "negative results".