Experimental Evaluation of Software and Systems in Computer Science

Keynote by LLVM Founder & Architect Chris Lattner
CGO 2012 - April 2, 2012

Content of most relevant slides:

Reproducibility/Believability of Results
• Results vary widely with:
– Target Architecture & Source Language
– Compiler Infrastructure
– Quality of Implementation
– Benchmark set

Reproducibility of Results: Wishlist
• Use a specific version of a well known compiler:
– “LLVM 3.0”, “GCC 4.7”, ”GHC 7.4.1” ...
– Avoids measuring artifacts of an immature foundation
• Measure, measure, measure:
– Dynamic performance, static metrics
– Code size, compile time
• Publish implementation and dataset:
– github link in your paper!
– No need for it to be production quality

Empirical Meta-Comparison Studies
• Solutions to important problems have:
– multiple different algorithms
– many implementation refinements
– different tradeoffs (e.g. analysis time vs quality)
• Need more studies to fairly compare these:
– ideally by third parties
– code and dataset made available for scrutiny
– as apples-to-apples as possible
• Obvious “citation bait”!