Irreproducible Metrics

A metric's name may seem to unambiguously define the meaning of that metric, however, often the actual implementation of the metric leaves a lot of flexibility. A study needs to precisely define how the metric is measured, otherwise future studies cannot produce comparable measurements.

Example: You claim that your compiler optimization reduces the number of executed instructions. Your evaluation failed to report whether you included the number of speculatively executed instructions in your count.

Real-World Examples