Irreproducible Workloads
The workloads used to evaluate an innovation should include at least some workloads that others have access to; this will enable others to reproduce the results. We are careful not to say: "one should only pick workloads that others can access" because sometimes valuable insight can be gained from commercially pervasive, but proprietary, workloads that cannot be disclosed.
Example: You claim that an upgrade of the operating system will reduce the median trading latency by up to 10%. Your evaluation used a workload consisting of real‐time market data, which you fed concurrently to both systems. However, you neither captured nor characterized that workload in any way, making it impossible for someone else to examine a similar workload.
Real-World Examples
...
