Inappropriate Workloads
A workload is inappropriate when it is flawed or does not reflect the workload that is implicit in the claim.
Example: You claim that your algorithmic trading system completes 99.9% of transactions within 3 milliseconds. Your evaluation used a workload that represents quiescent markets, which does not capture high‐frequency trading activity.
Real-World Examples
- Present performance figures for an inner kernel, and then represent these figures as the performance of the entire application. It is quite difficult to obtain high performance on a complete large-scale scientific application, timed from beginning of execution through completion. There is often a great deal of data movement and initialization that depresses overall performance rates. A good solution to this dilemma is to present results for an inner kernel of an application, which can be souped up with artificial tricks. Then imply in your presentation that these rates are equivalent to the overall performance of the entire application.
-- Point #2 in David H. Bailey. Twelve ways to fool the masses when giving performance results on parallel computers in Supercomputing Review, August 1991
...
