An artificial intelligence inference benchmark can measure throughput, latency, energy use, memory consumption, cost, or combinations of those quantities. It should specify the model, batch behavior, precision, request pattern, kernels, and target quality.
A complete suite is more reliable than one favorable test because serving tradeoffs differ across workloads. Preliminary results should remain provisional until preferred methods and independent checks are finished.
It should report the model, hardware, software, precision, batch size, request pattern, quality target, latency, throughput, power boundary, and test procedure.
Why can inference benchmark results vary?
Kernels, compiler versions, batching, caching, quantization, traffic shape, power limits, and measurement choices can substantially change the result.
