Ashok Chandrasekar and Jason Kramberger distinguish quick model-server benchmarksA benchmark is a defined test or collection of tasks used to compare a system's performance under specified conditions. from production-scale inference testsInference is the use of a trained model to produce an output from supplied input., where many replicas, autoscaling and changing workloads make a single throughput numberAI inference throughput measures how much model-serving work a system completes per unit of time. inadequate. They argue that a useful test must reproduce customer traffic, measure latency and identify the load at which the serving system saturates.
Ashok Chandrasekar shows how a client asked to send 200 requests per second delivered only 38 in one setup, while another measurement incurred client-side latency inflation of up to 58 seconds. He also describes how temperature settings, token sampling and truncation can make apparently comparable benchmark runs measure different workloads.
Jason Kramberger presents Inference Perf's multiprocess load generator, explicit configuration and shared workload catalogue as ways to make runs more observable and reproducible. A production-scale example illustrates the comparison of serving optimizations, while the closing guidance emphasizes checking client concurrency, metrics and dataset fidelityPerformance profiling measures where a program spends time or computational resources so optimization can target observed bottlenecks. before attributing results to the model server.
Watch on YouTube




