Artificial intelligence inference throughput can be expressed as tokens, requests, sequences, or accepted tasks completed per second. It depends on hardware, memory, batching, model architecture, kernels, serving software, workload shape, and the latency targets the service must meet.
High throughput lowers unit cost when demand keeps resources utilized, but it can conflict with interactive latency. Operators balance batch size, queueing, cache use, and capacity so aggregate efficiency does not make individual users wait too long.
