The workshop frames LLM inference as a recurring operating cost that grows with every user, session and generated token. It identifies three initial bottlenecks: KV-cache memory grows with context and concurrent users, time to first token rises with longer prompts, and a naive sequential implementation delivers poor throughput under concurrent demand.
The presenters trace these bottlenecks through the transformer pipeline. Attention creates key and value vectors for every token, while prefill is compute-bound and decode is largely memory-bound. The distinction explains why prompt processing determines time to first token, why sequential generation determines inter-token latency, and why GPU memory bandwidth can cap generation speed.
Capacity planning becomes a tradeoff among quality, latency and throughput. Premium chat may prioritize long context and responsiveness, while asynchronous agent workloads may favor quality and concurrency. The workshop shows how GPU choice, model precision and expected batch size should be evaluated together, since a more expensive accelerator can still produce a lower cost per million tokens when it serves enough work efficiently.
Model-side techniques include post-training quantization and several attention designs, including multi-head, multi-query, grouped-query, latent and sparse attention. Serving-side techniques include paged attention, continuous batching, prefix caching and KV-cache quantization. The presenters connect each method to a specific source of memory fragmentation, repeated computation or idle GPU capacity rather than treating optimization as a single benchmark score.
Benchmarks in the workshop compare a raw Hugging Face baseline with vLLM and SGLang on an H100. The presenters report a major throughput improvement from vLLM's default serving optimizations and a setup-specific advantage for SGLang on repeated agentic branching, while finding similar results for ordinary API workloads. Their decision rule is to start from the workload, model fit and required service levels, then choose quantization, caching and an inference engine that address the measured bottleneck.
Watch on YouTube
