Yuchen Fama explains that agentic workloadsAgentic AI inference is model serving for AI agents whose long, multi-turn and tool-using sessions create variable context, latency and caching demands. break the steady-state assumptions behind many inference benchmarks. Real sessions can span thousands of turns, reuse system prompts at cache-hit rates above 90 percent and exhibit very large input-to-output ratios, making cache lifetime, interactive latency and traffic variance central scheduling concerns.
Yuchen Fama demonstrates KV-cache-aware routingKey-value cache-aware routing sends a model request to a worker that can reuse relevant cached attention state while also considering current load. that scores available inference pods by load and prefix locality. Reusing a cached system prompt sends a follow-up turn to the same pod and cuts the demonstrated response startup time from about three seconds to about one, while offload tiers and session-aware eviction policies help useful context survive longer.
Ashish Kamra explains prefill-decode disaggregationPrefill-decode disaggregation runs prompt processing and token generation on separate worker pools so each stage can be scaled and optimized independently. as a way to separate compute-heavy prompt processing from memory-bandwidth-sensitive token generation. In the presented results, disaggregation reduces P99 inter-token latencyInter-token latency is the time between consecutive output tokens while an AI model is generating a response. from roughly 900 milliseconds to roughly 100 milliseconds, but it is not a universal improvement: it is most useful for long-context workloads in the middle concurrency range and requires a fast network fabric for cache transfer.
A closing case study combines independent prefill scaling, decode workers and model-parallel strategies for GLM 5.2 on H200 systems. The early configuration delivers four times faster time to first tokenTime to first token is the delay between sending a model request and receiving the first generated output token. and handles 60 percent more requests, illustrating why agentic inference needs coordinated routing, cache management, autoscaling and hardware-aware tuning.
Watch on YouTube
