Yuchen Fama explains that agentic workloads break the steady-state assumptions behind many inference benchmarks. Real sessions can span thousands of turns, reuse system prompts at cache-hit rates above 90 percent and exhibit very large input-to-output ratios, making cache lifetime, interactive latency and traffic variance central scheduling concerns.
Yuchen Fama demonstrates KV-cache-aware routing that scores available inference pods by load and prefix locality. Reusing a cached system prompt sends a follow-up turn to the same pod and cuts the demonstrated response startup time from about three seconds to about one, while offload tiers and session-aware eviction policies help useful context survive longer.
Ashish Kamra explains prefill-decode disaggregation as a way to separate compute-heavy prompt processing from memory-bandwidth-sensitive token generation. In the presented results, disaggregation reduces P99 inter-token latency from roughly 900 milliseconds to roughly 100 milliseconds, but it is not a universal improvement: it is most useful for long-context workloads in the middle concurrency range and requires a fast network fabric for cache transfer.
A closing case study combines independent prefill scaling, decode workers and model-parallel strategies for GLM 5.2 on H200 systems. The early configuration delivers four times faster time to first token and handles 60 percent more requests, illustrating why agentic inference needs coordinated routing, cache management, autoscaling and hardware-aware tuning.
Watch on YouTube
