Byung-Gon (Gon) Chun frames an agent's task, rather than an individual model request, as the useful performance unit. Planning, tool calls and observations form a loopAn agent loop repeatedly observes a task's state, chooses an action, checks the result and decides whether to continue or stop., sometimes with parallel subagentsA multi-agent system contains multiple AI agents that interact, coordinate, divide work, or influence one another while pursuing tasks., so users experience the time until the whole task finishes.
Long-running agent steps often reuse a growing context. Prefix cachingPrompt caching reuses compatible computation from previously processed prompt content to reduce repeated model-input work. can avoid reprocessing the shared portion, while key-value cache management uses GPU memory, host memory and diskA key-value cache stores intermediate attention information so an AI language model can reuse earlier computation during generation. to keep more context available across steps and replicas.
Cache-aware routing can send a request toward a reusable prefix while balancing load. Agent-aware scheduling can use knowledge of the larger workflowInference is the use of a trained model to produce an output from supplied input. for prefill, preemption and cache eviction decisions, with the aim of reducing end-to-end task latency.
Watch on YouTube




