Agentic Inference for AI Agents - Byung-Gon Chun

AI Engineer14m 59s
Byung-Gon Chun in a blue sweater gestures beside the blue and white headline FASTER AI AGENTS on black.
1 VIEW
0 comments · 0 votesOpen discussionClose discussion
Sign in to join the discussion

    Byung-Gon (Gon) Chun frames an agent's task, rather than an individual model request, as the useful performance unit. Planning, tool calls and observations form a loopAn agent loop repeatedly observes a task's state, chooses an action, checks the result and decides whether to continue or stop., sometimes with parallel subagentsA multi-agent system contains multiple AI agents that interact, coordinate, divide work, or influence one another while pursuing tasks., so users experience the time until the whole task finishes.

    Long-running agent steps often reuse a growing context. Prefix cachingPrompt caching reuses compatible computation from previously processed prompt content to reduce repeated model-input work. can avoid reprocessing the shared portion, while key-value cache management uses GPU memory, host memory and diskA key-value cache stores intermediate attention information so an AI language model can reuse earlier computation during generation. to keep more context available across steps and replicas.

    Cache-aware routing can send a request toward a reusable prefix while balancing load. Agent-aware scheduling can use knowledge of the larger workflowInference is the use of a trained model to produce an output from supplied input. for prefill, preemption and cache eviction decisions, with the aim of reducing end-to-end task latency.

    Original YouTube thumbnailWatch on YouTube

    Video topic coverage

    • Agents & workflows100
    • Coding & tools25
    • Models & learning50
    • Infrastructure & compute100
    • Research & evaluation75
    • Safety & security0
    • Business & productivity50
    • Creative & media0
    • Robotics & perception0
    • Society & policy0

    Share this video

    Byung-Gon Chun in a blue sweater gestures beside the blue and white headline FASTER AI AGENTS on black. With WWW.ARTIFICIAL-INTELLIGENCE.VIDEO, 19 September 2026 and duration 14m 59s.

    Byung-Gon Chun explains how shared-context caching, memory management, routing and scheduling can reduce the time an AI agent takes to finish a task.