Sitanshu Gupta on Scaling AI Inference Workloads

AI Engineer15m 22s
Sitanshu Gupta gestures beside the blue-and-white ONE INFERENCE STACK headline on a black background.
0 comments · 0 votesOpen discussionClose discussion
Sign in to join the discussion

    Sitanshu Gupta lays out CoreWeave's serverless and dedicated inference modelsInference is the use of a trained model to produce an output from supplied input.. Serverless abstracts hardware and charges by token, with a provisioned-throughput optionAI inference throughput measures how much model-serving work a system completes per unit of time. for predictable demand; dedicated customers reserve GPU capacity and control their model deployments.

    Gupta compares agentic, chat, live voice and video, and batch workloads. Agentic systems can have long repeated inputs and rapid turns, real-time media is especially sensitive to latency, and batch jobs have flexible deadlines. Those differences shape gateway controls, routing and capacity scheduling.

    He explains KV cache locality, storage offload, optional prefill/decode separation, quantizationModel quantization represents AI model values with fewer bits to reduce storage, memory use and often inference cost. and speculative decodingSpeculative decoding accelerates text generation by proposing several tokens cheaply and checking them together with the target model. as performance choices. Reusing cached prefixesPrompt caching reuses compatible computation from previously processed prompt content to reduce repeated model-input work. avoids costly prefills, while reserved GPUs can run batch work during quieter hours. The right configuration depends on workload shape.

    Original YouTube thumbnailWatch on YouTube

    Video topic coverage

    • Agents & workflows50
    • Coding & tools0
    • Models & learning50
    • Infrastructure & compute100
    • Research & evaluation25
    • Safety & security0
    • Business & productivity75
    • Creative & media25
    • Robotics & perception0
    • Society & policy25

    Share this video

    Sitanshu Gupta gestures beside the blue-and-white ONE INFERENCE STACK headline on a black background. With WWW.ARTIFICIAL-INTELLIGENCE.VIDEO, 19 September 2026 and duration 15m 22s.

    Sitanshu Gupta describes how one inference platform can serve serverless, dedicated and batch workloads by routing for KV cache locality and tuning GPU capacity for latency and cost.