Sitanshu Gupta lays out CoreWeave's serverless and dedicated inference modelsInference is the use of a trained model to produce an output from supplied input.. Serverless abstracts hardware and charges by token, with a provisioned-throughput optionAI inference throughput measures how much model-serving work a system completes per unit of time. for predictable demand; dedicated customers reserve GPU capacity and control their model deployments.
Gupta compares agentic, chat, live voice and video, and batch workloads. Agentic systems can have long repeated inputs and rapid turns, real-time media is especially sensitive to latency, and batch jobs have flexible deadlines. Those differences shape gateway controls, routing and capacity scheduling.
He explains KV cache locality, storage offload, optional prefill/decode separation, quantizationModel quantization represents AI model values with fewer bits to reduce storage, memory use and often inference cost. and speculative decodingSpeculative decoding accelerates text generation by proposing several tokens cheaply and checking them together with the target model. as performance choices. Reusing cached prefixesPrompt caching reuses compatible computation from previously processed prompt content to reduce repeated model-input work. avoids costly prefills, while reserved GPUs can run batch work during quieter hours. The right configuration depends on workload shape.
Watch on YouTube




