Baseten's Philip Kiely frames the talk as an update to his book Inference Engineering. He contrasts local inference, where fitting a model in limited memory comes first, with data-center serving, where throughputAI inference throughput measures how much model-serving work a system completes per unit of time., latency and routing dominate. Across both, he sees a tighter loop between training and inference optimizationInference is the use of a trained model to produce an output from supplied input..
For quantization, Kiely says four-bit TurboQuant can reduce KV-cache memory and bandwidth, but its added decode computation cut throughput too much for the Baseten workloads he describes. He says the team instead favors weight quantizationModel quantization represents AI model values with fewer bits to reduce storage, memory use and often inference cost. and system-level KV routing, sharing and offload. He then sketches learned KV compaction as a potential middle ground between a full cache and a much shorter summarized memory.
Kiely spends the largest section on speculative decodingSpeculative decoding accelerates text generation by proposing several tokens cheaply and checking them together with the target model.. He says Baseten uses diffusion-based drafting in production and cites more than a threefold gain for one specific model and hardware comparison against an earlier drafting approach; that is a reported workload-specific result, not a universal speed guarantee. A newer hybrid research method has no production results yet. He notes that continuous retraining may improve draft acceptance but requires permission to use live dataData privacy governs how personal, confidential, or sensitive information is collected, used, shared, retained, and protected in AI systems., storage and compute. Closing hardware forecasts are explicitly his opinion. Book and booth promotion are omitted.
Watch on YouTube
