Baseten's Philip Kiely frames the talk as an update to his book Inference Engineering. He contrasts local inference, where fitting a model in limited memory comes first, with data-center serving, where throughput, latency and routing dominate. Across both, he sees a tighter loop between training and inference optimization.
For quantization, Kiely says four-bit TurboQuant can reduce KV-cache memory and bandwidth, but its added decode computation cut throughput too much for the Baseten workloads he describes. He says the team instead favors weight quantization and system-level KV routing, sharing and offload. He then sketches learned KV compaction as a potential middle ground between a full cache and a much shorter summarized memory.
Kiely spends the largest section on speculative decoding. He says Baseten uses diffusion-based drafting in production and cites more than a threefold gain for one specific model and hardware comparison against an earlier drafting approach; that is a reported workload-specific result, not a universal speed guarantee. A newer hybrid research method has no production results yet. He notes that continuous retraining may improve draft acceptance but requires permission to use live data, storage and compute. Closing hardware forecasts are explicitly his opinion. Book and booth promotion are omitted.
Watch on YouTube
