Cerebras' Daniel Kim and Gimlet Labs cofounder Michelle Nguyen describe inference as a balance between compute, memory and communication. Kim explains tensor parallelism, continuous batching, paged key-value caches, prefix reuse and chunked prefill. These mechanisms address different sources of wasted work and contention; adding hardware does not eliminate coordination overhead.
Nguyen moves from individual-worker optimization to distributing work across systems. Separate prefill and decode pools can scale according to workload shape, and speculative decoding uses a smaller draft model with verification by the main model. Attention and feed-forward computation can also be split, but no single disaggregation pattern is presented as best for every workload.
The presenters discuss combining specialized hardware and using agents to generate optimized kernels. Their speedups are vendor-reported examples with specific workloads, not general guarantees. Future ideas include workload-specific compilers and silicon design, while the immediate engineering lesson is to match resources to the model, context lengths and latency or throughput requirements.
Watch on YouTube
