John Ousterhout argues that inferenceInference is the use of a trained model to produce an output from supplied input. and agentic workloads increasingly depend on short coordination messages, even while training still moves large amounts of data. When distributed GPU jobs must synchronize between compute phases, a slow message can leave processors idle and limit overall throughput.
The talk traces tail latencyTail latency measures the slowest fraction of requests, commonly expressed with high percentiles such as p95, p99 or p99.9. to incast, where large transfers fill a switch queue and short messages wait behind them. It also explains why sender-side congestion control reacts after network feedback and why byte streams make it difficult to prioritize individual short messages.
John Ousterhout presents Homa as a message-based transport that knows message lengths, lets receivers pace additional packets with grants, and uses switch priority queues to favor short exchanges. In one benchmark described in the talkA benchmark is a defined test or collection of tasks used to compare a system's performance under specified conditions., Homa's 99th-percentile latency for short messages is reported at less than 100 microseconds, versus more than one millisecond for TCP, roughly a 13-fold difference. That is the speaker's sample benchmarkEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria., not an independently verified general performance guarantee.
Watch on YouTube




