Valeria Wu Fon and Tom Ouyang explain how speech-to-speech research at Google DeepMind builds on earlier speech recognition and synthesis systems. Tom Ouyang describes a move from separately engineered components to multimodal pre-training that learns relationships among text, audio and video within a shared representation.
Valeria Wu Fon and Tom Ouyang frame the research goal as combining natural low-latency conversation, reasoning and task completion, and multimodal input and output. Increasing reasoning effort can improve answers while delaying the first spoken response, so advancing one capability can impair another. The speakers also emphasise multilingual performance rather than treating English as the sole target.
Valeria Wu Fon and Tom Ouyang illustrate these trade-offs through streaming translation, visually grounded search, roadside-assistance dialogue and a conversational avatar. The examples highlight speaker continuity, accurate alphanumeric information, deciding when background sound merits a response, and coordinating speech with tools and visual output. They are demonstrations of particular settings, not proof of universal reliability.
Watch on YouTube
