Neil Zeghidour traces voice agents from constrained assistants to transcription-based language-model pipelines and speech-to-speech systems. Neil Zeghidour argues that natural-sounding speech and useful tool actions have improved, but many systems still behave like walkie-talkies because they cannot listen and speak naturally at the same time.
Neil Zeghidour explains how neural codecs turn audio into compact tokens and how multi-stream models can represent the user and assistant concurrently. These techniques support interruptions, acknowledgements and full-duplex conversation, but the talk identifies a trade-off between conversational naturalness and the reasoning capabilities of larger text models.
Neil Zeghidour describes a hybrid approach in which a small, responsive speech model handles the conversation while a background text model supplies reasoning, retrieval and tool use. Keeping those components separate could allow the reasoning backend to improve without retraining the audio model, although the architectural trade-offs remain a research and engineering challenge.
Watch on YouTube
