Pat Simmons demonstrates a voice-driven assistantA voice interface lets people interact with software through spoken input and audible responses. using GPT Realtime 2 and explains how tool calls connect conversation to computer actionsComputer use lets an AI agent interact with software through interface controls such as screens, buttons, keyboards and pointers.. He uses a coding agent to build the applicationCode generation uses AI or another automated system to create source code from instructions, examples, schemas, or higher-level specifications. and add functions incrementally.
The demonstration adds push-to-talk to avoid continuous streaming, browser search and link selection, and an application connection for creating notes. It then uses accessibility information and a desktop-control tool to operate a video editor.
Simmons shows successful actions alongside misrecognition and permission prompts. He notes that application accessibility varies and that latency and usage costs remain practical limits.
Watch on YouTube
