Bilawal Sidhu traces how video models moved from silent, inconsistent clips to photorealistic audiovisual generation. Jointly training on image, video, text, and audio lets models coordinate dialogue, ambience, sound effects, and motion, while image and voice references can make recurring characters and assets more consistent across generations.
He contrasts offline diffusion with autoregressive world models. Diffusion can denoise a whole space-time block at once, producing coherent short clips, but an autoregressive model predicts each next frame from previous frames and live user input. That architecture enables responsive camera movement, promptable events, and theoretically open-ended generation, although compute demands and finite context still constrain practical sessions.
Longer consistency requires spatial memory rather than retaining every frame. Hybrid approaches can attach selected reference frames to their 3D camera poses, letting a model recall important views without filling its context window. Explicit meshes and Gaussian splats can also supply stable geometry while a generative model handles appearance, motion, and creative scene extension.
Sidhu expects real-time feedback and control to reshape production workflows. Creators could direct camera pose, timing, performance, and scene events while the model renders, then combine generated environments with familiar 2D and 3D tools. Fully embodied agents that reason and perform inside these worlds remain a longer-term step, but the same technology already points toward simulation for robotics as well as entertainment.
Watch on YouTube
