Károly Zsolnai-Fehér examines how DeepMind gives the compact Gemma 4 model multimodal capabilities without relying on a large collection of specialist networks. Smaller systems commonly pass visual or audio representations from separate encoders into a language model, creating distinct perception and reasoning stages.
Gemma 4's larger configuration instead divides images into patches and audio into short chunks, projects those inputs into its internal representation and feeds the resulting tokens directly into the main transformer. The model must therefore learn perception and reasoning together rather than receiving a translation from a separate vision or audio network.
Károly Zsolnai-Fehér argues that this architecture removes hundreds of millions of specialist parameters while allowing a comparatively small local model to process text, images and audio. Because DeepMind published the design, the same approach may also help other open models improve multimodal capability and efficiency.
Watch on YouTube
