A multimodal agent combines information from formats such as text, images, audio, video, documents, or interface state while pursuing a goal. It can choose the representation that best captures the user's intent rather than forcing every exchange into prose.
Useful multimodal behavior requires alignment between the formats. A reference image should influence the same decision state as explicit constraints and conversation history, while the final response should preserve the source facts behind any visual or textual recommendation.
ELI5
A multimodal agent can understand and use different kinds of information in the same task. Instead of relying only on words, it might also inspect an image or present choices visually.
For example, a shopper can show the agent a photo of a chair style that is hard to describe. The agent can combine that image with a written budget and room size to search for suitable options.
