Context manipulation targets information that a model uses during a task, such as instructions, retrieved documents, tool output, conversation history, memory, or intermediate notes. An attacker or misaligned agent may insert, suppress, reorder, or distort that information.
Defenses include source provenance, trust boundaries, instruction hierarchy, content isolation, retrieval filtering, state validation, and audit logs. Sensitive actions should not depend on untrusted context alone, especially when external content can influence tool use.
ELI5
Context manipulation means changing the information an AI system sees so it makes a different decision. The altered information can be an instruction, memory, search result, document, or tool message.
For example, a hidden instruction inside a webpage might tell a browsing agent to ignore its real task and reveal private data. Separating untrusted page content from trusted instructions helps prevent that.
What information can context manipulation target?
It can target instructions, retrieved documents, messages, memory, tool results, metadata, intermediate reasoning records, or any other task context.
How can AI systems resist context manipulation?
Use provenance, trust labels, content isolation, retrieval controls, instruction priority, state validation, restricted tools, monitoring, and approval for sensitive actions.
