Model activations are temporary internal values created as information passes through network layers. Their patterns can represent aspects of syntax, objects, topics, goals, uncertainty, or other features that the model uses to compute its output.
Researchers can record, compare, classify, or intervene on activations to study model behavior. An activation pattern is not automatically a clean human concept, so interpretations require multiple examples and causal checks rather than relying on one appealing visualization.
ELI5
A model activation is a temporary signal inside an AI model while it thinks through an input. Different signals become stronger or weaker depending on what information the model is processing.
For example, a certain internal pattern may become active when the model reads text about a city. Researchers can compare many examples and test the signal before deciding what it represents.
