What is interpretability?

Definition

Interpretability methods examine features, internal activations, relationships or decision processes to make aspects of a model understandable. Some approaches describe observed patterns, while others test whether changing a component changes the output.

An understandable explanation is not automatically a complete or causal account. Findings may apply only to particular inputs, features or models, so independent checks and stated limits remain important.

ELI5

Interpretability is an attempt to understand what is happening inside a complicated AI instead of judging it only by the final answer. It is similar to examining both a machine's output and the parts that influenced it.

For example, a researcher might test which internal features change when an AI recognizes a risky instruction. That clue can help investigation, but it does not automatically explain every decision.

Does interpretability explain every answer?

No. A method may reveal only a small part of a model's behavior or work only under certain conditions.

Why does it matter for safety?

Understanding internal mechanisms can help investigate failures, but it does not by itself guarantee safe behavior.

  1. A digital shield beside the headline AI Safety.
  2. Nate B Jones beside blue and white text reading Reading AI Neurons.
  3. Nate B Jones beside a headline about AI interpretability.
  4. The words Inside Claude's Reasoning beside a flat brain diagram with connected concept nodes
  5. K谩roly Zsolnai-Feh茅r beside the headline AI Activations Explained.
  6. Nathaniel Whittemore in a blue top on the left beside the blue-and-white CLAUDE WORKSPACE headline against black.
  7. Neel Nanda and Hannah Fry beside the words Reading AI's Hidden Thoughts
  8. Theo Jaffee, Nick Cammarata, Eliezer Yudkowsky and Ronny Fernandez against black beside the headline Already Clean?.
  9. Nathaniel Whittemore, Nick Cammarata, Liv Gorton and Neel Nanda against black beside the headline Readable AI?
  10. Nick Cammarata against a black background beside the white and blue headline 'READING NEURAL NETWORKS'.
Definition card for interpretability: What is interpretability?

Interpretability is the ability to understand how a model represents information or produces its behavior.