What is training data attribution?

Definition

Training data attribution connects a model prediction, representation, or behavior with influential examples from training. Methods can compare gradients, parameters, embeddings, activations, or the effect of adding, removing, or reweighting candidate examples.

Attribution can help diagnose errors, identify contamination, investigate memorization, and improve datasets. Results are estimates rather than perfect historical traces, especially in large models where many examples interact during optimization.

ELI5

Training data attribution tries to find which learning examples most influenced an AI model's answer or behavior. It helps researchers connect a surprising result with possible sources in the training data.

For example, if a model repeatedly gives an incorrect fact, attribution tools can highlight training examples that may have taught or reinforced that mistake. A human can then inspect those examples.

Frequently asked questions

What can training data attribution help investigate?

It can help investigate errors, memorization, duplicated data, contamination, bias, harmful patterns, unexpected capabilities, and dataset quality.

Is training data attribution exact?

Usually not. It estimates influence in a complex training process where many examples and parameter updates interact.

Videos explaining training data attribution

  1. Tim Scarfe and Tom McGrath beside the words Design AI from the Inside