What is data debugging?

Definition

Data debugging investigates errors such as incorrect labels, duplicates, missing values, contamination, formatting defects, leakage, harmful examples, and unrepresentative sampling. It connects observed model failures with possible problems in the underlying data pipeline.

Useful methods include dataset profiling, provenance tracking, targeted searches, influence estimates, clustering, model-error analysis, and controlled retraining. Fixes should be validated because removing or changing data can create new gaps or alter behavior elsewhere.

ELI5

Data debugging means finding mistakes or weak spots in the information used by an AI system. Better data can prevent the model from learning or repeating avoidable errors.

For example, if a model confuses two medical terms, a team can inspect influential training examples and discover that some labels were swapped. They correct the data and test the model again.

Frequently asked questions

What problems can data debugging find?

It can find incorrect labels, duplicates, leakage, contamination, missing values, formatting errors, harmful examples, provenance gaps, and unrepresentative coverage.

Why must a data fix be evaluated?

Changing data can improve one failure while reducing coverage, introducing bias, or affecting unrelated behaviors, so the resulting model must be retested.

Videos explaining data debugging

  1. Tim Scarfe and Tom McGrath beside the words Design AI from the Inside