What is differential testing?

Definition

Differential testing uses one implementation, model or specification as a comparison point for another. When both receive the same inputs, a disagreement identifies a case that deserves investigation even when the expected answer was not written in advance.

The method is useful when translating verified models into production languages or comparing optimized implementations. Agreement is evidence, not a proof, because both systems can share the same mistake or fail on cases that were never generated.

ELI5

Differential testing gives the same inputs to two or more implementations and compares their outputs. A difference points to a case that needs investigation even when nobody wrote the correct answer beforehand.

For example, a newly optimized AI inference engine and a trusted reference engine can process the same test prompts and numeric inputs. If their results differ beyond an allowed tolerance, engineers inspect that case, although agreement alone cannot prove both systems are correct.

Acronyms and aliases

differential software testing variant

Frequently asked questions

What does differential testing compare?

It compares outputs, errors or state changes from multiple implementations that receive the same generated or curated inputs.

Is differential testing a form of formal verification?

No. It is a testing technique based on sampled executions, although it can compare production code against a formally modeled reference implementation.

Videos explaining differential testing