Out-of-sample testing reserves data from the process used to design or tune a model or strategy. Performance on this unseen holdout gives a more independent estimate of generalization than results on the same data used for development.
The test loses independence if researchers repeatedly inspect it and revise the strategy in response. Time-series work should also keep training data earlier than the test period and account for execution conditions that would have existed at the time.
Acronyms and aliases
OOS testing acronymholdout testing synonym
General terms
Related terms
Frequently asked questions
Why is out-of-sample testing important?
It helps distinguish patterns that generalize from rules that merely fit noise or quirks in the development data.
Can out-of-sample data be reused for tuning?
Once it influences tuning or selection, it is no longer a clean holdout and a new independent test set is needed.
