Out-of-sample testing reserves data from the process used to design or tune a model or strategy. Performance on this unseen holdout gives a more independent estimate of generalization than results on the same data used for development.
The test loses independence if researchers repeatedly inspect it and revise the strategy in response. Time-series work should also keep training data earlier than the test period and account for execution conditions that would have existed at the time.
ELI5
Out-of-sample testing evaluates a model or strategy using data that was not used to design or tune it. The unseen data gives a more honest estimate of how the system may perform beyond the examples developers already know.
For example, a trading strategy can be built on earlier market data and tested once on a later reserved period. If researchers repeatedly inspect that period and revise the strategy, it stops being an independent test.
Why is out-of-sample testing important?
It helps distinguish patterns that generalize from rules that merely fit noise or quirks in the development data.
Can out-of-sample data be reused for tuning?
Once it influences tuning or selection, it is no longer a clean holdout and a new independent test set is needed.
