Out-of-Sample Testing (OOS)
Education only · our voice · free public data
Definition
Evaluating a strategy on data that played no part in its design, parameter choice, or selection. In-sample (IS) is where you fit; out-of-sample (OOS) is the honest exam. Reported as OOS years, number of folds, and the IS/OOS performance ratio.
How to read it
The only performance that predicts the future is performance on data the model has never touched. A strategy that shines in-sample and fades out-of-sample is over-fit. Watch the degradation ratio (OOS Sharpe / IS Sharpe): 0.7-1.0 is healthy, 0.3-0.5 is fragile, near 0 or negative means the IS result was curve-fitting. The subtle trap is that OOS stops being out-of-sample the moment you iterate on it - every time you look at the test set and adjust, you leak information and the 'OOS' becomes a second in-sample.
How practitioners use it
Used as context among multiple indicators — never as a standalone signal to act.
Less common professional uses
Sequential OOS on time series must respect purge-and-embargo to prevent leakage from overlapping labels and serial correlation; naive k-fold shuffling of time-series data is a classic leakage bug (see Purged Walk-Forward). Repeated OOS peeking is a hidden multiple-testing problem: the effective number of trials includes every time you revisited the hold-out, so the correct honesty check is Deflated Sharpe over ALL those looks, not a single OOS t-stat. Combinatorial Purged Cross-Validation (CPCV) generates many IS/OOS path combinations to produce a DISTRIBUTION of OOS Sharpes rather than one number, exposing how lucky a single split was. A high IS/OOS correlation of RANKINGS across variants matters more than any single OOS point: if the best IS variant is not near the best OOS variant, your selection procedure has no predictive validity.
Sources & provenance
Lopez de Prado (2018), 'Advances in Financial Machine Learning', Ch. 7 & 12; Bailey et al. (2014), 'Pseudo-Mathematics and Financial Charlatanism', Notices of the AMS
This page is educational content published by Pachira Aquatica Global LLC. It is not investment advice and not a recommendation.