Out-of-sample testing is the habit that separates a real trading edge from a story told by old data. The idea is simple: you judge a strategy on prices it has never seen, after every rule has been fixed. The hard part is keeping that promise when a favourite idea starts to fail.
At TradeLab Research I learned that good intentions are not enough. You need rules that make cheating impossible, including the quiet cheating we all do by accident. This post explains the four rules we use, the results they produced and a checklist you can apply to your own work. Every number below comes from historical simulations, not investment advice.
Table of Contents
What out-of-sample testing means, and why most backtests skip it
A normal backtest uses one block of history for everything. You pick an indicator, tune its settings, adjust the stop, add a filter and check the result on the same prices. Each change makes the curve a little better, and each change also fits the rules a little more tightly to that particular past.
This is called overfitting. An overfitted strategy has learned the noise in its data, not a pattern that repeats. It looks excellent on the chart you built it on and then disappoints on the next year of prices.
Out-of-sample testing breaks that loop. You keep a block of data aside, you do not look at it while building, and you use it once at the end. If the strategy only worked because it was tuned to the first block, the second block will tell you.
The stakes are real for retail traders. EU regulators’ studies, cited by ESMA in 2018, found that 74-89% of retail CFD accounts lost money. Many of those traders were following rules that had looked good on past charts. Honest out-of-sample testing is one way to find out before money is at risk.
Rule 1: Split time into SEARCH, VALIDATION and LOCKBOX
Our first out-of-sample testing rule is about how we cut the data. We use broker 1-minute prices from 2017 to 2026, with the broker’s recorded spread at each fill. That history is split into three periods, always in time order:
- SEARCH, June 2017 to December 2022: tuning is allowed. We can try settings, compare families and look at charts.
- VALIDATION, 2023 to 2024: run once, only on candidates that were frozen at the end of SEARCH.
- LOCKBOX, 2025 onwards: sealed. It is opened once for the whole programme, and it has never been opened.

Why the split must follow time order
Some people split data at random, putting some days in the training set and some in the test set. For trading this leaks information. Markets have regimes, and a random split lets the strategy learn from days right next to the ones it is tested on.
A time-ordered split mirrors real life. You build a strategy with what you know today and trade it on tomorrow’s prices. Out-of-sample testing should recreate that situation as closely as possible, so the test period always comes after the period used for building.
What each period is allowed to do
The important part is not the dates. It is the rules about what each period may influence. SEARCH can change anything. VALIDATION can only say yes or no to a frozen candidate. The LOCKBOX exists for the final check of the whole programme, not for any single idea.
If a VALIDATION result makes you change a rule and run again, VALIDATION has become part of SEARCH. They test on “new” data, see a loss, tweak a filter and test again. After three rounds the new data is no longer new, and the out-of-sample testing has quietly turned into in-sample tuning.
How long should each out-of-sample testing period be?
There is no perfect length, but there is a floor. The held-out period must contain enough trades to judge. Our VALIDATION bar asks for at least 100 trades, and SEARCH candidates need at least 200. A rule that trades twice a month cannot be judged on six months of out-of-sample testing.
Slow rules make this hard. We dropped a crypto funding-rate rule before testing because it had only 94 to 193 independent episodes, under the 200-trade minimum.
Rule 2: Pre-register every rule and seal it with a hash
The second rule stops us from moving the goalposts. Before every run, we write down the rules, the grid of settings and the pass threshold. Then we compute a SHA-256 hash of that document. The hash is a fingerprint: change one character and it changes completely.
Because the hash is recorded before the run, nobody can quietly adjust the bar after seeing the result. If we find a genuine mistake in a plan, we write a dated amendment. We never make silent edits. This is what pre-registration means in practice, and it is the backbone of our out-of-sample testing.
Our VALIDATION pass bar, written before any run, has four parts:
- A t-statistic of at least 2.0 on at least 100 trades.
- A t-statistic of at least 2.5 if the idea had already been tested on that data.
- The rule beats 95% of 200 coin-flip runs with the same entries and random direction.
- The result stays positive when we charge 1.5 times the spread.
The coin-flip part is explained in detail in our luck test guide. The spread part matters because costs decide many results; our post on trading costs shows how much.
Why go this far? Because out-of-sample testing only works if the test is decided before the answer is known. Without a fixed bar, it is very easy to decide afterwards that t 1.9 is “close enough”, or that one bad market “doesn’t count”. A hash makes those conversations impossible.
A simple way to seal your own plan
You do not need special software. Write your plan in a plain text file: the rules, the settings you will try, the period you will test on and the pass bar. On Linux or macOS, the command sha256sum plan.txt prints the fingerprint; Windows has a similar built-in tool.
Save the fingerprint somewhere you cannot edit later, such as an email to yourself or a message to a friend. Then run your out-of-sample testing. If the result tempts you to change the plan, the dated fingerprint reminds you what you promised.
Rule 3: Demand a plateau, not a lucky peak
The third rule decides which ideas are allowed to reach out-of-sample testing at all. A strategy usually has several settings: a lookback length, a stop distance, a time window. If you try many combinations, one of them will look best, often by luck.
So we do not accept a single best setting. A SEARCH candidate needs a plateau: all neighbouring settings must also be positive. If the 20-bar version works but the 18-bar and 22-bar versions lose, the 20-bar result is probably a lucky peak.
Our full list of SEARCH candidate requirements is short and strict:
- A plateau of positive neighbouring settings.
- Positive on at least two thirds of the markets tested.
- At least 200 trades.
- A t-statistic of at least 2.5.
The plateau rule did a lot of work in our smart money concepts research. We tested 34 mechanical SMC + MSNR settings across seven markets. The best plateau reached t 0.65; the best single setting reached t 1.05. A single setting can look promising while its neighbours show nothing. Neither number came close to the 2.5 needed, so none of the 34 settings went on to out-of-sample testing. More detail is in our SMC strategy backtest post.

Rule 4: Treat re-tested ideas as contaminated
The fourth rule is the one most traders have never heard of. Every time you look at a period of data, you learn something about it. If you then test an idea you already tried on that data, your result is contaminated by what you saw before.
Our contamination rule for out-of-sample testing is simple. If an idea had already been tested on the data in question, it needs t ≥ 2.5 in VALIDATION instead of 2.0. The bar rises because the test is less independent than it looks.
The same logic applies when many ideas are tested at once. This is the multiple comparisons problem: more tries mean more lucky winners. When we tested 25 suggested improvements on markets they had never seen (AUDUSD, GBPJPY and ETH), we used a Bonferroni-corrected bar of t ≥ 3.11. Zero of the 25 were confirmed.
That was a useful form of out-of-sample testing too. Ideas that come from looking at one set of markets should be judged on markets that played no part in creating them. If a fix only works where it was invented, it was probably fitted to that place.
The voided pass: when out-of-sample testing exposed a bug
Here is the out-of-sample testing story I tell most often. In round 1 of our edge search, a Williams volatility breakout rule on indices passed VALIDATION. In 2023-2024 it made a t-statistic of 2.24 on 868 trades. On the pre-registered bar, that was a pass.
Then we checked the trades. The rule’s end-of-day exit had been set at the broker’s daily break. That meant trades were held over the reopen, and over weekends, when the rule was meant to be flat. The extra holding supplied about 120R of the rule’s 156R.

We voided the pass and recorded it as a dated amendment. The corrected rule then failed. No edit was made in silence, and the bug report is part of the record.
The lesson is that out-of-sample testing can only judge the strategy you actually simulated. If the code does something the written rules do not intend, a clean test of the wrong rule is still wrong. We see this in client bots too: duplicate signals, missing timeouts and exits that do not match the plan. That is a large part of our independent bot testing work.
Why the lockbox stays sealed when nothing passes
Our edge search ran six rounds across 38 families and 806 rule settings. Fourteen candidates made it out of SEARCH. None survived VALIDATION. There were zero valid passes.
Round 2’s best candidate was a trend-day continuation rule on US500, US100 and US30. In SEARCH it made +0.13R per trade with t 2.84 on 985 trades, positive on all three markets and in 5 of 6 years. In VALIDATION it made −0.022R per trade, t −0.35, and 49.5% of coin-flip runs did as well.
Another case was the Williams breakout re-search. In SEARCH its profit came entirely from Bitcoin and Ethereum; without crypto it was about zero. In 2023-2024 the crypto effect vanished, with t-statistics of 1.02, 1.22 and −0.43. Out-of-sample testing turned a search-period win into a clear fail.
The same pattern showed up in our independent assessment of a 39-strategy library. Four trend rules, including Donchian and Supertrend, were positive on Bitcoin H1 in 2017-2023 but were not confirmed in 2024-2026. Out-of-sample testing did not call them failures; it called them inconclusive, which is the honest label.

So why not open the LOCKBOX anyway, just to see? Because it can only be opened once. The LOCKBOX is the last stage of our out-of-sample testing, there to confirm something that already passed. With no pass to confirm, opening it would only give us a new set of data to search, and the last clean test of the whole programme would be gone.
This is the part of out-of-sample testing that feels most painful. You have unused data sitting there and a list of near misses. Our near misses are a 20-day Donchian breakout long (t 2.04 on 197 trades, with Bitcoin and Ethereum supplying 88% of the profit), buying near the 52-week high with a 5-day hold (t 2.03, not stable) and a pre-FOMC drift in US indices (+0.040R per trade, t 2.38 on 191 trades).
None of these is a pass. They go to forward paper testing, with a read-out date of 30 November 2027. Fresh data arriving in real time is the most honest out-of-sample testing there is, because nobody can have seen it in advance.
We have since run the same out-of-sample testing method on two more programmes: Binance perpetual futures (52 settings, lower costs) and Weltrade synthetic indices (24 settings). The same rules applied, with pre-registered bars, frozen candidates and a sealed final period. Neither produced a single candidate, so neither final period was ever opened.
What out-of-sample testing cannot fix
Out-of-sample testing protects you from fooling yourself with old data. It does not fix costs, and it does not fix bad data. In our cost work, Asian-session FX mean reversion made +0.04 to +0.11R per trade before costs and paid 0.14 to 0.22R per trade in spread. All 12 settings were negative after costs.
Data quality is just as important. In one study, a broker export recorded a spread of zero for whole months of Bitcoin, 12% of all its minutes. A zero-spread month makes any short-term rule look better than it can be. Check the spread column before you start any out-of-sample testing.
Finally, a split does not help if you have already looked at the test period while designing the rule. If you read about a pattern that worked in 2023 and then “test” it on 2023, that is not out-of-sample testing at all. The test data must be unseen by you, not only by the code.
Out-of-sample testing checklist for retail traders
You do not need our infrastructure to use the same discipline. Here is a practical checklist I would give anyone building a strategy at home. Use it before you trust any backtest, your own or one you are being sold.
| Step | What to do | Why |
|---|---|---|
| Split time | Keep the most recent years aside | The test must follow the build |
| Write it down | Rules, settings and pass bar, dated | Stops you moving the goalposts |
| Fingerprint it | Save a hash or send the file to a friend | Proves the plan came first |
| Check neighbours | Nearby settings must also be positive | Rejects lucky peaks |
| Count your tries | Raise the bar for re-tested ideas | Corrects for contamination |
| Run once | One run on the held-out period | Keeps the test honest |
| Add costs | Real spread and overnight financing | Most small edges die here |
A few points deserve extra emphasis:
- Do not open the test period early “just to have a look”. One peek is enough to contaminate it.
- If the result fails, write down that it failed. Do not tweak and rerun on the same period.
- Check your trades against your written rules. A bug can pass a test as easily as an edge can.
- Use a luck test on the held-out results as well, so random direction cannot explain them.
- Keep a near-miss list for forward testing, but never call a near miss a pass.
If your backtest only has one period, treat every number in it as optimistic. The data dredging problem is not about bad people; it is about normal people trying one more setting. Out-of-sample testing is the cheapest protection you can buy against it.
How to read your out-of-sample testing result
When the held-out run is done, compare it with your written bar and nothing else. A pass means the idea deserves forward paper testing, not real money yet. A fail means the idea, as written, is finished. An in-between result is a near miss: put it on a watch list.
Be suspicious of a result that is much better out of sample than in sample. That is rarely good news. More often it means a bug, a data gap or a period that favoured the rule by chance, like the voided pass above. Good out-of-sample testing treats surprises as questions, not rewards.
Key takeaways
- Out-of-sample testing means judging a strategy on data it has never seen, after every rule is frozen.
- Split time in order: SEARCH for tuning, VALIDATION run once, and a sealed LOCKBOX for the final check.
- Pre-register rules and pass bars, and fingerprint them with a hash so nothing can change afterwards.
- Accept only plateaus, not single best settings, and raise the bar to t ≥ 2.5 for re-tested ideas.
- Our 806 settings produced 14 candidates and zero passes, so the LOCKBOX stays sealed.
- A voided pass showed that a bug can look like an edge; check the trades, not only the totals.
You can read every rule we apply on our how we test page, and see the overall picture in our backtest results post. If you want your own bot or written strategy put through the same out-of-sample testing on your broker’s real costs, you can get your bot tested.
