Backtesting & results
Out-of-sample and walk-forward
The two deep-backtest settings that decide how your window gets cut — what each one holds back, what the dates under them mean, and how to read the verdict that comes out.
Updated 2026-09-07·Rev. 2026.09
Two of the eight checks in the Edge Report don't just read your run — they change how it's scored. Both live in the deep backtest controls, and both are asking the same question at different resolutions: is this result a property of the strategy, or a property of the window you happened to test?
Out-of-sample holdout
Start with the two words. In-sample is data your strategy has already been measured on — every candle you had in front of you while building it. Out-of-sample is data it hasn't been: the same market, a stretch of it deliberately kept back.
The distinction matters because you are part of the fitting. Every time you moved a period from 20 to 22 because the result got better, you fitted the strategy to that window a little harder — the same thing an optimiser does, just slower and by hand. A number measured on the window you tuned against can't tell you whether you found something or only described what already happened there.
The holdout breaks that circle by keeping a slice away from you:
- You pick 20%, 25% or 30%. Spera cuts the window at that point in time — one clean before and after, not a random sample of candles.
- Everything you did while building happened on the earlier part.
- When the deep run finishes, the strategy is scored a second time on the later part on its own.
- The two risk-adjusted returns are divided. That ratio is the verdict.
It's the difference between your score on the practice questions you revised from and your score on a paper you haven't seen. Only the second one predicts anything.
Every edit, every setting you tried, every result you looked at.
Scored once, at the end. Never seen while tuning.
| Setting | What it holds back |
|---|---|
| Off | Nothing. The report is scored in-sample only, and the run records that it was not judged. |
| 20% / 25% / 30% | The final 20/25/30% of the window, by time. |
Under the control, Spera resolves your percentage to an actual date — "Holds out everything from Aug 28, 2026 (~7 days)". That's the point of showing it: a percentage is an abstraction, a date is something you can check against what you remember changing. The run snaps the split to the nearest candle close.
What the holdout verdict means
The check compares risk-adjusted return before and after the split, as a retention figure — 0.85×
means the held-out stretch kept 85% of the in-sample Sharpe.
| Verdict | Retention | Reading |
|---|---|---|
| Holds up | ≥ 0.70× | On data it was never tuned against, the strategy kept most of its risk-adjusted return. |
| Degraded | 0.30× – 0.70× | It kept only part of it. |
| Fitted | < 0.30× | The edge did not survive. The backtest was largely fitted to its own history. |
| Not enough evidence | — | The held-out stretch was too thin to judge. |
| No in-sample edge | — | There was no edge before the split, so there's nothing to confirm or break. |
| Account lost | — | The account was wiped out inside the window. That's the outcome, not a missing reading. |
Retention below 0.50× caps your letter grade.
What each verdict asks of you: Holds up — nothing, move on to paper trading. Degraded — size smaller than the in-sample numbers suggest, because the held-out stretch is the more honest of the two. Fitted — go back to the graph; tuning it further against this window will only make the reading worse.
When the holdout can't score
The check refuses to answer rather than guess. It needs all of:
- at least 20 closed trades in the held-out stretch — fewer can't separate skill from luck;
- a held-out stretch covering at least 10% of the tested window;
- an in-sample Sharpe above 0.1 — there has to be an edge before the split for retention to mean anything.
If you're seeing Not enough evidence, the fix is a longer date range, a lower timeframe, or a strategy that trades more often — not a different holdout percentage.
Quick runs deliberately show you no holdout number
On a quick backtest the panel says only "In-sample only — run a Deep backtest for the out-of-sample check", with no figure attached. That's on purpose. The moment a held-out number is visible inside your edit-test loop, you start tuning against it, and the holdout quietly becomes in-sample.
Walk-forward windows
The holdout is one split, and one split can land on a favourable slice by luck. Walk-forward asks the same question repeatedly: cut the window into equal consecutive periods and score each one on its own, with the same graph and the same settings.
Money made in a clear majority of periods, no period materially negative.
One period carried the whole result. Remove it and there is nothing left.
| Setting | What it does |
|---|---|
| Off | The report won't say whether the edge held in every period; the run records that it was not judged. |
| 4 / 6 / 12 | Cuts the window into that many equal periods. |
Same idea as the holdout date: the control resolves your count to a span and a first boundary — "6 periods of ~5 days, the first ending Aug 11, 2026". "6 windows" says nothing on its own; "6 periods of ~5 days" is a thing you can argue with. Each boundary snaps to the nearest candle close.
What the walk-forward verdict means
| Verdict | Reading |
|---|---|
| Consistent | The strategy made money in a clear majority of periods, and no period was materially negative. |
| Concentrated | The edge lived in a minority of periods, or one period was materially negative. The whole-window result rests on a favourable stretch. |
| Not enough windows | Too few periods could be scored to describe a distribution. |
| Account lost | The account was wiped out inside the window. |
Concentrated is the finding worth taking seriously. An edge that only appears in one slice is a lucky quarter wearing a track record — and the whole-window Sharpe above it will look perfectly respectable while that's true.
When walk-forward can't score
A period is only scored if it carries at least 10 closed trades and at least 30 bars, and the check needs at least 3 scored periods before it will describe a distribution.
That's the real constraint on window count: 12 windows over a short range produces twelve periods too thin to score and no verdict at all. More windows is not more rigour.
Choosing settings
Start at 25% and 6
They're the middle options for a reason. Push the holdout to 30% when your range is long enough that the held-out stretch still clears 20 trades; drop to 20% when it doesn't. Move to 12 windows only when each period would still carry 10+ trades.
Both checks are deep-run only — a quick backtest covers just the candles on your chart, too short to split meaningfully.
Turning them Off makes the letter easier, not more honest
Both are grade gates, and a gate that never ran can't lower anything. A run with the holdout Off will happily show a letter that a Fitted verdict would have capped. The grade on an unjudged run is not the same claim as the grade on a judged one — the Edge Report records which you're looking at, and the letter alone does not.
Back to the Edge Report for the other six checks.