Education · Guide

What a backtest does and doesn't tell you (in-sample vs out-of-sample)

What a backtest can and cannot prove: in-sample vs out-of-sample data, selection bias, small samples and fill assumptions, with Stacked Edge's own caveats as worked examples.

A backtest replays a set of trading rules over past prices and adds up what would have happened. It is a useful tool for rejecting bad ideas. It is a much weaker tool for proving good ones. This article explains the difference using plain definitions and the honest caveats we publish on our own pages.

Education only, not financial advice. Every Stacked Edge result mentioned here is a backtest, hypothetical, and has inherent limitations. There are no live or forward results. Trading futures involves substantial risk of loss.

What a backtest does tell you

Whether the rules ran at all. Every entry, exit and stop is defined well enough that a computer can follow it.
How the rules behaved in that specific history: profit, profit factor, win rate, number of trades, and, most importantly, drawdown.
A rough scale for sizing. If a strategy's worst peak-to-trough loss in the test was $10,000 per contract, you know you must not size so that a similar loss ends your account.
Whether an idea is clearly broken. A strategy that loses money even with hindsight is almost certainly not worth trading.

What it does not tell you

Whether the same result will happen again. The future is a different sample of the market.
How the rules behave in conditions that were not in the data. A two-year rising market says little about a crash or a long sideways grind.
What your actual fills will be. Backtests assume a fill model. Real orders face spreads, queues and fast markets.

In-sample versus out-of-sample, in plain English

In-sample data is the data you used to design, tune or select a strategy. Out-of-sample data is data the strategy has never seen, set aside before you made any decisions. A forward test is the strongest version: the rules are fixed and timestamped first, then evaluated on prices that did not exist yet.

The reason to separate them is simple. If you try enough rules on one data set, some will look good by luck alone. In-sample performance tells you how well you fitted the past. Out-of-sample performance is the first honest evidence about anything else.

A backtest result is in-sample whenever the same data influenced which rules you kept. That is the normal situation for almost every retail backtest, including ours.

How we apply this to our own numbers

Here is what our published backtests are, stated without softening. Every figure below is a backtest, hypothetical, from TradingView's Strategy Tester over 2024-09-23 to 2026-09-23.

1. Everything is in-sample. The strategies were found and kept using the same two years they are measured on. We say so on every stats page: the data is the same, so selection bias is present and live results should be expected to be worse.

2. The best-looking set is a pick from a search. Our NQ "in-sample best" set (Connors > Supertrend > half-size VZO > RMI) shows +$133,730, profit factor 2.02 (backtest, hypothetical). It was picked from 372 subsets of nine candidates scored on the same window, and 64% of its profit came in year 2. Our mixed All-Instrument tested set shows +$210,412, profit factor 1.94, closed-trade max drawdown $7,138 (backtest, hypothetical), picked from 16,455 subsets. Its net-profit-to-drawdown ratio of 29.5 looks spectacular, and that is exactly why it should be distrusted: the more subsets you search, the more flattering the best one becomes. Three of that set's six members are flagged Tier B, meaning fragile.

3. One window, and a strong one. The test window was a strongly rising market (NQ from about 20,200 to about 31,300). Profit skews to year 2 in every strategy where we could check. The GC Stack (backtest, hypothetical) made +$460 in year 1 and +$95,300 in year 2, which is 99.5% of its profit from one year.

4. Small samples. ES B12_01 has 42 trades; NQ Supertrend Flip has 45; NQ Connors has 53. NQ Connors showed a net-profit-to-drawdown ratio of 3.05 until four later losing trades arrived, which moved it to 2.48. Four trades changed the headline by roughly a fifth.

5. Fills are idealised. Backtest entries fill at the close of the signal bar with 4 ticks of slippage and $2.50 per contract per side. We have not tested next-bar-open fills for any strategy.

6. Robustness is mostly untested. Neighbouring timeframes and parameter nudges of about 15% have not been completed for most strategies, so we do not know how sensitive any of them is to its exact settings. A strategy that works only at one exact setting is a warning sign.

7. The one longer-history check we ran failed. NQ DPO was tested on 2019–2026 data: profit factor 1.29, max drawdown $21,275, a profit factor of 0.28 through the 2020 crash, and walk-forward results positive in only 48% of folds. Those figures are backtest, hypothetical. DPO is flagged for it, and it stays out of our NQ Stack numbers.

A short checklist for reading any backtest

Is max drawdown shown beside net profit? If drawdown is missing, it is an advertisement.
How many trades? Under about 100 means a handful of trades can swing the result.
How many variants were tried? Ask how many rules, parameters or subsets were searched before this one was shown.
Was any data held out? Look for out-of-sample or walk-forward results, not only the optimised period.
What market regime did it cover? One rising year is not a market cycle.
Are costs and fills realistic? Look for slippage, commissions and the fill rule.
Is profit concentrated? Check year splits and the share earned by the best five trades.
Is anything live? A forward record, with skipped setups and losing trades included, beats any backtest.

How weekly validation helps, and what it cannot do

We re-run every strategy weekly with the same costs and publish a status: Pass, Watch, Flagged or Removed. That can catch a strategy that stops meeting our bar. It is a process, not a guarantee, and it is still evaluated on data that includes the original window. Our forward track record page is empty on purpose: it will fill with timestamped results only after real follow-along posts are published and trades close, and it will include skipped setups and losers.

What to do with this

Treat a backtest as a hypothesis. Use drawdown to decide size, expect live results to be worse than the test, and wait for out-of-sample or forward evidence before leaning on any number. You can see every caveat next to every statistic on our performance page and the validation log. The free daily brief and the waitlist are the only things to sign up for today; billing is not open.