How Do You Spot a Fake Backtest?
Ask how many versions were tried before this one. A backtest is a search, and searching enough parameter combinations on one dataset guarantees an excellent-looking result by chance — which is why sample size and out-of-sample data matter more than the equity curve does.
Why does a great backtest mean so little?
Because you are being shown the winner of a search you did not see. Try 100 parameter combinations on one dataset and the best of them will look excellent whether or not the underlying idea works, because you have selected for noise as well as signal.
This is not fraud and usually not even carelessness. It is what happens when the same data is used to choose the strategy and to evaluate it.
How big are the error bars?
Small samples make near-random results look decisive. The standard error on a win rate is sqrt(p(1-p)/n). For a strategy showing a 60% win rate over 30 trades that is sqrt(0.6 x 0.4 / 30) = 8.9 percentage points, so a 95% interval runs from roughly 42.5% to 77.5%. That result is consistent with a coin flip.
Increase the sample and the interval collapses: at 300 trades the standard error is 2.8 points, and the interval is about 54.5% to 65.5%. So the first question about any backtest is how many trades it contains, and the answer is usually far too few.
What is being left out of the returns?
Costs. Commissions, spread and slippage. A strategy averaging 0.4% per trade over 500 trades a year is earning about 200 points gross; if realistic costs are 0.15% per trade, 75 of those points do not exist. Intraday strategies are frequently profitable on paper and negative after costs, and this single omission explains most of them.
Fills. Backtests assume you were filled at the level. Stops in fast markets are not, gaps skip them entirely, and limit orders at the extreme of a bar often would not have executed.
Survivorship. A universe of today's listed symbols excludes everything that delisted, merged or went to zero. Testing "all S&P 500 members" using today's membership tests a list chosen with hindsight.
Look-ahead. Using a day's close to decide a trade taken during that day, or restated fundamental data as if it were available on the original release date.
What should you ask the author?
- How many parameter sets did you try before this one?
- How many trades are in the result, and over how many distinct market regimes?
- What are the results on data you had never looked at when the rules were fixed?
- What costs and slippage are modelled, and what happens if you double them?
- What does the worst drawdown look like, in percent and in months?
An author with a real process answers all five immediately. The answers are also, in themselves, the most informative thing you will learn about how they work.
What replaces a backtest?
Forward results, recorded as they happen. Every trade, including the losers, timestamped before the fact, on a record the person being evaluated cannot edit cannot be overfitted, because there is no dataset to search — the commit exists before the fact.
This has been measured rather than argued. Bailey, Borwein, López de Prado and Zhu showed that a high backtested Sharpe ratio is easy to reach by trying a modest number of strategy configurations, and that the probability a backtest is overfit rises with the number of configurations tried.[1] Their practical point is the one to take to any backtest you are shown: almost nobody reports how many variants they ran, so you cannot assess the overfitting, which means an unreported trial count is itself the finding.
The same problem shows up in peer-reviewed finance, which is the strongest evidence that it is structural rather than a retail failing. Harvey, Liu and Zhu reviewed hundreds of published return predictors and concluded that, given how many had been tried, a new factor should clear a t-statistic above 3.0 rather than the conventional 2.0.[2] Survivorship compounds it: study only what survived and apparent skill appears from nothing, because the failures never enter the count.[3]
That is what kappi records instead: the trade is committed before it resolves, sealed, then published on a Merkle-anchored log with PnL, RME, correlation to SPX, mean R:R and trade count over 30, 100 and 200-day windows. Fifty forward trades on that record say more than five thousand backtested ones. $15/month.
Sources
- Bailey, Borwein, López de Prado & Zhu, 'Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance', Notices of the AMS 61(5), 2014, 458 read 2026-08-16
- Harvey, Liu & Zhu, '… and the Cross-Section of Expected Returns', Review of Financial Studies 29(1), 2016, 5–68 — argues a newly claimed factor should clear a t-statistic above 3.0 read 2026-08-16
- Brown, Goetzmann, Ibbotson & Ross, 'Survivorship Bias in Performance Studies', Review of Financial Studies 5(4), 1992, 553–580 read 2026-08-16
Frequently asked questions
What is overfitting in a trading backtest?
Tuning a strategy until it fits the data it was tested on. Trying enough parameter combinations on one dataset guarantees an excellent-looking result by chance, whether or not the underlying idea works.
How many trades does a backtest need?
Enough that the error bars are smaller than the claim. At 30 trades a 60% win rate has a standard error of 8.9 points, giving a 95% interval of roughly 42.5% to 77.5% — consistent with a coin flip.
What costs do backtests usually leave out?
Commissions, spread and slippage, plus the assumption of perfect fills. A strategy earning 0.4% per trade over 500 trades a year loses about 75 of its 200 gross points to costs of 0.15% per trade.
Is a backtest worth anything at all?
As a filter for ideas, yes. As evidence of skill, no — it is a search over data that already exists. Forward results committed before the fact cannot be overfitted, which is why fifty of them beat five thousand backtested trades.