NeuralTEST
Backtest overfitting detectorRun the test
Checklist · taken from our own stumbles

10 ways your backtest fools you

None of these ten mistakes is visible in the equity curve. All of them inflate it. We made every one of them while measuring our own strategies for months, and each one became either a check in the test or a question we ask you before giving a verdict.

The figures come from our own studies (2026). No third parties are named because none are needed: the mistakes are the same everywhere.

  1. Measuring the past with today's list of assets

    What it is
    If your universe is today's, it only contains the survivors. The ones that went bankrupt, left the index or were delisted are missing, and they are exactly the ones that would have lost.
    How it bit us
    An insider-buying strategy showed a Sharpe of 1.16. With the index composition as it was on each day, it fell to 0.17. 84 % of the gain came from a single stock, before it joined the index.
    How to check it
    For every date, the asset list must be the one that existed that day, including those that later disappeared. If you do not have it, say so in the report: it is not a detail.
    Does the test see it?
    Partly: we ask you, and your answer is recorded in the report.

  2. Choosing while looking at the test period

    What it is
    You pick pairs, parameters or assets using data that includes the period you later call “out of sample”. It is no longer out of sample: the choice saw the future.
    How it bit us
    A set of stock pairs had an attractive curve. The pair selection had seen the test period. Repeated honestly, the curve came out at zero.
    How to check it
    Write down the order of your steps, with dates. If any decision (which assets, which parameters, which filter) used data after the start of the test, the test does not count as one.
    Does the test see it?
    Partly: we ask you, and your answer is recorded in the report.

  3. Trying many variants and showing only the best

    What it is
    With enough attempts, the best of all looks like an edge even when every one is noise. The best of a hundred noises has a beautiful curve.
    How it bit us
    We tried 345 variants of an intraday breakout rule: 0 of 345 worked out of sample. We tried 60 technical indicators on crypto: none survived the correction. In the literature: 6,402 rules, zero.
    How to check it
    Keep every variant you tried, not just the winner, and upload them together: with the full table the probability that the best one is a mirage (PBO) can be computed. Without it, declare how many you tried.
    Does the test see it?
    Yes: it is one of the checks.

  4. All the gain in a few days or in one year

    What it is
    A curve can rise a lot and owe everything to five days or to one exceptional year. That is not a repeatable edge; it is having been in the right place once.
    How it bit us
    A crypto momentum strategy had its entire gain in the ten best months of 2021. In another, the best pair went from Sharpe 0.76 to 0.04 when the best 1 % of days were removed.
    How to check it
    Remove the five best days and look again. Remove the best year and look again. If nothing is left, there was nothing.
    Does the test see it?
    Yes: it is one of the checks.

  5. Taking one year for a regime

    What it is
    One period is a sample of size one. What worked in one decade may belong to that decade, and a written warning at the bottom does not stop anyone from using it anyway.
    How it bit us
    A portfolio weighting rule passed on 2008-2026 data and failed on 1972-2006; another did the opposite. With a single era each looked like a firm conclusion.
    How to check it
    Repeat the test in another era or another market. If you only have one, say so and do not call it robust. Returns by year show whether you depend on a single one.
    Does the test see it?
    Yes: it is one of the checks.

  6. Badly built data

    What it is
    Rows that are not sessions (holidays), gaps treated as zero, candles not adjusted for splits, delisted assets valued at their last price. Each one inflates on its own.
    How it bit us
    144 holiday rows in the price database flipped the verdict of a momentum strategy: counting sessions by rows measured a different window. A 5.9 % return was false because delisted assets were valued at their last price and gaps were treated as zero.
    How to check it
    Match your dates against a market calendar. Review the valuation trade by trade for assets that disappeared or split. Look at how the series was built before computing anything on it.
    Does the test see it?
    No: nothing that looks only at the curve can see it. That is why it is on this list.

  7. Smoothed series or stale prices

    What it is
    If the series is smoothed (averages instead of closes) or updates late, returns come out too similar from one day to the next. A trend rule “sees” the move ahead of time.
    How it bit us
    We had gold as monthly averages instead of closes. A trend rule got ahead of the move without anyone having programmed it that way.
    How to check it
    Measure the correlation of each return with the previous one. If it is high, the series is not what you think. Check how it was built before computing.
    Does the test see it?
    Yes: it is one of the checks.

  8. Idle days that do not count

    What it is
    If the results file only contains days with trades, the Sharpe is computed on fewer days and without the zeros. It comes out better than it is.
    How it bit us
    A Freqtrade backtest carried 13 entries for 151 days. Filling the flat days with zero, the Sharpe went from inflated to 1.55, the same as the official figure.
    How to check it
    Count the days of the backtest calendar, not the days with activity. The days you did nothing are part of the result.
    Does the test see it?
    Yes: it is one of the checks.

  9. Costs, slippage and size

    What it is
    Commissions, spread, slippage and the real liquidity of the asset. Many strategies win on paper and lose as soon as every trade is charged or a realistic size is required.
    How it bit us
    Several of our ideas (earnings announcements, daily FX, ETF arbitrage, short-term signals) passed everything except this: no net edge for an individual after costs.
    How to check it
    Use realistic costs per trade and per side, including slippage, and ask whether the size you simulate fits the real volume. If you have not done it, say so.
    Does the test see it?
    Partly: we ask you, and your answer is recorded in the report.

  10. What trades is not what you validated

    What it is
    The live program drifts from the backtest: another version, another rounding, another data source. And a frozen feed that keeps answering “OK” while delivering stale data.
    How it bit us
    Three times what we deployed was not what we had validated. A detector ran for three weeks on old data because the source answered 200 OK; the system itself was warning and nobody looked.
    How to check it
    Compare live execution with the simulation over the same days. Watch the date of the latest data point, not just whether the source responds. If your bot and your backtest are not the same program, declare it.
    Does the test see it?
    Partly: we ask you, and your answer is recorded in the report.

What the test can and cannot see

Four of these traps leave a trace in the returns and the test measures them (3, 4, 7 and 8). The other six are invisible from the curve: the test asks you about them and keeps your answer in the report so they are not forgotten. It does not pretend to detect the undetectable.

Put your backtest through the test

Run the test   How to prepare your file

Want to hear when there is something new?

Optional. If you leave your email we will write only when NeuralTEST has news (new importers, PDF report). We store the email, the date, the language and where you came from; nothing else, and we never share it. To unsubscribe, reply “stop” to any email or write to privacy@neuraltest.app.