AI Forex Backtesting: Data Leakage, Overfitting and Honest Validation
An AI forex backtest can look convincing while answering the wrong question. It may show how well a model describes prices it has indirectly seen, how cheaply trades would work without realistic execution costs, or which of many experiments happened to fit one period. None of those results establishes a usable trading advantage.
A more useful test asks whether a frozen decision process could have acted on information available at the time and produced worthwhile results after costs. This guide shows how to construct that test, check its weak points and use AI to help with the research. The examples are hypothetical; no predictive model or profitable strategy has been validated for this article.

In this guide
Start with one prediction and one trading decision
Separate the AI assistant from the model being tested. An assistant might help write a data-cleaning script or explain a calculation. A predictive model is a particular fitted object that turns specified inputs into an output. Asking an assistant to describe yesterday's chart does not produce the same evidence as saving a forecast before the next price arrives.
Write a short experiment contract. For example: use completed five-minute GBP/USD bars to predict the direction of the following 20-minute mid-price change. Generate a score at each eligible decision time, but enter only when a predetermined threshold is met and no position is open. Specify the executable entry quote, exit rule, sizing and transaction costs separately. A directional label is not itself a trading return.
Record the data provider, symbol, timezone, date range, missing-row policy and model version. Define an up move, a down move and an unchanged price explicitly. If you need help turning a trading idea into rules, begin with the AI-assisted trading-plan workflow.
Check when every input becomes available
Data leakage means allowing information unavailable at prediction time to influence the model. The scikit-learn guide to common pitfalls explains why this produces overly optimistic evaluations. In trading, a timestamp attached to a row is insufficient: you need to know when the value in that row was observable.
| Item | Timing | Treatment |
|---|---|---|
| Completed bar | 10:00–10:05, labelled by its 10:00 opening time | Its final high, low and close are available only after completion. |
| Decision | Just after 10:05 | Use completed inputs and allow processing time. |
| Entry | First eligible quote after the decision | Do not assume the model can fill at an earlier closing price. |
| Prediction target | Price change from 10:05 to 10:25 | The ending price belongs to the future label, never the input features. |
A feature containing the day's eventual high cannot guide a morning trade. A centered moving average can include later observations. An economic-data column may contain a revised number that differed from the original release. Inspect these issues at the source; renaming a column “historical” does not resolve them.
Have the assistant produce a feature ledger with the formula, observation time and availability time for every input. Manually reconstruct a few rows around a session boundary and a missing-data interval. The official lagged-feature forecasting example demonstrates past-only features and the problem of evaluating time series with a shuffled split; its dataset is bike demand, not evidence of forex profitability.
Fit data preparation inside the training window
Split the record before fitting a scaler, missing-value imputer, feature selector or model. A scaler trained on the full dataset has already incorporated the later distribution, even if the prediction model sees only earlier rows. Fit each learned transformation on the training portion, then apply that fitted transformation to the later portion.
A scikit-learn pipeline can keep these fitted steps together during cross-validation. It cannot repair a feature that was constructed from future prices before entering the pipeline. A past-only rolling calculation and a globally fitted transformation therefore need different checks.
Retain the raw file and record each cleaning decision. Duplicate timestamps, silently filled weekends and missing spreads can change the opportunities your strategy appears to receive. A useful audit reports how many rows were removed and why, rather than presenting only the clean final sample.
Split forward in time, then inspect overlapping labels
For an initial design, reserve an early training block, a later development block and a final untouched block. Within development, you can use several expanding or rolling training windows, each followed by a later validation window. Keep the final block outside all choices about features, thresholds, models and costs.
TimeSeriesSplit supplies chronological indices and a gap that excludes observations from the end of training. Its comparable-duration interpretation assumes equally spaced samples. A missing weekend or irregular trade-event dataset means that “four rows” may represent a different elapsed time; inspect timestamps explicitly.
Now consider the 20-minute target above. Suppose the first validation decision is at 11:00. A training row with a 10:50 decision uses a target ending at 11:10. The row starts before the boundary, yet its outcome would not be known when the 11:00 prediction is made.
For this example, conservatively exclude training rows whose label end is at or after 11:00. With uninterrupted five-minute decisions, that removes 10:40, 10:45, 10:50 and 10:55; the 10:35 label ends at 10:55. Four rows happen to work here. Variable holding periods, delayed publications and other label definitions need their own timestamp checks, not a universal four-row setting.
Keep fitting and prediction events in their actual order. A model refitted at 11:00 must use only outcomes already available by that cutoff. If its training takes time, the first executable prediction must also account for that delay.
Count unsuccessful experiments as part of the evidence
AI makes it easy to request another indicator, threshold or model after a disappointing result. That speed can turn development into an extensive search without an experiment log. The winning chart then hides the number of opportunities you gave yourself to find a lucky fit.
Bailey and colleagues' research on the probability of backtest overfitting examines this selection problem and the limitations of a simple holdout. Keep a register of every candidate, including rejected versions. Record the question, change, validation result and reason for continuing before requesting another modification.
Limit the initial search to a small, stated set of plausible alternatives. Compare the AI model with a simple baseline under identical execution and sizing assumptions. A complicated model needs to earn its additional maintenance burden through evidence; complexity does not increase the value of a result by itself.
After you inspect the final test, do not adjust a threshold and continue calling that period untouched. It has become development evidence. Save the failure or mixed result, identify what changed, and wait for genuinely new observations for the next independent check.
Turn forecast accuracy into net trade outcomes
Classification accuracy ignores the size of wins and losses, trading frequency and costs. A forecast may be right about direction but arrive too late, predict too small a move or require more trading than the available spread allows.
Consider 100 hypothetical trades at equal size. There are 55 winners averaging four pips before costs and 45 losers averaging four pips. Gross performance is 55 × 4 − 45 × 4 = 40 pips. With an assumed total round-trip cost of 1.2 pips per trade, costs are 120 pips and the net result is −80 pips, or −0.8 pips per trade. The 55% win rate has not produced a profitable strategy.
In this illustration, the gross figures exclude every cost. If a simulator already enters at Ask and exits at Bid, spread is embedded in its trade result; do not subtract it again. Add commission, financing and other charges only where they are not already included. Use account-currency results when trade sizes or pip values vary.
The strategy profitability audit explains that reconciliation. The trading cost calculator can help explore assumptions, but its inputs still need to match your account and symbol. Repeat the analysis under plausible wider spreads and worse fills; avoid treating one favourable cost estimate as a fact.
Review the trade record behind the headline score
Preserve each decision's input timestamp, prediction, eligibility checks, simulated entry and exit, position size and net outcome. Include skipped signals. Reconcile the headline result to that ledger before asking AI to interpret the chart.
Review trade count, exposure, turnover, average net outcome, drawdown and concentration by time period. Check whether one week or a few unusually large winners supply most of the result. More bars do not automatically mean more independent evidence: overlapping 20-minute predictions share price moves, and simultaneous positions may share the same currency exposure.
Describe uncertainty without inventing a confidence score. If you commission statistical intervals, ask how the method handles dependent observations and model selection. A narrow-looking interval based on treating every five-minute label as independent can create misleading precision.
Also distinguish historical testing of a fitted model from asking a general assistant to predict an old event. Without knowing what historical material the assistant encountered, you cannot establish that its answer was independent of the event's outcome. Use it to inspect your process, then collect timestamped prospective predictions to investigate the remaining question.
A prompt that asks AI to challenge the experiment
Supply a small anonymized sample, the column dictionary, rule sheet and actual code. Ask for findings tied to specific rows or functions before requesting improvements.
Audit this forex prediction experiment. Do not invent missing data or performance. For every feature, list observation time, availability time and formula. Check that only completed information reaches each prediction. Trace every label start and end across the chronological boundaries. Check when scalers, imputers, feature selection and models are fitted. List all threshold and model choices made using validation or test results. Reconstruct three sample trades from decision to executable entry and exit. Identify costs already embedded in fills and any additional charges. Check duplicate signals, overlapping exposure and missing timestamps. Return: issue, exact evidence, effect on the result, and a proposed check. Separate confirmed errors from questions requiring more information. Do not rewrite the strategy or claim it is profitable.
Carry the frozen version into prospective observation
If you reproduce the strategy in MT4, document the tester's assumptions. MetaQuotes' MT4 testing setup explains that stored history contains Bid prices and that the tester models Ask using the selected spread setting. That setting is not automatically a reconstruction of every historical spread.
Keep the data, fitted model, code, inputs and results together under one version. Then use the forward-testing workflow to record new predictions and demo decisions without retrospective editing. The next useful milestone is a reproducible record of what the model knew and did. Profitability remains a question to investigate from that evidence.
Sources and verification
Primary documentation checked on 13 October 2026: scikit-learn: leakage and fitted preprocessing; TimeSeriesSplit: ordering and gaps; lagged-feature forecasting example; Bailey et al.: backtest overfitting; and MetaQuotes: MT4 tester setup. The trading contract, boundary example, cost calculation and audit prompt are original hypothetical teaching examples.
No comments