Machine learning in trading: why the model that shines on your laptop dies live

A model that predicts tomorrow's direction with 71% accuracy on held-out data is a remarkable achievement. It is also, nine times out of ten, a bug you have not found yet.

That is not a dismissal of machine learning. ML is real, it works, and serious money uses it. But the distance between "this scores well on my laptop" and "this makes money after costs" is enormous, and almost nobody names the specific ways that gap opens. This article does.

What machine learning is genuinely good at

Start with the fair version, because straw men teach nothing. ML finds patterns too subtle or too high-dimensional to write down as rules, and given clean examples it beats people at that. It has real wins in markets: classifying regimes, forecasting short-horizon volatility, modeling order-book microstructure where samples run into the millions, execution algorithms that slice a large order. They share a common thread: big samples, stable relationships, a clear objective. And none of them is "which way will price go."

Trouble starts when the same machinery is pointed at daily direction on one instrument, with 2,000 rows and 400 features. That is not a hard ML problem. It is a problem that will nearly always return a confident wrong answer.

The market keeps changing the rules

ML assumes the future is drawn from the same distribution as the past. Statisticians call the opposite non-stationarity, and markets are the textbook case. Volatility regimes shift, decade-old correlations invert, contract specs change, and the dominant participants turn over: floor traders, then hedge funds, then high-frequency market makers, then a retail flow that did not exist in 2010.

Worse, markets adapt because people find patterns. An inefficiency big enough to notice attracts capital that grinds it down. A physical process does not care that you modeled it. A market does.

So a model trained on 2015 through 2021 is a model of one seven-year stretch, not of "the market." Whether it learned something durable is the whole question, and accuracy on a random holdout cannot answer it.

The signal is tiny and the noise is enormous

Here is arithmetic worth carrying around. Daily returns on a US index have historically averaged something near 0.03% with a standard deviation around 1.1%. The noise is roughly thirty-five times the size of the signal.

Standard error shrinks with the square root of sample size, so reaching a t-statistic of 2 on a mean that small takes about 5,400 observations, roughly 21 years of daily data, just to establish that the drift exists at all.

Now consider what you are asking a model to do: find something smaller and more conditional than that drift, in maybe 3,000 usable days, using hundreds of features. It will find something. It would find something in pure noise too, which is the problem.

In a low-signal environment, the ability to fit is a liability, not a feature.

Overfitting is not one mistake, it is a process

Everyone knows the naive version: a model that memorizes the training set. Regularization and a validation split handle that. The version that kills strategies happens one level up, at the researcher.

You run an experiment, it fails. You change a feature, try a different label horizon, add a filter, drop 2020 because it was "anomalous." Two hundred iterations later you have a beautiful equity curve. You did not overfit a model: you overfit yourself, using the test set as a search space.

The math is unkind. Test 1,000 strategies with zero real edge at a 5% threshold and about 50 will look significant. The best of 1,000 draws from a standard normal sits about 3.2 standard deviations above zero purely by selection. A Sharpe of 2.0 found after 500 trials is not the same object as a Sharpe of 2.0 from one pre-registered hypothesis, which is why researchers built deflated Sharpe ratios: the raw number is systematically inflated.

The defense is unfashionable. Count your trials, fix the hypothesis before testing it, and keep a block of data you have never looked at, because once you look, it is training data forever.

Data leakage: the reason your model looks like a genius

Leakage is information reaching the model that would not have existed at decision time. It produces spectacular results worth nothing, and it is endemic, because time series punish habits that are fine everywhere else in ML.

• Scaling before splitting. Normalize the whole dataset, then split it, and the mean you subtracted contains the future. Every row now knows something about what comes next.
• Shuffled cross-validation. Standard k-fold randomizes rows, which on a time series means training on Thursday to predict Wednesday. Most tutorials do it by default.
• Overlapping labels. If the label is "return over the next 5 days," neighboring rows share four days of outcome, so the train and validation folds hold near-duplicates. Serious workflows purge samples around the boundary and add an embargo after it.
• Revised data. Macro series get revised, sometimes heavily. Train on today's value and you trained on a number that did not exist on the day you claim to have traded it. You need the original print, which matters if you model the 8:30 AM ET release.
• Feature selection on the full sample. Ranking 400 features by full-history correlation and keeping the top 20 leaks the answer before training starts.

Useful rule: when a result surprises you in a good way, assume leakage and go hunting. You will usually find it.

The loss function has never paid a commission

A classifier optimizes cross-entropy, a regressor squared error. Neither has crossed a spread or been filled a tick worse than the screen. Yet the model decides how often you trade, and frequency is the biggest cost multiplier in this business.

Work it out. The Rentabilio backtest on this site takes 4,557 trades across more than seven years (88 months). At that count, every extra dollar of round-turn cost removes $4,557 from the result. Commissions at roughly $1 per micro contract took about 5% off the top there: $274,406 gross became ≈$260,700 net. Those are hypothetical, simulated figures, not a live account. The cost arithmetic is real either way.

Now picture a model firing every five minutes instead of once a day: on the order of 200,000 decisions over the same period, and at $3 round-turn that is $600,000 of friction against an edge measured in fractions of a tick. The model never saw the number, because it was never in the objective. If you use ML for signals, costs belong inside what is optimized: a net-of-cost objective, a minimum edge before a trade is allowed, a penalty on turnover.

Testing properly: out-of-sample and walk-forward

Both terms get used loosely. Precisely:

Out-of-sample is data the model never touched in any form: not for training, not for feature selection, not for picking a threshold, not for deciding when to stop. On a time series it must also come after the training data, since the point is to simulate not knowing the future. You get one shot: the moment it informs a decision, it is contaminated and you need fresh data.

Walk-forward is the honest way to test a system that gets refitted over time:

1. Train on a window, say 2016 through 2018.
2. Trade the next block untouched, all of 2019, and record only that.
3. Roll forward: train 2017 through 2019, trade 2020, record.
4. Repeat, then stitch the out-of-sample blocks together. That curve is your result, never the in-sample fits.

An anchored walk-forward expands the window from a fixed start; a rolling one keeps it fixed-length and drops old data, adapting faster with less to learn from. Pick one up front, not after seeing both.

Three rules make it honest rather than decorative. Every choice (features, hyperparameters, thresholds) comes from inside the training window only. Leave a gap between train and test so overlapping labels cannot bleed across. And report every block, including the ugly ones, because the spread says more than the average. The same logic applies to rule-based systems, which is why the performance page publishes the losing months.

The model nobody can read is a maintenance problem

Say everything above went right and the model is live. Six weeks in, it is down. One question: is it broken, or is this a normal bad stretch?

With rules you can inspect the answer: which conditions fired, how this drawdown compares to the worst on record. With a 400-feature ensemble, "why did it do that" has no readable answer, so every decision becomes a guess.

Retraining is not neutral either. A retrained model is a different system with no track record: each retrain resets your evidence to zero while feeling like maintenance. It is the problem from what a trading bot actually is in a harsher form: a system you cannot inspect is one you cannot judge, and you will turn it off at the worst possible moment.

So where does ML belong?

Not nowhere. Just further from the trigger than the marketing suggests.

• Diagnosis, not decisions. Clustering historical days into regimes shows which conditions a system struggled in, even if it never places an order.
• Sizing. Volatility forecasting has large samples and a stable objective, and maps onto position size.
• Filtering an existing edge. Asking a model to skip the worst days of a system that already works is a far more testable question than "predict the market."

None of that resembles "an AI that trades for you." The wider picture is in the AI trading guide, the practical use of a chatbot is in what ChatGPT is actually useful for in trading, and telling a real system from a story starts with how to spot a trading bot scam.

Frequently asked questions

Does anyone actually make money with machine learning in markets?

Yes, and the firms that do share a profile: enormous datasets, short horizons where samples are large, dedicated research infrastructure, and costs measured in fractions of a tick because they usually provide liquidity rather than take it. That is a different business from running a model on daily bars at home. The methods are legitimate; the conditions that make them pay are the hard part.

How much data do I need to train a trading model?

More than you have, almost always. What counts is not rows but independent observations of what you are predicting, and a daily strategy might have a few thousand across two decades. Higher-frequency data adds rows without proportional information, because adjacent observations are heavily correlated. Assume your effective sample is far smaller than your file size suggests.

Is walk-forward testing enough to trust a strategy?

It is necessary, not sufficient. It rules out several failure modes, but it cannot rule out that you ran a hundred walk-forwards and are showing the best one, and it says nothing about whether the next regime resembles your data. Treat it as the minimum bar for taking a result seriously, then check trade count, cost assumptions and worst-case drawdown.

Should I avoid any system that mentions AI?

No, but ask what the word is doing in the sentence. A seller who says "AI-powered" and shows no trade list, no drawdown and no way to reproduce the result is using it as decoration. One who explains what the model does, over what period, out of sample, after costs, is describing something real. The test is never how impressive the technology sounds but whether the evidence is checkable, the subject of how to choose a trading bot.