Somebody sends you an equity curve. It goes up and to the right at the same angle every month for four years. Barely a dip. A 94% win rate. The seller calls it consistency.
It is the opposite. That curve is the most reliable sign a system has been fitted to its own history and will fall apart the first week it meets a market it has not seen. Real edges are lumpy. A straight line means somebody turned knobs until the past looked perfect, and the past is the one thing you will never trade.
What curve fitting actually is
Every price history contains two things mixed together: signal, behavior that recurs because it comes from how markets and participants work, and noise, accidents of that sequence that will never repeat.
Curve fitting (overfitting, under a more technical name) is what happens when a system learns the noise. Add enough rules and tuned parameters and you can describe any past sequence perfectly. But a description of one sequence is not a prediction of the next. The system memorized rather than understood, and memory is worth nothing in a market that never repeats exactly.
The uncomfortable part is that curve fitting rarely looks like cheating from the inside. It looks like diligence. You test, the result is mediocre, you adjust, it improves, you adjust again. Every step feels like progress. What you are doing is walking the system deeper into the accidents of one dataset.
The one thing to remember
A backtest is not evidence that a system works. It is evidence of what a set of rules would have done on one particular stretch of history. Whether that carries forward depends entirely on how the rules were arrived at. That is a question about process, not about the size of the number at the bottom.
How optimization manufactures it
Here is the mechanism, in numbers. Take a system with six adjustable parameters: entry threshold, stop distance, target multiple, filter length, time window, volatility cutoff. Test each at twelve values. That is 12 to the sixth power, or 2,985,984 combinations. Most platforms run that sweep overnight and hand you the best one.
Now consider what "the best of three million" means. Feed the same optimizer a series of coin flips with no edge at all and it still returns a top result, and it looks magnificent. It has to: with three million draws, the luckiest is extreme by construction. The optimizer cannot tell you it found nothing. Only which combination was luckiest.
Statisticians call this the multiple-comparisons problem, and the retail version is stark: test a thousand variations at a 5% significance level and about fifty look significant purely by chance. The winner's reported performance is not its edge. It is its edge plus whatever luck it took to beat three million rivals.
An optimizer never says "there is nothing here." It says "this one was luckiest."
Degrees of freedom against sample size
The practical defense is a ratio. Every free parameter is a degree of freedom, a knob that lets the system bend toward the data. Every trade is evidence pushing back. What matters is which side has more.
Five parameters and 120 backtest trades is roughly 24 trades per parameter: not enough evidence to fix five knobs, and whatever comes out is mostly fitted. The same five with 4,000 trades gives 800 each, a completely different situation. Rules of thumb vary (some want 30 to 50 trades per parameter as a floor, others insist on hundreds), but the direction is not controversial. Fewer knobs and more trades beats more knobs and fewer trades, and a system needing many parameters usually does not work.
The other test costs nothing. Take the winning parameter set and nudge each value one step in each direction. If a 20-tick stop produces a great result while 19 and 21 produce mediocre ones, you found a spike on the performance surface. That is an accident. A genuine edge sits on a plateau, where neighboring values also work, a little worse. Ask a seller what happens when their parameters move 10%. The answer, or the refusal, is informative.
Why a smooth equity curve is a warning sign
Everyone wants the straight line. The line is the problem.
Real edges are small statistical advantages applied repeatedly, and small advantages produce noisy results in the short run. A system that wins 46% of its trades must have losing streaks, weeks and months. That is arithmetic, not sentiment, and the math is in risk-reward and win rate. When a curve has no losing months across four years, one of three things is true: the sample is too short to have met a bad regime, losses sit in positions never closed, or the system was fitted until the dips disappeared.
So invert the usual reading. A published maximum drawdown of $4,379 on a $50,000 account, alongside visible losing months across 88 months, is not a blemish on Rentabilio's report. It is what makes the rest plausible. A report with no rough patches has been sanded. What that figure means in practice is in what drawdown is.
Hypothetical performance. Every Rentabilio figure here comes from a backtest on a $50,000 account, not a live account. Simulated results are prepared with hindsight, carry no financial risk, and cannot fully reflect real execution or slippage. Past performance is not indicative of future results.
In-sample, out-of-sample, and the mistake almost everyone makes
The standard discipline is to split the history. You develop on the first portion (the in-sample data), then, once and only once, run the finished system on a portion you have never looked at: the out-of-sample data. Excellent in-sample and mediocre out-of-sample means you fitted. Comparable results mean you may have something. That comparison is the most informative number in system development, and it is almost never published.
Now the mistake, subtle and nearly universal. You run the out-of-sample test, the result disappoints, so you adjust the system and run it again. The moment you do, the out-of-sample data has become in-sample data. You used it to decide. Every later test on it is contaminated and no relabeling fixes that. Out-of-sample data is spent the first time you look at it, which is why serious developers hold back more than one block.
Walk-forward analysis
Walk-forward is the honest version of optimization. It exists because a single split only tests one moment of transition.
The procedure is mechanical. Optimize the parameters on a window of history, say two years. Trade the next six months with those settings frozen and record the result. Roll forward six months, re-optimize, trade, record. Repeat across the dataset, then stitch together only the out-of-sample stretches. That stitched curve is your estimate of live behavior, and it is usually far uglier than the optimized backtest.
The comparison has a name, walk-forward efficiency: out-of-sample performance divided by in-sample performance. Around 50% or better suggests the edge survives re-optimization. Well under that means you were measuring how well the optimizer fits noise. It is also the honest way to answer "how often should parameters be updated," which otherwise gets settled by whoever is most confident.
The point of a reproducible backtest
Every figure can be regenerated in the Strategy Analyzer inside your own NinjaTrader 8, over the same period, with the same settings. A number you can rerun is a different kind of claim.
The other ways a backtest lies
Overfitting gets the attention, but a clean, un-optimized system can still produce a fantasy curve through modeling error.
- Lookahead bias. The system uses information it could not have had at that instant: deciding an entry from the bar's closing price, or a data revision published later. That produces curves which are not optimistic but impossible, and it is the most common serious bug in homemade systems.
- Fill assumptions. Backtests fill limits because price touched the level and fill stops at exactly the stop price. Live markets do neither reliably, and it matters most in fast conditions, precisely when a system trading one window a day at 8:30 AM ET is working.
- Costs left out. Rentabilio's figures show gross at $274,406 and net at ≈$260,700, about 5% off at roughly $1 per micro contract. A gross-only report hides a real number, and the higher the frequency the more it hides.
- Extrapolating a hot stretch. The system's record shows $43,322 gross in the last 7 months, roughly double its historical monthly average. That means good stretches happen, not that the average moved. Treating the best seven months as a new baseline is curve fitting applied to your expectations.
Tells a buyer can check without being a quant
You do not need a walk-forward study to filter out most of what is for sale. Eight questions and ten minutes will do it.
No losing monthsAcross years of data this is not excellence. It is a short sample, a hidden open position, or a fitted curve.
High win rate, no ratioOver 80% with the average win and loss unpublished usually means a large loss financed by many small wins.
Too few tradesUnder a few hundred trades, a good result is indistinguishable from luck. Ask the trade count before the profit.
A short windowTwo or three years covers one regime. Ask why the test starts where it starts, and what happens if it starts earlier.
Gross numbers onlyWithout commissions subtracted, the headline is not a result. It is an upper bound nobody could have collected.
Unmovable parametersIf the settings cannot be varied by 10% to show a plateau, you cannot tell an edge from a spike, and neither can the seller.
No out-of-sampleAsk which portion of history built the system and which was held back. A blank stare is the answer.
Nothing you can runScreenshots are not reproducible. A backtest you can regenerate in a standard platform is a different category of claim.
Run those against anything, this site included. The same list drives how to choose a trading bot and how to spot a trading bot scam in five minutes.
What a long backtest does and does not prove
Rentabilio's published run covers more than seven years (88 months, 4,557 trades) with a 46.2% win rate, a 1.58 profit factor, visible losing months and a stated maximum drawdown. Each property makes fitting harder to hide: thousands of trades constrain the parameters, a sub-50% win rate is not what a fitted system produces, and a published drawdown is what a fitter would have optimized away.
None of that is proof and it should not be sold as proof. A long backtest with many trades is harder to fit, not impossible. It remains a simulation over historical data, performed with hindsight, with no money at risk. The one thing that upgrades it from a claim is that you can run it yourself in the Strategy Analyzer, over the same period, and see whether the numbers land where they were said to. That is what the performance page is for, and the whole argument of what a trading bot actually is.
Frequently asked questions
Does optimizing a strategy always mean it is overfitted?
No. Some optimization is unavoidable, since every system has parameters that need values. What separates reasonable tuning from fitting is how many parameters are adjusted, how much data supports each, whether the settings sit on a plateau rather than a spike, and whether the result was validated on data that played no part in the tuning. Optimization without those controls is where trouble starts.
How many trades does a backtest need to be meaningful?
Enough that a normal losing streak cannot dominate the result, which for most systems means hundreds at minimum and thousands preferably. The other half is the ratio of trades to adjustable parameters: a hundred trades cannot support six free parameters however impressive the curve. Ask for both figures together; either alone can be made to look reassuring.
What is walk-forward analysis in plain terms?
You optimize a system on a block of history, trade the block that follows with settings frozen, record only that result, then roll the window forward and repeat. Stitching together only the untouched stretches estimates how the system behaves on data it has not seen. More work than a single backtest, and considerably more honest.
Can I check for overfitting without technical skills?
Largely, yes. Ask for the trade count, the number of adjustable parameters, whether any data was held back, what happens when the parameters move slightly, and whether commissions were subtracted. Then ask whether you can reproduce the run in a standard platform. None of that requires statistics, and most overfitted systems fail two of those questions within minutes.
In short: a backtest shows what rules would have done on one stretch of history, and an optimizer always hands you a winner even when there is nothing there. Judge the process, not the number: how many parameters against how many trades, whether the settings sit on a plateau or a spike, whether data was genuinely held back, whether costs were subtracted, and whether you can rerun it yourself. A smooth curve with no losing months is not the good version. It is the tell.
