Overfit check: the search verdict

Every run you make on the same data is a try. Run 35 variants of one idea on one window and the best of them looks good partly by luck, even if each one alone grades B: the honesty grade judges one run's mechanics, not the search around it. So every finished backtest also carries a search verdict.

The verdicts

Verdict	Means	Do
ok	One run, or the deflated Sharpe probability is at least 0.95	Go on; still one backtest is in-sample
caution	The deflated Sharpe probability is below 0.95	One research run instead of more sweeps
likely_overfit	Below 0.5: this run is likely luck of the search	Run the suggested research (or test once on a holdout)
no_edge	The run lost money (annualised Sharpe ≤ 0, or a negative total return), whatever the number of variants	Change the idea (entries, exits, filters, market or timeframe), not its parameters

Report it together with the grade, for example: "grade B, 35 variants tried on this window, likely overfit".

How it is computed

• Variants counted: per account (every API key and the app together), over the last 30 days, on windows that overlap this run's, on a shared symbol and the same timeframe. A variant is a run of the same code (other parameters are the same strategy), edited code that shares a parameter name, or the same code with other numbers written into it. A different idea (different code) is not counted. The same spec twice counts once, and a run on a window none of the variants touched starts fresh.
• Luck: expected_max_sharpe_by_luck is what the best of that many worthless variants would reach (Bailey & López de Prado), from how much the variants' Sharpe ratios differ. deflated_sharpe_prob is the probability that this run's true Sharpe beats that luck. Sharpe ratios are annualised.
• When the verdict is caution or likely_overfit, or 10 or more variants ran on this exact window, the result also suggests a ready-made research run and a holdout split (a window to tune on, and a later one to test on once).

How to avoid overfitting

• Don't sweep parameters with many single runs. Use one walk-forward research run: it optimises and validates on test folds the optimiser never saw.
• Or keep a holdout window you never tuned on. Tune on data before the last months, then run the chosen parameters once on the rest and report that run.
• Group one idea's runs into a study (tag study:<name>). A study's verdict counts every variant you put in it, on any window, using the same scale as above.
• Send at most 3 runs at once and read their results before the next batch.

The holdout vault

A study can declare a holdout before tuning. While it is unused, OptAlgo refuses the study's runs over that window, then runs your chosen backtest once on it as the final test. The result puts it next to the in-sample run:

Verdict	Means
holds	The holdout makes money and keeps at least half of the in-sample Sharpe
degraded	It makes money, with less than half the in-sample Sharpe
failed	It lost money on data it never saw: the in-sample result does not hold

Once done, the holdout is spent. A second final test on the same holdout is refused: that would be tuning on it.

Robustness check

Before deploying, a robustness check runs the backtest's own code under four pressures: the trades resampled 1,000 times (bootstrap), fees and slippage doubled, each parameter moved ±10 %, and the window started 5, 10 and 15 % later. The verdict is robust, fragile or broken. robust is still in-sample: score the run once on its study's holdout before trusting it. broken: do not deploy.

The deploy gate

Running a strategy on OptAlgo is refused for a backtest whose search verdict is likely_overfit, unless an out-of-sample research run of the same code passed with the verdict robust. A losing backtest (no_edge) is always refused.

In a chat app with the OptAlgo connector, the agent reports this verdict for you. The full field-by-field reference is in the LLM reference.