A backtest replays a rule against past data. It is a useful sanity check and a dangerous sales tool, because the more settings you try on the same history, the more likely it is that the best one simply fitted that history’s noise.
An experiment you can run
Open the Quant Lab and leave the expected return at zero. With no trend in the data, a trend-following rule has nothing real to find. Now change the moving-average window a few times on the same seed and note the best result. Then keep that window and change only the seed. The “edge” usually vanishes, because it was never there.
What disciplined testing looks like
- Decide the rule and its settings before looking at the test data, and write them down.
- Hold back data the rule has never seen and judge it only on that sample.
- Report how many variants were tried, not only the winner.
- Prefer rules that work across many seeds or periods to rules that shine on one.
The supervisory view
Model-risk guidance from the U.S. Federal Reserve (SR 11-7) asks banks to validate models independently, compare outcomes with what actually happened, and keep monitoring after deployment. The same habits protect anyone reading a backtest: assume it is optimistic until tested on data it did not choose.