The mechanic that makes a backtest valid is withholding data. Fit the model on data up to a cutoff date, generate a forecast for the period after the cutoff, then score it against the actual result. Any model can fit the past it was shown. Only a model tested against data it has never seen is telling you anything about the future.
Holdout windows and walk-forward testing
The simple version reserves the last few periods as a holdout, trains on everything before, and scores once. It is fast and it is fragile, because a single test period can be unusual and one score carries no weight.
Walk-forward testing fixes that. Train through Q1, forecast Q2, score it. Move the cutoff to Q2, retrain, forecast Q3, score it. Repeat across the full history. You end up with a series of out-of-sample forecasts made under production conditions, which supports a stable accuracy estimate and shows whether accuracy is improving or degrading over time.
Match the horizon to how you actually forecast. If the business calls a quarter on day one, backtest a day-one forecast. Scoring a model on a week-11 forecast and then deploying it on day one measures the wrong thing entirely.
Scoring the result
Report error size and error direction as separate numbers. Absolute percentage error tells you the magnitude. Mean error with the sign preserved tells you the lean, and a persistent lean is a different defect requiring a different fix.
Set the bar against a real reference. ORM reports that forecast accuracy on new and expansion revenue usually runs around 90%, produced with substantial manual effort and going stale as conditions change. ORM targets 95% and holds it without manual adjustment from day 1 through day 90 of the quarter. A backtest that clears 90% out of sample is competitive. One that clears it in sample has proven nothing.
What a backtest cannot rescue
Backtesting validates a model against history, so it inherits the assumption that the future resembles that history. ORM attributes most forecast misses to exactly this gap, where a model built on old assumptions keeps running while a competitor compresses deal size or buyer uncertainty stretches cycles. A model that backtested beautifully through a stable year will still miss the quarter the market turns.
Two practices reduce that exposure. Rebacktest on a schedule and after any structural change to territories, pricing, or segment mix, since a model validated once is validated for conditions that no longer exist. And check that the model retrains on current data instead of running on a fixed fit. ORM builds a fully trained model on a company's historical sales performance in 4 to 6 weeks, then keeps updating it as the quarter progresses, which is what keeps accuracy stable from day 1 to day 90.
Track the results as an ongoing forecast accuracy record rather than a one-time certification, and read the model's output against pipeline coverage so a drifting model gets caught by a second signal.
Frequently Asked Questions
What is walk-forward backtesting?
The model trains on data up to a cutoff, forecasts the next period, then rolls the cutoff forward one period and repeats. This mimics how the model would have been used in production, where each forecast is made knowing only what was knowable at the time. It produces many test forecasts instead of one, so the accuracy estimate is far more stable.
Which error metric should a backtest report?
Report absolute error and bias together. Mean absolute percentage error tells you how far off the model lands. Mean error, keeping the sign, tells you which direction it leans. A model that is 6% off with no lean is fixable through tighter inputs. A model that is 6% off and always low has a systematic problem worth correcting directly.
What is data leakage in a forecast backtest?
Leakage is when the model sees information that would not have existed at forecast time. The common version in revenue data is using the current value of a CRM field that has been overwritten since, such as a close date or amount edited after the deal closed. That inflates backtest accuracy and the model then underperforms in production.
How often should you rerun a backtest?
Every quarter at minimum, and immediately after any structural change such as a territory redesign, a pricing change, or a new segment. Models decay because the business moves underneath them, and a backtest run once at implementation says nothing about whether the model still holds today.
Put these metrics to work
ORM builds custom revenue forecast models that turn concepts like forecast model backtesting into prescriptive action for your team.
Schedule a Demo