The Hidden Trap in the Word 'Retrain'
When a production forecast starts drifting, the immediate reaction is often "we need to retrain the model." But as the Forecasting Data Science team at Airbnb discovered, this simple phrase masks three fundamentally different decisions: refitting, respecifying, and holding. Each has distinct costs, risks, and use cases—and choosing wrong can be expensive.
The stakes are high when other teams build their plans on your forecasts. A small bias doesn't stay small once hundreds of decisions ride on it. This is why the team developed a disciplined framework for deciding when a struggling forecast needs new data, a new model, or no changes at all. The full context of their journey through the COVID-19 pandemic is detailed in the original Airbnb Engineering post.
The Three Options: A Closer Look
1. Refitting: The Cheapest, Most Common Option
Refitting means keeping the same model structure and features but updating them with newer data. This is what most people mean by "retraining." It's the right choice for ordinary drift as the world evolves.
However, refitting isn't free. It requires validation and shipping, and if recent data is unusual, it can degrade a model that was actually fine. Every refresh carries a small risk.
2. Respecifying: The High-Reward, High-Risk Path
Respecifying changes the model itself—adding or dropping features, altering structure, priors, or likelihood. This is where real improvement lives, but it's also a significant commitment. You're replacing a model you understand with something partly new.
3. Holding: The Uncomfortable but Often Correct Choice
Holding means looking at the miss and deciding it doesn't warrant action. This takes the most nerve because it looks like ignoring a problem. But as the team found, "We decided to do nothing" is often the right call.
Three Common Failure Modes
Chasing Noise: The Off-Cycle Refit Trap
When a forecast misses for a quarter and residuals look alarming, the impulse is to refit early. But surprises are the windows you understand least. The team learned this when a large event pulled bookings forward, and an early refit caused the model to treat the unusual quarter as the new normal.
Carrying Ghosts: The Silent Structural Error
A structural assumption from a shock stays switched on long after the shock passes, hidden behind routine refits. During COVID, cancellation timing shifted, and the model learned from it. Even after market behavior normalized, the stale assumption remained until someone found it and respecified the model.
Respec-as-Panic: Overcorrecting to Noise
When something genuinely moves—like an FX swing—the reflex is to rebuild. But the team found that widening priors instead of rebuilding allowed the existing model to ride out the shock, avoiding a move to a more fragile model.
The Decision Framework: What Changed?
The team now uses a simple rule: the answer depends on what changed.
- Refit if the process generating data is still what the model assumes, but parameters have drifted.
- Respecify if the process has changed in a way the model cannot represent. The tell is direction: a misspecified model misses the same way repeatedly.
- Hold if the miss sits within the range the model already calls normal. This usually means declining to refit off-cycle.
The COVID Case Study: When Refit Failed
Before 2020, the team's forecasts used a hierarchy where markets with long histories anchored estimates for thinner markets. COVID broke this assumption. Markets that used to move together diverged, and the borrowing structure began contaminating forecasts.
Refitting failed spectacularly—error ran about three times the pre-shock baseline. The instability was the signal: a structural problem, not a parameter problem. The fix was respecification: changing what the model borrows across, using geographic adjacency and shared recovery dynamics instead of a fixed hierarchy. This cut error by half on held-out data.
The key insight? The framework would have told them not to refit, saving a quarter of accuracy that downstream teams felt.
Why This Is Hard in Practice
Holding Looks Like Negligence
When a watched forecast misses, "we're choosing not to act" is unpopular. The pressure is toward visible activity. But as the team's maxim says: "Don't just do something; sit there."
Respecifying Is Expensive and Frightening
Replacing a model that has worked for years means giving up something you understand for something you don't. This fear is healthy but leads to under-respecification and accumulating ghosts.
The Pandemic Trained the Wrong Reflex
Forecasting through COVID meant changing models under pressure. Coming out the other side, teams kept reaching for rebuilds when refits would do, while leaving pandemic-era assumptions switched on.
Limitations and Cautionary Notes
This framework assumes you have the luxury of time to analyze a miss. In truly novel situations, the "hold" option may be riskier than it appears. Additionally, the team's approach relies on having deep domain expertise to distinguish parameter drift from structural change—something not every team has. The distinction between "refit" and "respecify" can blur in models with complex feature engineering.
Next Steps for Your Team
- Audit your current retraining cadence: Are you refitting on a schedule without questioning whether it's the right move?
- Document your "ghosts": List any structural assumptions from past shocks that might still be active.
- Build a decision log: Record every refit, respecify, or hold decision and its outcome to build institutional knowledge.
Conclusion
The discipline of forecasting isn't just about learning from shocks—it's about learning to forget them, on purpose, when needed. Refitting keeps a model current; it doesn't keep it honest. The assumptions underneath the numbers are what go stale, and those are exactly what a refit leaves alone.
For a deeper dive into the broader development landscape and how teams are adapting to new technologies, check out our analysis of web development trends beyond frameworks. And if you're curious about how the developer community is evolving, our coverage of the Python community's open source shift offers valuable perspective.
