Your forecast model looks great. It uses advanced statistical techniques to extract insights from years of data. And, based on what you have experienced, the outcome seems to be right in line with what you thought it should be. It's completely plausible — and that's a problem. Your model quietly assumes that the future you are forecasting will work like the world that produced its underlying data. Customers will behave the same. Competitors will do what they have done. And, of course, the economy will keep cranking along pretty much as it has.
The trouble is that these assumptions are highly unlikely in anything but the very near term. The pace of change in our world is constantly accelerating. So, structural breaks — those times when the past stops predicting the future — are happening more frequently, more suddenly, and with less warning.
In The Black Swan, Nassim Taleb used a turkey to illustrate this idea. If a turkey were to chronicle its life and then use that data to model the relationship between itself and a farmer, it would forecast a continuing wonderful benevolent friendship. For a thousand days the turkey would record being fed by its benefactor. The data would be an analyst's dream — consistent, long history, strengthening trend. The model's confidence in a bright future would be at an all-time high.
Of course, you know where this is going: Thanksgiving.
The interesting thing here, though, is that the model was not inaccurate. In fact, it is exactly the opposite. The model was extremely accurate — up until the moment it actually mattered. And no amount of data and no statistical analysis on that data would have made the turkey any safer. It would have just made it more confidently and tragically wrong.
A version of this problem shows up in just about every significant business decision. Any time you're doing something meaningfully different — entering a new market, changing pricing, launching a product — you're asking a model of the past to describe a future it has no information about.
Historical data can tell you a lot about how the world worked. But a decision is a bet on how the world will work. Those are completely different concepts, and the gap between them is widening. The missing ingredient usually isn't more data. It's the forward-looking knowledge that already exists in someone's head.
The real question you need to ask of a model isn't what will happen. It's what would cause it to happen.
← Back to Writing