Looking For Insight In Flawed Predictions

All forecasts are wrong, but some are useful, the mathematician George E. P. Box famously advised. But how can wrong forecasts be useful? One way is to use our best guesstimates as early warning sign of a major turning point in the trend. This is one of the more compelling reasons to generate forecasts. It’s a subtle point, and one that’s too easily overlooked in the rush to dismiss forecasts as garbage. In truth, you can learn a lot from flawed forecasts. I raise this point now because we’ve seen a run of disappointing economic updates in recent weeks (personal income, payrolls and retail sales). The value of analyzing flawed predictions may be unusually valuable these days.

As an example of how this works, imagine that we’ve been generating forecasts for some period of time and that our incorrect forecasts have been wrong in a random way relative to the actual data that’s subsequently published. Sometimes the forecasts are right, sometimes they’re wrong, and sometimes a prediction may even be exactly correct. Over the span of the forecasting period, however, it’s clear that the errors bounce around the actual number in random fashion. That is, the forecasts vary with no predictable pattern. That’s a good thing because that’s a sign that we’ve built a robust prediction model. (Actually, we’d also like to see that the forecasts vary in a relatively narrow band around the actual data. In statistical language, we’re looking for a low root mean square error, which tells us that the predictions do a reasonably good job of anticipating the actual data.)

Let’s say that you’ve been forecasting an economic indicator and the errors have varied randomly (positively and negatively) for, say, the past three years. But now there’s a change: the actual data begins to routinely fall below the forecasts. The prediction errors are no longer randomly dispersed around the actual data. The change may be a sign that the model’s no longer valid and that it’s time to revise. But if we rule out that possibility, the change may signal that a meaningful shift in the trend is in progress.

With that in mind, let’s see how today’s industrial production report compares with the forecasts. If we again see data that falls well short of the predictions, we’ll have another reason to wonder if there’s trouble brewing for the business cycle. You can’t tell much from any one release. But a pattern of disappointing numbers can be a warning sign. The bottom line: Wrong forecasts that turn wrong in a predictable way may be telling us something quite valuable for analyzing the macro trend.