When Good Models Go Bad

What do you do when your trading system goes wrong?

Two possibilities:

1. Has the world changed? Do you still have an edge?
2. Or is this a "blip" that is within the expected error tolerance?

If you are an experienced researcher, you can employ various statistical tests, determining whether or not the actual results are plausible. Most actual traders have much less patience!

A Super Bowl Example

Regular readers know that I am a big sports fan and I also love the statistical analysis of sporting events. Looking at these examples is good for investors since it takes them away from pre-conceived market opinions and into a different universe. So let us discuss the Super Bowl.

John Dewan's Stat of the Week provides a consistently strong stream about baseball, my favorite sport. I have enjoyed the regular posts, helping to illustrate the strengths and weaknesses of both teams and players. If you share my love of baseball, you should subscribe to this weekly update.

But this is about football. John has a Super Bowl system that worked 90% of the time! Until the last seven years, where it went 2-5. He is reporting the forecast, but retiring the system – and not using it for his official pick.

The key point? He does not know exactly what went wrong, but a 90% system does not go 2-5. It was time to move on.

Implications for Investing

Today's GDP report underscores the error of those who have predicted recessions and economic collapse. The investment world is replete with broken models – forecasts that have been sadly wrong. I am going to split these sources into two broad camps, based upon the financial rewards of those providing the information. If the investor had to pick one thing to follow, that would be a good choice.

Profit from Product Sales

This group includes those who do not focus on results. They sell conspiracy theories, seminars and conferences to confirm your biases, bonds, research that is mostly for bond clients, and page views.

Profit from Results

I want to focus on this group. Their success depends upon helping readers and clients.

For over a year I have had a post prepared with the title "RIP ECRI Recession Forecast." I have been waiting for the intelligent and engaging people at the ECRI to re-evaluate their methods. To everyone except them, it is obvious that something went wrong with their model. Those of us who follow their work closely can even make an educated guess, having reverse-engineered the components. (They use too many correlated market variables and not enough fundamentals. They wanted to be fast, avoiding revisions. More variables does not equal better. When commodity traders embraced the inflation thesis after the start of QE, it drove those prices really high in 2011. When the inflation did not develop, the commodity prices collapsed. To the ECRI model this looked like death. That was a big mistake. They needed a good consultant who understood the fundamentals). Whether you agree with my guess or not, their model is obviously broken. They should choose to say, "Something went wrong, we tweaked, time to move on…" and we would all have accepted that. Instead there is a stubborn adherence to a mistake.

I have also been stalling on some posts related to the famous fund manager whose methods always seem to start with the conclusion and then identify the "independent" variables. He is personally engaging and generous. He would seem to be aligned with clients, since his performances is on the line. (Or maybe I am wrong. He is so good at marketing. Maybe people are happy to invest with a fund that reflects their market viewpoint, regardless of performance). The problem is his methodology. He is overdue to bring in an outside consultant who will critique his work instead of relying on employees who provide research to order for the company line. I was offered a job like that once, and I turned it down.

Investment Conclusion

Any experienced model developer has both successes and failures. Sometimes the reasons are technical. On other occasions the market catches up with your approach. The serious modeler goes to work, analyzing and fixing if necessary.

If you have been paying attention, you have noticed a world littered with broken methods – omens, seasonal forecasts, cycles, and predictions for "mean reversion." Those with broken models should have had a New Year's resolution to re-examine and fix. If someone is no longer working hard to improve his methods, maybe we should not be paying any attention.