Assessing Forecasts

Ages ago, as this blog goes, I did a post on the pitfalls of econometrics. It’s as valid today as it was 2 years ago when I wrote it.

What brings this up is a really great post by etheorist on systemic change (and his approach to forecasting), and Greg Easterbrook’s annual Bad Predictions Review (excerpt):

…Michael Wilbon, then of The Washington Post, predicted the Steelers would not make the playoffs, as did the New York Post. Gene Collier of the Pittsburgh Post-Gazette predicted the Steelers "look like maybe a 6-win team." Adam Schein of Fox Sports predicted Pittsburgh would miss the playoffs. The Steelers made the Super Bowl.

None of USA Today's eight preseason predictions had Pittsburgh reaching the Super Bowl. Three USA Today predictors forecast Mike Singletary as coach of the year, one forecast Wade Phillips as coach of the year; both were fired.

On ESPN, Tedy Bruschi predicted a Super Bowl of Cowboys over Bengals, while Trent Dilfer forecast Cowboys over Chargers -- none made the playoffs. Chris Mortensen also had the Bengals reaching the Super Bowl, while Tom Jackson predicted the Texans would. Twelve of the 24 Super Bowl entrants forecast by various ESPN on-air predictors failed to make the playoffs. Cris Carter predicted Dwayne Bowe would be the "breakout" player of the year. Perhaps he meant to say "shutout," as Bowe had no receptions in Kansas City's home playoff loss…

…At Sports Illustrated, Banks had Tom Cable as coach of the year; he was fired. Peter King had John Fox as coach of the year; he was fired. King predicted the Chargers, Cowboys and Forty-Niners would make the postseason; none did. He predicted the Bears would miss the playoffs; they hosted the conference title game. King predicted "Carson Palmer returns with a vengeance" -- yes, but the vengeance was over Cincinnati fans -- and "Alex Smith starts 16 games." Smith was benched twice. King predicted Tampa would have the league's worst record; the Bucs finished 10-6. Best King prediction: "Carolina's my surprise team of the year and will play in January." Carolina had the league's worst record.

Greg’s one of my favourite sports commentators – that’s just a (short) sample of the full article. While it concentrates mostly on the NFL, there’s a whole bunch of stuff thrown in as well including US politics and sciences.

So what’s the point here (apart from a having a good snicker)?

Partly because the article (which is published every year) is a sendup of forecasting in general, and partly because it punctures the myth surrounding “experts”, especially does who approach their area of expertise subjectively. You might notice that EPL commentators are cut from the same cloth – how many get the winners and the scores right? As Greg points out, you’ll probably do better with a coin toss or a simple rule (and proceeds to prove it).

And that probably goes double for economics experts as well, as Greg later writes:

Night to Follow Day, Roubini Predicts

Economist Nouriel Roubini of New York University has become a celebrity, and a darling of the Davos set, for predicting, in 2005, that the housing market had become a bubble based on liars' loans. Obviously he was right. Around the same time, Roubini also predicted China would stop buying Treasury bills and runaway inflation would begin. In both cases, the opposite happened. Roubini's predictions are always negative. If you endlessly make negative predictions, occasionally you'll be correct if only by chance. This hardly makes one a seer.

What about more recent predictions? In August 2010, Roubini declared a 40 percent risk the United States was slipping back into recession. Instead fourth quarter GDP growth was a healthy 3.2 percent. In September 2010 he predicted, "Third quarter GDP growth very likely to be below 1 percent and likely to be closer to 0 percent." In reality, third quarter growth was 2.5 percent.

You’ll note that I rarely make forecasts of any kind on this blog, except for short samples ahead (like for the trade models), or for concurrent data (like for GDP). I don’t do them in my day job either, as I’m too well aware of the error rate in stats models when you get further out of sample. While that seems to leave the subjective approach as the only way forward, in truth as etheorist says:

As forecasters, we have to know what are the key parameters which give rise to the existing situation, which key parameter is likely to change first and when that happens what happens next. In the mind of a forecaster, therefore, the main task is always to assess the integrity and condition of the key parameters as well as to look out for any extraneous factors that are likely to intrude into the system which will alter the whole picture. In the end, the quality of a forecast rests on judgement, based on the experience in the assigned field as well as the breadth and depth of knowledge of the world around him or her and us.

"“Legitimate” forecasts are constructed from both objectivity (data and relationships between data) and subjectivity (judgement and knowledge). And you’d do well to maintain a healthy scepticism about any “expert” commentary…including mine.