The Pitfalls of Econometrics

de minimis has an interesting post advocating using more econometric models to guide policy making in Malaysia. While I tend to lean that way myself (there's too much unsubstantiated rhetoric flung around the news and blogosphere for my taste), I don't want to be blind to the potential pitfalls and shortcomings of an applied econometric approach to policy. So this post is both to clarify some of the issues, as well as serve as a reminder to myself not to be too "assertive", as my wife puts it.

First is that econometric modeling (as etheorist remarked the other day) is really an art, not a science. There are many, many ways of looking at an economy and generating forecasts, from simple time series techniques to hideously complex dynamic general equilibrium models. So model choice and specification (as well as accompanying underlying assumptions), and not to mention the ideological bent of the modelers, can lead to very different conclusions about the state of the economy at any given time. The issue is compounded by Malaysia being such an open economy, which means that ideally, you'd have to incorporate all the major trade partners into your model as well.

Secondly, the evolving economic structure within a developing country means that even if you do come up with a model close to reality at some point in time, it might be out of date very quickly later on and you won’t know it until something goes wrong. This is one point where I would be critical of DOS: the Malaysian input-output tables haven’t been updated in years, and you need this to model intra-industry dynamics.

Thirdly, any econometric model necessarily uses historical data, which means there will always be an unobservable error component in any forecast in the presence of a current shock. A corollary of this is that, almost by definition, a trade shock such as we just suffered cannot be predicted on the basis of concurrent data. Models are more useful as a predictive guide to inventory driven recessions and business cycle downturns. You can of course use models to predict what happens when a shock occurs, but not when or if a shock will occur.

Fourth, data accuracy is inversely proportionate to the speed at which data is published. In other words, the faster you publish it, the larger the error rate. Where I think DOS can improve on that score is to follow the general practice in the EU, US and yes, Singapore, i.e. issue advanced, preliminary, and final estimates of major statistical series. The current practice of a 6-week to 8-week lag and quietly revising the historical series, isn't transparent (the loose hair around my workplace is testament to that). Data revisions should always be made clear, especially for national accounts data, which has to be revised even 2-3 years down the road.

One exception to this observation is financial sector data, which is available very quickly. (Side note: I've visited BNM to study their data gathering process, and I was a member of one of the teams responsible for implementing CCRIS in one of our banks - I am very impressed with BNM's operation in this instance. The disaggregated trial balance of the entire banking system is available at about t+4 after every month end – in other words, don’t be fooled by the monthly publishing schedule). (Side note to the side note: this is one reason why monetary policy is generally the first recourse in any crisis – you have better data much faster than real economy data).

However, I should point out that a 2-month to 3-month lag hits the sweet spot between accuracy and timeliness, and is fairly typical worldwide. China for instance tends to issue data on a 1 month lag, but subsequent revisions tend to be very large. Some Canadian series have no revisions at all, but you have to wait 6 months(!) to get them. I cannot fault DOS on that score, though they have made some absolute boo-boos before (pay attention to 2004-2005 trade data before and after revision, for instance).

Fourth: some of the most critical variables required for a predictive model are unobservable. For example, consumer and investor expectations have a big impact on private consumption and investment, but can’t be quantified. It’s possible to use proxies, such as consumer confidence or business expectations surveys, but these are subject to error as well.

Take all the factors above, and you shouldn’t be surprised that most whole economy econometric models have very little predictive power more than a quarter or two ahead, out of sample. I’d note that I’d be very surprised if the government doesn’t have on hand some whole economy econometric models, especially for trade and tax policies. Could more use be made of modeling? Absolutely! Just don’t fall for the promise that they’ll be a panacea and perfect guide to policy.