Quoting David Beckworth quoting Barry Ritholtz (excerpt):
I recently made the case that many observers are not thinking properly about the Fed's Quantitative Easing (QE) programs. Using the analogy of George Bailey's life in the film It's a Wonderful Life, I argued that the critics who question the efficacy of the QE programs are doing the wrong counterfactual. Today, Barry Ritholtz makes the same point:
One of the analytical errors I seem to constantly come across is what I call the non-result result. It goes something like this:If you do X, and there is no measurable change, X is therefore ineffective.
The problem with this analysis is the lack of a control group, If you are testing a new medication to reduce tumors, you want to see what happened to the group that did not get the tested therapy. Perhaps their tumors grew and metastasized. Hence, no increase in tumor mass or spreading is considered a very positive outcome.
This seems to get loss in the debate over QE. The debate — either ignorantly or disingenuously — makes claims such as “Look how few jobs have been created, and look how high unemployment is.”
Understanding this logic, and lacking a control group, we must employ a counter-factual. The question one should be asking is “How many less jobs would have been created?; How much higher would unemployment be?”
I made the exact same point a year and a half ago:
...Lots of things can have an impact on any given target variable, and you can’t assume that whatever changes occurred are purely from the effect of a particular policy. Improvements or a deterioration in the situation targeted by that policy could have come about from completely unrelated sources...
…In a nutshell, figuring out whether a particular public policy is working you’ll need to measure it, not against what it has achieved or not achieved, but against what would have happened in its absence…
...That’s one big reason why economists rely on models, which are simplified enough to be understandable, yet close enough to the real world to hopefully gain real insights. It’s also why the field of economics is increasingly mathematically demanding...
...But the underlying problem still remains – you can’t really rigorously evaluate the impact of any particular policy or set of policies based on what has happened alone. It has to be measured against the “would have been” and “might have been”.
This isn’t just about QE or the ETP or any other public policy, it’s about all of them. Next time somebody says a particular policy has succeeded or failed, ask what the counterfactual would be.