Evaluating Public Policy: Dreaming of Parallel Earths

How do you decide if a public policy that’s been implemented is actually working? It’s unfortunately quite a bit harder than it looks.

You could take a simplistic approach and just look at what happens – if the desired outcome is reach, you declare victory and move on. Much of public policy evaluation, at least as politicians and the man on the street are wont to take it, tends to be of this variety.

And if things don’t turn out as well as expected, that’s when the fingers start pointing and public discussion heats up again.

The problem is that, from an economics standpoint it’s really not that simple.

Lots of things can have an impact on any given target variable, and you can’t assume that whatever changes occurred are purely from the effect of a particular policy. Improvements or a deterioration in the situation targeted by that policy could have come about from completely unrelated sources.

I’m continually reminded of this conundrum every time a public policy is debated in the press and blogosphere, from the ETP down to PPSMI, to the current debate over PTPTN, and even the old NEP.

In the sciences, when the effect of something is of interest, researchers set up experiments where that effect can be isolated and measured. For example in medicine, drug testing is almost always conducted with two separate groups of test subjects, one of which gets the drug and the other receives a placebo. That way, hopefully, the effect of the drug can be statistically tested, as all other conditions are as equal as they can be made to be.

A proper evaluation and validation of any public policy would ideally follow the same scientific principle – changing just the one thing and holding everything else unchanged, so we can figure out what the actual effect is.

By the very nature of public policy, this is of course pretty much impossible. You can’t exactly implement any important national policy and then say to half the population, “Hang on, you’re the control group, so this policy doesn’t apply to you.”

Even if you could do so (and limited trial programs are one way of getting around this problem), you still come up against the fact that you can’t exactly hold everything else equal. Things move and change, as do people, and you’re still faced with the need to disentangle all the separate effects to examine the one you’re interested in.

Also, given the nature of much of public policy, where the effects will be felt over multiple years, any such experiment would be subject to potential “contamination” from other sources of change – and you see we’ve got a big problem.

In a nutshell, figuring out whether a particular public policy is working you’ll need to measure it, not against what it has achieved or not achieved, but against what would have happened in its absence.

For example, to rigorously evaluate whether the ETP has actually increased the rate of investment and growth of the economy, you have to measure the programme not against the targets Pemandu have set, but rather against the performance in the hypothetical world that would have existed if the ETP had not been implemented.

Same thing with PTPTN: the proper measurement would not be the number of graduates that were produced (and the level of debt they’re having to carry), but rather the number of graduates we would have had if PTPTN had not existed.

And the same issue arises with PPSMI, affirmative action, and so on.

These measuring sticks, by their very nature, are of course unknown, unknowable and untestable, unless you believe in multiple dimensions and a parallel Earth! If that were true, if we lived in the world of “Fringe”, we could implement a policy on one Earth and hold it back on another, and then test to see if there are any real differences. Now that would be the scientifically rigorous approach, even if ethically dubious.

Since alas, to our knowledge, parallel Earths don’t actually exist, we’re left with muddling through on less than perfect means of evaluation.

Worse, the existing or non-existence of a public policy also has implications beyond the immediate target being aimed at – people change their behaviour depending on the environment they are in, whether its physical, social or legal.

Because of these inconvenient problems, a public policy:

1. Could in fact have a significant positive effect, even if the actual outcomes look increasingly worse.
2. Or it could have a significant negative effect, even if the outcomes look better.

You can’t really tell one way or another, unless you account for all the varying (and dynamically changing) other factors that go into the situation that you’re looking at.

That’s one big reason why economists rely on models, which are simplified enough to be understandable, yet close enough to the real world to hopefully gain real insights. It’s also why the field of economics is increasingly mathematically demanding.

Even then, the putative public policy wonk has to be careful about the structure of the model he or she is building, as different assumptions will lead to different, and often contradictory, insights. There’s also insights to be gained by research done in other countries and at other times, though differences in the social, economic, cultural and legal contexts supply many caveats.

But the underlying problem still remains – you can’t really rigorously evaluate the impact of any particular policy or set of policies based on what has happened alone. It has to be measured against the “would have been” and “might have been”.

Some parallel Earths would be pretty useful right now.