Wednesday, April 25, 2012

Evaluating Public Policy: Dreaming of Parallel Earths

How do you decide if a public policy that’s been implemented is actually working? It’s unfortunately quite a bit harder than it looks.

You could take a simplistic approach and just look at what happens – if the desired outcome is reach, you declare victory and move on. Much of public policy evaluation, at least as politicians and the man on the street are wont to take it, tends to be of this variety.

And if things don’t turn out as well as expected, that’s when the fingers start pointing and public discussion heats up again.

The problem is that, from an economics standpoint it’s really not that simple.

Lots of things can have an impact on any given target variable, and you can’t assume that whatever changes occurred are purely from the effect of a particular policy. Improvements or a deterioration in the situation targeted by that policy could have come about from completely unrelated sources.

I’m continually reminded of this conundrum every time a public policy is debated in the press and blogosphere, from the ETP down to PPSMI, to the current debate over PTPTN, and even the old NEP.

In the sciences, when the effect of something is of interest, researchers set up experiments where that effect can be isolated and measured. For example in medicine, drug testing is almost always conducted with two separate groups of test subjects, one of which gets the drug and the other receives a placebo. That way, hopefully, the effect of the drug can be statistically tested, as all other conditions are as equal as they can be made to be.

A proper evaluation and validation of any public policy would ideally follow the same scientific principle – changing just the one thing and holding everything else unchanged, so we can figure out what the actual effect is.

By the very nature of public policy, this is of course pretty much impossible. You can’t exactly implement any important national policy and then say to half the population, “Hang on, you’re the control group, so this policy doesn’t apply to you.”

Even if you could do so (and limited trial programs are one way of getting around this problem), you still come up against the fact that you can’t exactly hold everything else equal. Things move and change, as do people, and you’re still faced with the need to disentangle all the separate effects to examine the one you’re interested in.

Also, given the nature of much of public policy, where the effects will be felt over multiple years, any such experiment would be subject to potential “contamination” from other sources of change – and you see we’ve got a big problem.

In a nutshell, figuring out whether a particular public policy is working you’ll need to measure it, not against what it has achieved or not achieved, but against what would have happened in its absence.

For example, to rigorously evaluate whether the ETP has actually increased the rate of investment and growth of the economy, you have to measure the programme not against the targets Pemandu have set, but rather against the performance in the hypothetical world that would have existed if the ETP had not been implemented.

Same thing with PTPTN: the proper measurement would not be the number of graduates that were produced (and the level of debt they’re having to carry), but rather the number of graduates we would have had if PTPTN had not existed.

And the same issue arises with PPSMI, affirmative action, and so on.

These measuring sticks, by their very nature, are of course unknown, unknowable and untestable, unless you believe in multiple dimensions and a parallel Earth! If that were true, if we lived in the world of “Fringe”, we could implement a policy on one Earth and hold it back on another, and then test to see if there are any real differences. Now that would be the scientifically rigorous approach, even if ethically dubious.

Since alas, to our knowledge, parallel Earths don’t actually exist, we’re left with muddling through on less than perfect means of evaluation.

Worse, the existing or non-existence of a public policy also has implications beyond the immediate target being aimed at – people change their behaviour depending on the environment they are in, whether its physical, social or legal.

Because of these inconvenient problems, a public policy:

  1. Could in fact have a significant positive effect, even if the actual outcomes look increasingly worse.
  2. Or it could have a significant negative effect, even if the outcomes look better.

You can’t really tell one way or another, unless you account for all the varying (and dynamically changing) other factors that go into the situation that you’re looking at.

That’s one big reason why economists rely on models, which are simplified enough to be understandable, yet close enough to the real world to hopefully gain real insights. It’s also why the field of economics is increasingly mathematically demanding.

Even then, the putative public policy wonk has to be careful about the structure of the model he or she is building, as different assumptions will lead to different, and often contradictory, insights. There’s also insights to be gained by research done in other countries and at other times, though differences in the social, economic, cultural and legal contexts supply many caveats.

But the underlying problem still remains – you can’t really rigorously evaluate the impact of any particular policy or set of policies based on what has happened alone. It has to be measured against the “would have been” and “might have been”.

Some parallel Earths would be pretty useful right now.


  1. I beg to offer a different perspective;

    1.We can gauge certain outcomes by measuring /benchmarking against neighbouring countries.Thus,is FDI good?Well S'pore & Indonesia is looking smarter.
    2. Base assumptions must be correct.For intance..BAU growth shld can be based on last 10 years trends.Or modified accordingly to anticipated future events.( oil depletion,palm oil prices).Thus the 2.8% inflation n 1% population growth are good.But why isn't BAU GNI growth defined?

    3. And any numbers adopted within the projections must have a sound basis.For instance High Income threshold shld be based on last 10/20 or 30 years trends..not just last one year.And there should be a"safety"factor inbuilt.Of cos,year to year it can be adjusted..but along same principles.

    I believe ETP was not done correctly.It was not validated by experienced economists.It is and still is a Public Relation exercise first...and Marketing Program for Big Biz.

    I really hope Pemandu will review and do it all over again on a more professional n academic manner.

    Think NEM,GTP n SRIs are good.ETP is the big failure that envelopes the whole exercise tho.

    I could go on.

    1. @anon

      I think you missed the point of my post. You're confusing ex-ante target setting with ex-post evaluation.

      To be clear, taking the ETP as an example, suppose we clarify and make consistent all the target numbers.

      Suppose then, that the economy hits all the targets specified, or near enough as to make no difference. Does this mean that the ETP was successful?

      In terms of what I've written in this post, the answer is no.

      If on the other hand, the economy significantly underperforms, does this mean the ETP was a failure?

      Again, in an economic context, the answer is no.

  2. True...future too complex,too many variables..I would think one can't "program" economy in total.Possibly only set the right framework and policies.

    But Pemandu claims they fact ETP is a program n not a plan.Cos its founded on numbers and with true North.Thus with that claim,their numbers needs to be better.

    And its about measuring how true those numbers valid their assumptions are.Thats their KPIs..not the economy per se.

    Thus if a non NKEA blooms into a most prolific economic activity..ETP have failed.Or an EPP flounders and is a drag on the economy (MRT for intance) ETP is a failure.

    Actually,we are not too far apart.My concern is ETP focuses on the wrong areas and will spend the next 3 years trying to tell us they are doing good.Your take is that its impossible to evaluate the exact drivers cos interactively its one huge dynamics.

    Sorry for flogging a dead horse.I am just trying to really understand economics.

  3. End-to-end integrity in implementing policies is also important. If eroded, people will perceive policy-based projects as quick-rich schemes for the wired. This perception will in turn disrupt the very inter-connectivity and cooperation that is the basis of all economic progress. Without wide support, projects will only realize sub-par acceptance. New policies for new generations will not improve matters because real progress is built one generation on another.

  4. Its getting beyond funny.
    2009 GNI is nominal 830 bil.Target usd15k at current exchange rate of RM3 is RM 1.44 trillion.Thus nominal growth required is 6.3%.Assuming inflation of 2.8%..the real growth is 3.8%.Surely our economy can hit that figure without too much intervention n Pemandus.
    The very least that shld be done is to set high benchmarks eg 2.2 trillion or just let it be thru good n solid policies.
    This lie about how difficult it is to be a high income economy must stop.Its outright tipu n disgraceful.

  5. Made an error...830 gni is 2011..n we need only 6.3% nominal growth for nxt 9 years to hot usd15k.

  6. Why be wishy washy..with all the mumbo jumbo of ‘would have been’ and ‘might have been’…why can’t one of us - who's credible enough - to at least insinuate that we’re being SAP’ed or neoliberalised to the core. We’re already seeing huge disparities, inequalities..can’t imagine what’ll look like in 2020.

    Income and wealth distibution already askewed today...and shifting more to the wealthier 1% or 5% end of the scale, going forward.

    We don’t have parallel earth but there is history and there are parallel economies. What about Chile, Argentina, Egypt, Mexico, Haiti..and more?

    Let’s avoid semantics. Let’s be bold and call a cangkul a cangkul.

  7. Anon7.45pm,
    Why you blame Pemandu..they're themselves generating some 750 million GNI this year excluding multiplier effects.
    They are a respectable EPP in themselves, probably second only to the MRT project - in terms reliability of meeting targets..sure to gobble up all the RM750 million budget allocated.

  8. Walla, what's yr take on neoliberalism?