Thursday, January 31, 2013

Surveys And Opinions: Has No One Ever Heard Of Selection Bias?

I feel like banging my head against the wall (excerpt):

Study shows Barisan expected to win GE13

KUALA LUMPUR: Barisan Nasional is expected to win the 13th general election (GE13) which will be held this year, according to a study.

Titled “Study on Feedback of Undergraduate Voters”, it covered 3,000 respondents from Universiti Malaya (UM), Universiti Kebangsaan Malaysia (UKM) and Universiti Putra Malaysia (UPM).

The respondents involved undergraduates aged 21 and above from the Malay, Chinese and Indian communities, some of whom would be voting for the first time.

The outcome of the study, carried out from Dec 9 to 12 last year, also showed that Prime Minister Datuk Seri Najib Tun Razak is a popular leader, has good performance and qualified to be the Prime Minister.

The chairman of Yayasan Hal Ehwal Siswa, Datuk Mohd Radzi Abdul Latif said 57% of the respondents, representing UKM (75%), UPM (67.4%) and UM (41.4%), said the Prime Minister's leadership was good and popular.

“They feel that Najib is the best candidate for Prime Minister.

“The study also found that respondents who are on the fence will be the decisive factor for both sides,” he was quoted by Bernama as saying when announcing the result of the study here yesterday.

At the very least, this is a direct question put to potential voters, with a decent sample size and what looks like a good methodological approach (e.g. the question asked was essentially “who will win”, not “who will you vote for”, which removes some potential bias). The breakdown of the PM’s popularity looks plausible – though I’d dearly love to know what the number, gender and ethnic background of respondents for each university was.

But…when dealing with surveys such as this, you have to be really, really, really careful about making broad inferences. Right at the outset, the survey is already biased – it’s potentially random within the target sample, which is good (though I can’t tell for sure); but only for a subset of a subset of a subset of the population, which is very, very bad.

Can anyone pretend that voting preferences for 21-23 year olds, in tertiary education, in a select public university, is representative of all Malaysian 21-23 years olds? Much less the population at large? Can you say selection bias?

Currently, something like 30% of each Malaysian age cohort actually makes it into a university, and something like half that get into a public university. Presumably they got there because they’re smarter or harder working then their peers (questionable assumptions, I know, especially with quotas), but as such they may have very different viewpoints and preferences on politics and political parties than those who don’t

So you have a target sample that may not be representative of the population subset from which it is drawn. Furthermore, this larger population subset (21-23 year olds) may itself be uncharacteristic of the larger population set (all Malaysian voters).

Unless you know, or can objectively estimate, the extent of these differences (the bias built into the survey by its design), you can’t draw any broad conclusions, only conclusions specific to the target sample and its population subset.

Possibly I’m getting upset just over the headline – this won’t be the first time Bernama’s spinning makes me dizzy. But I’d be a blockhead to accept the conclusions of this survey (or that headline) at face value.

Why, oh why, doesn’t someone do a well-designed, fully representative, randomised, and country-wide survey? Surely there’s enough people interested and money floating around to get that done?


  1. Hi HishamH, I just came across your informative website today. I'm a Malaysian residing overseas temporarily. Since you've labelled this post under Forecasting and Politics, may I kindly seek your forecast or in the very least, a general opinion on where Malaysia' economy will head towards if (a) BN retains control, and (b) if Pakatan wrestles control?

    If you wish to evade the above question to maintain neutrality, then heck, I wouldn't blame ya!

    In addition, I've several questions of a different nature, meaning they're not flavoured by Malaysian politics:

    1. Are world governments in the midst of a currency war? I'm asking this because that's what I've been stumbling across in many major news outlet of late.

    2. After spending time reading up on this elsewhere on the internet, there are some who are saying that a currency war is a prelude to a global trade war, do you concur?

    3. Which some are further saying that a full blown trade war will eventually lead to a hot war (World War), is that true?

    4. And finally, that the next World War will eventually lead to the destruction of all paper currencies in this world, do you agree with this scenario?

    Hoping that you, being an economist and all can comment on these. Thanks, really appreciate it if you can respond to my questions.

  2. Hi Fami,

    I don't see any substantive differences in the economic policies espoused by either side - Malaysia's political divide is not ideological in nature. It's differences in degrees, and not in kind. All they're really arguing about is who makes the better curry, and not one offering curry and the other offering tandoori - if you get my drift.

    As for your other questions:

    1. No

    2. No

    3. No

    4. No

    A currency war presumes some sort of nominal anchor, and the world has not had one since 1972. People presume that quantitative easing is the same thing as currency intervention, but again that's not true, especially with demand for money so high (the Yen, the US dollar and the Swiss Franc are all suffering from safe haven status).

    A second point is that a trade war presumes competition in independent goods and services, but this is patent nonsense in a world where supply chains are global, and not local e.g. half the parts in German cars actually come from outside Germany, and 60%-70% of the components of electronics goods "made in
    China" actually come from somewhere else.

    Under those conditions, exchange rates don't actually matter that much because your export receipts and your import costs rise and fall more or less together i.e. the only gains to be made from competitive depreciation/devaluation come from local value-added.

    A third related point is that policy makers know very well the lessons of the Great Depression, and won't repeat the same mistakes. We'll make fresh mistakes, but not the ones that have been made before.

    Finally, hot wars have historically resulted in more widespread adoption of paper currencies, not less.

    1. I understand your drift about who makes better curry, but what if I don't want any curry at all? If you catch my drift...

      Thank you for answering those questions and I must say that you have such a tremendous optimism and confidence with regards to the present and future actions of policy makers, of which I unmistakably lack.

      From that third related point you've made, is it correct for me to assume that you think policy makers won't drive their economies to the ground again, or that they won't cause another Great Depression because they've learned the lessons of the previous one?

    2. Fami, if you don't like curry, you have to go to another restaurant :)

      With respect to policy-making, its less about optimism and more the empirical evidence - people have been shouting currency war for nearly three years now, and there still haven't been tit for tat competitive devaluations (technically depreciations).

      Part of the reason is that everyone recognises the need for global re-balancing of demand - the Yen for instance is by consensus overvalued, and Japan is running a trade deficit. The US dollar too by most measures is overvalued (and they're running a trade deficit too).

      Lastly, three big policy lessons from the Great Depression - don't constrict the money supply; don't block trade; and don't anchor your currency to anything. Not all the lessons have been fully learned - Europe is relearning that last one.

    3. You're absolutely correct about going to another restaurant! But hey, there's one more option for me to consider -- home cooking!

      Thanks man, have a nice day!