Thursday, January 31, 2013

Surveys And Opinions: Has No One Ever Heard Of Selection Bias?

I feel like banging my head against the wall (excerpt):

Study shows Barisan expected to win GE13

KUALA LUMPUR: Barisan Nasional is expected to win the 13th general election (GE13) which will be held this year, according to a study.

Titled “Study on Feedback of Undergraduate Voters”, it covered 3,000 respondents from Universiti Malaya (UM), Universiti Kebangsaan Malaysia (UKM) and Universiti Putra Malaysia (UPM).

The respondents involved undergraduates aged 21 and above from the Malay, Chinese and Indian communities, some of whom would be voting for the first time.

The outcome of the study, carried out from Dec 9 to 12 last year, also showed that Prime Minister Datuk Seri Najib Tun Razak is a popular leader, has good performance and qualified to be the Prime Minister.

The chairman of Yayasan Hal Ehwal Siswa, Datuk Mohd Radzi Abdul Latif said 57% of the respondents, representing UKM (75%), UPM (67.4%) and UM (41.4%), said the Prime Minister's leadership was good and popular.

“They feel that Najib is the best candidate for Prime Minister.

“The study also found that respondents who are on the fence will be the decisive factor for both sides,” he was quoted by Bernama as saying when announcing the result of the study here yesterday.

At the very least, this is a direct question put to potential voters, with a decent sample size and what looks like a good methodological approach (e.g. the question asked was essentially “who will win”, not “who will you vote for”, which removes some potential bias). The breakdown of the PM’s popularity looks plausible – though I’d dearly love to know what the number, gender and ethnic background of respondents for each university was.

But…when dealing with surveys such as this, you have to be really, really, really careful about making broad inferences. Right at the outset, the survey is already biased – it’s potentially random within the target sample, which is good (though I can’t tell for sure); but only for a subset of a subset of a subset of the population, which is very, very bad.

Can anyone pretend that voting preferences for 21-23 year olds, in tertiary education, in a select public university, is representative of all Malaysian 21-23 years olds? Much less the population at large? Can you say selection bias?

Currently, something like 30% of each Malaysian age cohort actually makes it into a university, and something like half that get into a public university. Presumably they got there because they’re smarter or harder working then their peers (questionable assumptions, I know, especially with quotas), but as such they may have very different viewpoints and preferences on politics and political parties than those who don’t

So you have a target sample that may not be representative of the population subset from which it is drawn. Furthermore, this larger population subset (21-23 year olds) may itself be uncharacteristic of the larger population set (all Malaysian voters).

Unless you know, or can objectively estimate, the extent of these differences (the bias built into the survey by its design), you can’t draw any broad conclusions, only conclusions specific to the target sample and its population subset.

Possibly I’m getting upset just over the headline – this won’t be the first time Bernama’s spinning makes me dizzy. But I’d be a blockhead to accept the conclusions of this survey (or that headline) at face value.

Why, oh why, doesn’t someone do a well-designed, fully representative, randomised, and country-wide survey? Surely there’s enough people interested and money floating around to get that done?


  1. Hi Fami,

    I don't see any substantive differences in the economic policies espoused by either side - Malaysia's political divide is not ideological in nature. It's differences in degrees, and not in kind. All they're really arguing about is who makes the better curry, and not one offering curry and the other offering tandoori - if you get my drift.

    As for your other questions:

    1. No

    2. No

    3. No

    4. No

    A currency war presumes some sort of nominal anchor, and the world has not had one since 1972. People presume that quantitative easing is the same thing as currency intervention, but again that's not true, especially with demand for money so high (the Yen, the US dollar and the Swiss Franc are all suffering from safe haven status).

    A second point is that a trade war presumes competition in independent goods and services, but this is patent nonsense in a world where supply chains are global, and not local e.g. half the parts in German cars actually come from outside Germany, and 60%-70% of the components of electronics goods "made in
    China" actually come from somewhere else.

    Under those conditions, exchange rates don't actually matter that much because your export receipts and your import costs rise and fall more or less together i.e. the only gains to be made from competitive depreciation/devaluation come from local value-added.

    A third related point is that policy makers know very well the lessons of the Great Depression, and won't repeat the same mistakes. We'll make fresh mistakes, but not the ones that have been made before.

    Finally, hot wars have historically resulted in more widespread adoption of paper currencies, not less.

    1. Fami, if you don't like curry, you have to go to another restaurant :)

      With respect to policy-making, its less about optimism and more the empirical evidence - people have been shouting currency war for nearly three years now, and there still haven't been tit for tat competitive devaluations (technically depreciations).

      Part of the reason is that everyone recognises the need for global re-balancing of demand - the Yen for instance is by consensus overvalued, and Japan is running a trade deficit. The US dollar too by most measures is overvalued (and they're running a trade deficit too).

      Lastly, three big policy lessons from the Great Depression - don't constrict the money supply; don't block trade; and don't anchor your currency to anything. Not all the lessons have been fully learned - Europe is relearning that last one.