A Question Of Sampling

I’m not an expert in sample design, but I think I’m qualified to comment on this (excerpt):

Opposition manipulated Umcedel study, claims prof

An academician has accused the opposition of manipulating the Universiti Malaya's Centre for Democracy and Elections' (Umcedel) finding that voters see opposition leader Anwar Ibrahim as more prime ministerial than incumbent Mohd Najib Abdul Razak.

Prof Zainal Kling (right), the Tun Abdul Ghafar Baba fellow at the Universiti Pendidikan Sultan Idris, said this in a national news agency Bernama report yesterday, rebutting the study's conclusion a day after it was announced to the press and academia...

...He also suggest that the methodology used by the study is suspect.

The article also quoted Universiti Teknologi Malaysia geostrategist Prof Azmi Hassan who questioned the profiles and location distribution of the respondents as well as the sampling size.

"Based on the number of eligible voters in the 13th general election, the study is too small as the respondents form only 0.010 percent of total voters," he said

He argued the study cannot "represent the people's aspirations" as its 1,407 sample size is not sufficient to represent the nation's 28 million population and the more than 13 million voters...

...However, Umcedel is standing firm on its study, blaming the criticisms on "confusion" over the connection between the term "population" and "sample" and certain quarters making "half-baked comments" without academic merit.

In a statement today, Umcedel's secretariat explained that a sample is a small number chosen to represent a population and does not have to be as large as the population.

It maintained that the standards used to select the sample and determine its size were on par with international pollsters such as MORI, YouGov and Gallup.

You can read the results for yourself here.

I think the sample size is just fine, and well above what I’d consider to be adequate. So with due respect to Prof Zainal, I think he’s blowing smoke on this particular criticism.

But there are problems with the survey, and it does have a particular (statistical) bias built-in. The demographic data of the respondents suggests the survey results are largely reflective of the preferences of young, urban, educated Chinese, not the population as a whole. For example half the survey respondents have tertiary education, but less than 20% of the labour force (and thus voters) do. Half the respondents are Chinese even though they are just a quarter of the total population.

A second issue which nobody else appears to have picked up on is the time span of the survey – two weeks is a long, long time in politics. This sort of time span usually isn’t a problem for surveys, but political opinion surveys are peculiarly sensitive animals – I’d actually prefer to see simpler, smaller surveys conducted on a daily basis to minimise potential bias.

To be fair to UMCEDEL, they claim the survey sample is representative of the “population” surveyed in its statistical meaning, not its literal one. You need to take the estimated sample error with that in mind. But to then make sense of the results, we’d need a much more granular idea of where and when the sampling was conducted, not just the vague “Semenanjung Malaysia” given.

I’ve had my reservations about UMCEDEL’s surveys before – this seems par for the course. I suspect a lot of it has to do with inadequate resources to do a thorough job. But time will tell how accurate they are.