Rank And File: The Corruption Perception Index

You can’t miss it in the news over the past few days – Malaysia’s ranking in Transparency International’s Global Corruption Perception Index has “improved” from 60th in 2011 to 54th. Or has it?

A friend told me the methodology had changed, and so it has (details here; warning: pdf link). This made me a little upset, because its clear that most of the work I put in for this series of posts was mostly wasted.

More relevant still, its also clear that the CPI rankings in previous years are pretty much worthless for tracking both corruption and the perception of corruption over time.

Here’s the relevant quote:

“Previously, the CPI was based on perceptions of corruption in each country/territory, relative to the other countries scored and ranked on this index. This was because the Index captured the rank position of each country in each data source, so that country scores were highly dependent on the changes in scores of the countries around it in the index.”

How TI got away with doing this for 15 years I don’t know. To explain in simple terms, the CPI aggregated the rankings in the underlying surveys rather than the actual scores, which means that not only does the CPI have only a passing resemblance to the actual incidence of corruption, it’s not even a good measure for the perception of corruption. You shouldn’t be using rankings, because for measuring the underlying phenomenon we’re interested in, rankings are pretty worthless except in the very special and unlikely case that the global incidence of corruption is a constant.

To give an example, let’s take country A who’s actual perception score stays the same across time. Say global awareness of corruption becomes higher in one year, leading to a higher perceived level of corruption. Our Country A’s ranking then improves, even if its score stays the same. The opposite happens if say countries around the same rank as Country A all see better perception scores, which makes Country A rank worse, again even if its score stays the same.

So for practical purposes, ranking alone doesn’t convey a whole lot of information, and the way TI have compiled the CPI over the years means that changes in the CPI scores are even less meaningful.

The new methodology is substantially better – it uses the scores rather than the rankings from the underlying surveys, which gives the composite score a better chance of being a “true” estimate of the real value and also has the virtue of being comparable across time.

Even here though, I’d like to insert a note, no a full symphony, of caution. The probability range for each country score is fairly large, which means any apparent changes in the rankings should always be taken with three or four pinches of salt, lots of prayer, and a healthy supply of boots in case of foot in mouth disease.

To illustrate, let’s take a look at the Malaysian data for 2012 (the full CPI dataset is available here). Malaysia’s 2012 score of 49 derived from 9 different surveys, with a standard error of 3.4. The published 90% confidence interval is between 55 to 44 i.e. 49 is just the point estimate and we can be 90% confident that the actual true value lies between 55 and 44. In terms of ranking, Malaysia’s actual rank could lie as  much as 8 notches higher to 12 notches lower (presuming the point estimates for all the other country scores are actually correct, which is unlikely).

To say, with statistical certainty, that there is an improvement in the CPI score the change will have to exceed the confidence range. Using the numbers above, Malaysia’s score next year would have to be above 55 to say with any certainty that perception of corruption is receding and we’re doing better. Similarly, the score would have to be below 44 before we can claim with some certainty that we’re turning into Nigeria.

Note here that I’m talking about the score (which matters) and not the ranking (which doesn’t). This isn’t the EPL or La Liga – going to the top of the table doesn’t earn you a trophy at the end of the season.

I’ve already seen commentary lauding the “improvement” in ranking for Malaysia, when by rights we cannot be sure that there has actually been an improvement. For that matter, almost all the commentary is about the ranking, which frankly doesn’t really matter a whole lot.

I understand the reasons for the way the CPI was constructed – TI’s purpose with this is to put pressure on societies and governments to get more serious about corruption, and peer pressure is a pretty effective tool for leverage – but that doesn’t mean we shouldn’t be cognisant of its weaknesses.