The Power Of Prediction

Not so much about economics, but the following shows mathematics – and statistics – are sexier than ever (excerpt):

How Companies Learn Your Secrets
By CHARLES DUHIGG

Andrew Pole had just started working as a statistician for Target in 2002, when two colleagues from the marketing department stopped by his desk to ask an odd question: “If we wanted to figure out if a customer is pregnant, even if she didn’t want us to know, can you do that? ”...

...Almost every major retailer, from grocery chains to investment banks to the U.S. Postal Service, has a “predictive analytics” department devoted to understanding not just consumers’ shopping habits but also their personal habits, so as to more efficiently market to them. “But Target has always been one of the smartest at this,” says Eric Siegel, a consultant and the chairman of a conference called Predictive Analytics World. “We’re living through a golden age of behavioral research. It’s amazing how much we can figure out about how people think now.”...

...Consumers going through major life events often don’t notice, or care, that their shopping habits have shifted, but retailers notice, and they care quite a bit. At those unique moments, Andreasen wrote, customers are “vulnerable to intervention by marketers.” In other words, a precisely timed advertisement, sent to a recent divorcee or new homebuyer, can change someone’s shopping patterns for years.

And among life events, none are more important than the arrival of a baby. At that moment, new parents’ habits are more flexible than at almost any other time in their adult lives. If companies can identify pregnant shoppers, they can earn millions...

This stuff is pretty different from what I’m used to, but its pretty illustrative of how far we’ve come in just a few short years – these kinds of analytics were akin to black magic just two decades ago.

It’s the marriage of statistical methodologies and computing power that’s really made this type of “data mining” possible, though I have to say that the term carries a very different meaning and connotation in the academic world.

The following passage shows the kind of pay-off that’s possible:

...About a year after Pole created his pregnancy-prediction model, a man walked into a Target outside Minneapolis and demanded to see the manager. He was clutching coupons that had been sent to his daughter, and he was angry, according to an employee who participated in the conversation.

“My daughter got this in the mail!” he said. “She’s still in high school, and you’re sending her coupons for baby clothes and cribs? Are you trying to encourage her to get pregnant?”

The manager didn’t have any idea what the man was talking about. He looked at the mailer. Sure enough, it was addressed to the man’s daughter and contained advertisements for maternity clothing, nursery furniture and pictures of smiling infants. The manager apologized and then called a few days later to apologize again.

On the phone, though, the father was somewhat abashed. “I had a talk with my daughter,” he said. “It turns out there’s been some activities in my house I haven’t been completely aware of. She’s due in August. I owe you an apology.”...

There’s some privacy concerns here – would you like your life to be such an open book? – and we’re probably being increasingly too desensitised about personal information being out in the open with the way social media has taken over private interaction.

But its a well-written, fascinating and insightful article that’s worth your time reading.

[Forbes magazine online via Dan Hirschman’s blog]