Experiment—A Commentary by Ian Ayres ’86 and Barry Nalebuff
By Ian Ayres ’86 and Barry Nalebuff
Wal-Mart is a sophisticated cruncher of historical data. Before Hurricane Ivan hit Florida in 2004, the retailer had already started rushing strawberry Pop-Tarts to stores in the hurricane's path. By analyzing sales in areas hit by prior hurricanes, it was able to predict that people would be yearning for the gooey comfort of Pop-Tarts, finger food that doesn't require cooking or refrigeration.
Wal-Mart and its database engineers are well known for such cleverness. But they could be much cleverer still. They could do randomized tests. To the best of our knowledge, while the company experiments with different store layouts and price changes, it does no randomized tests, those in which the stores (or people or whatever) being observed are randomly assigned to either a test group or a control group.
What's so great about randomized tests? Why are they common in scientific studies and the gold standard for drug companies testing new medicines? The reason is that they eliminate subtle biases that can corrupt comparisons between groups.
Suppose that Wal-Marts with drinks at the front of the store have better sales, as a group, than the ones with drinks at the back. Can you be sure that it is the layout that delivered the results? It could be that more of the front-row-drink stores happen to be in the South, where it is hotter and people thirstier. The way to get the answer is to select a random subset of stores across the country to get a different layout and to compare their results with those at other stores.
For years it was believed that a low-fat diet reduces the risk of cancer. After all, low-fat Japan has lower cancer rates than the high-fat U.S. But in 2006 the Women's Health Initiative reported the findings of its $415 million, eight-year study of 49,000 women who were randomly assigned to different diets. The results showed no effect of dietary fat on cancer. It turns out we don't know why different countries have different cancer rates. Without a random trial you risk confusing correlation with causation.
Some businesses are hip to randomization. Capital One runs thousands of randomized tests to find out what kind of credit card solicitation is most effective. Should it offer a three-month teaser rate of 2% or a six-month teaser rate of 4%? Sending out alternative solicitations to random groups of prospects produces the answer. In South Africa, Credit Indemnity, now part of African Bank, used a randomized trial to learn that adding a photo of a smiling woman in the corner of the solicitation letter raised the response rate (of male customers) by just as much as dropping the interest rate by 4.5 percentage points.
The Internet has created new opportunities for testing, because in cyberspace randomization is quick and cheap. Companies like Offermatica will let you test which landing page generates the most sales (or form completions or clicks) by randomly showing surfers different versions. Randomized Web pages have shown online retailers, sometimes after just a few hours, that displaying a VeriSign warranty can generate more sales.
Google's Adwords has a randomization feature that lets anyone test a series of ads to see which ad produces the most click-throughs. Ian used this feature to help name his new book on how data-driven decision making is changing the world. The final candidates for the title: Super Crunchers: Why Thinking-by-Numbers Is the New Way to Be Smart and The End of Intuition: Why Thinking-by-Numbers Is the New Way to Be Smart. Ian preferred the latter. But a test on people searching for "data mining" or "number crunching" quickly showed (with 250,000 page views) that viewers were 63% more likely to click through on a "Super Crunchers" ad. So much for Ian's intuition.
The possibilities for randomized testing go beyond marketing and ad campaigns. Firms should randomize qualification standards--for hiring, for lending, for issuing insurance. An insurer can mine historical data to see whether it's making money on its current customers. But historical data can't reveal the potential profitability of people turned away. The answer is to accept some of the rejects in order to look for pockets of profitability. While many insurance companies shied away from insuring motorcycles, Progressive Insurance found that middle-age bikers with college degrees were not born to be wild.
It's time that businesses stopped relying on intuition and started testing. As Cole Porter said in his song, "Experiment!"
Ian Ayres and Barry Nalebuff are professors at Yale Law School and Yale School of Management. Ayres' latest book, Super Crunchers, was published in August. Visit their homepage at forbes.com/whynot.