About | HeinOnline Law Journal Library | HeinOnline Law Journal Library | HeinOnline



26 Loy. Consumer L. Rev. 1 (2013-2014)
A Randomized Experiment Assessing the Accuracy of Microsoft's Bing It on Challenge

handle is hein.journals/lyclr26 and id is 9 raw text is: A RANDOMIZED EXPERIMENT
ASSESSING THE AccuRAcY OF
MICROSOFT'S BING IT ON
CHALLENGE
Ian Ayres*
Emad Atiq
Sheng Li**
Michelle Lu**
Tom Maher
& Christine Tsang***
Abstract: In advertisements associated with its Bing It
On campaign, Microsoft claimed that people preferred Bing
web search results nearly 2:1 over Google in blind comparison
tests. We tested Microsoft's claims by way of a randomized
experiment involving U.S.-based Amazon's. Mechanical Turk
(MTurk) subjects   and   conducted  on   Microsoft's  own
www.bingiton.com  website. We found that (i) a statistically-
significant majority of participants preferred Google search
results to Bing search results (53% to 41%); and (ii) participants
were significantly less likely to prefer Bing results when
randomly assigned to use popular search terms or self-selected
search terms instead of the search terms Microsoft recommends
test-takers employ on its website. Our findings suggest that some
of the claims implicit in Microsoft's advertisements warrant legal
scrutiny. The Bing It On Ad Campaign may be viewed as
(falsely) implying that: (i) Microsoft's claims about consumer
preferences for search engines were based on a generalizable
study; (ii) the preferences of five million individuals who have
taken the Bing It On Challenge online are either consistent with
or the basis for Microsoft's claim that consumers prefer Bing
* William K. Townsend Professor, Yale Law School. ian.ayres@yale.edu.
Yale Law School, J.D. 2014.
Yale Law School, J.D. 2013.

1