On the Near Impossibility of Measuring the Returns to Advertising

• Activity Number: 576
• Type: Invited
• Date/Time: Wednesday, August 7, 2013 : 2:00 PM to 3:50 PM
• Sponsor: Section on Statistics in Marketing
• Author(s): Randall Aaron Lewis*+ and Justin Rao
• Companies: Google, Inc. and Microsoft Research
• Keywords: advertising ; economics ; power ; noise ; returns ; marketing

Abstract

Classical theories of the firm assume access to reliable signals to measure the causal impact of choice variables on profit. For advertising expenditure we show, using twenty-five online field experiments (representing $2.8 million) with major U.S. retailers and brokerages, that this assumption typically does not hold. Statistical evidence from the randomized trials is very weak because individual-level sales are incredibly volatile relative to the per capita cost of a campaign--a "small" impact on a noisy dependent variable can generate positive returns. A concise statistical argument shows that the required sample size for an experiment to generate sufficiently informative confidence intervals is typically in excess of ten million person-weeks. This also implies that heterogeneity bias (or model misspecification) unaccounted for by observational methods only needs to explain a tiny fraction of the variation in sales to severely bias estimates. The weak informational feedback means most firms cannot even approach profit maximization.

Note: This work is based on the authors' publicly released work @ Yahoo!, Inc., completed during 2008-2011.