Measuring Ad Effectiveness Using Continuous Geo Experiments

• Activity Number: 448
• Type: Invited
• Date/Time: Wednesday, August 3, 2011 : 8:30 AM to 10:20 AM
• Sponsor: Section on Statistics and Marketing
• Author(s): Jon Vaver*+ and Deepak Kumar and Jim Koehler
• Companies: Google Inc. and Google Inc. and Google Inc.
• Keywords: geo experiments ; continuous measurement ; online advertising ; AdWords ; Google

Abstract

Google uses geo experiments to help AdWords advertisers assess the effectiveness of their paid search advertising spend. These experiments measure the value of incremental ad spend, identify optimal bid levels, quantify the impact of online ad spend on offlines sales, etc. In these experiments, geos are randomly assigned to a control or treatment group and users in these geos are served, or not served, search ads accordingly. Typically, such one-off experiments require a custom setup, they are disruptive to the advertiser's marketing plan, and the results only reflect the competitive landscape at a snapshot in time. In this paper, we describe the adaptation of geo experiments to a framework of continuous measurement. This approach mitigates the issues stated above while providing more definitive results than equivalent one-off experiments. For example, results indicate that a continuous experiment that changes spend in 1/8th of the geos can achieve the same confidence interval as an analogous one-off (non-continuous) experiment that changes ad spend in 1/2 of the geos.