Performance Tournaments with Crowdsourced Judges

• Activity Number: 129
• Type: Contributed
• Date/Time: Monday, August 5, 2013 : 8:30 AM to 10:20 AM
• Sponsor: Section on Statistics in Marketing
• Author(s): William Heavlin*+ and Daryl Pregibon
• Companies: Google Inc. and Google Inc.
• Keywords: Bradley-Terry model ; exploration and exploitation ; log-rank scores ; optimal ranking ; preference modeling ; Savage scores

Abstract

A performance slam is a competition among a fixed set of performances whereby pairs of performances are judged by audience participants. When performances are recorded on electronic media, performance slams become amenable to audiences that watch on-line and judge asynchronously ("crowdsourced"). In order to better entertain the audience, we want to show the better performances ("exploitation"). In order to identify the good videos, we want to glean a least some information about all videos ("exploration"). Our approach has three elements: (1) Our preference model we take from Bradley and Terry (1952). (2) Its parameters we calculate by rewriting the likelihood gradient into a fixed point estimate, one which mimics Mantel and Haenszel (1959). (3) Each pair of performances is chosen sequentially, always chosen to minimize the weighted variance of (the logarithms of) the Bradley-Terry parameter estimates. Our preferred weights consist of the log-rank weights proposed by Savage (1956). These algorithms we demonstrate with the 2011-2012 Youtube comedy slams.