Using a Crowdsourcing Contest to Find a More Predictive Chess Rating System: The Deloitte/Fide Chess Rating Challenge

• Activity Number: 104
• Type: Invited
• Date/Time: Monday, July 30, 2012 : 8:30 AM to 10:20 AM
• Sponsor: General Methodology
• Author(s): Jeff Sonas*+
• Companies: Sonas Consulting
• Keywords: Elo ; Kaggle ; chess ; rating ; crowdsourcing ; contest

Abstract

Many gaming and sporting organizations measure their competitors' performance through an "Elo rating". Arpad Elo, the inventor of the system that bears his name, designed his rating system to measure chess ability and to probabilistically predict chess outcomes, though the system has been used much more widely in different games and sports. Marvelous in its simplicity, the Elo formula has been the primary tool for half a century to estimate the relative skills of chess players around the world. Despite its foundation specifically intended to predict upcoming results, the Elo system is clearly inferior to other approaches at predicting the outcomes of chess games. Kaggle recently hosted two crowdsourcing contests to identify the most predictive chess rating algorithms, in which hundreds of competitors designed their own chess rating systems and could assess their contest standings via a live online leaderboard. As contest organizer, I will discuss the structure, benefits, and challenges of these contests, and we will also look at the details of the winning methodologies.