Estimating Managing and Coaching Skills Using Counterfactual Analysis of Fantasy Football Data

• Activity Number: 468
• Type: Invited
• Date/Time: Wednesday, August 12, 2015 : 8:30 AM to 10:20 AM
• Sponsor: Section on Statistical Learning and Data Mining
• Author(s): Kaiser Fung*
• Companies: New York University
• Keywords: counterfactual analysis ; observational data ; sports analytics ; decision making under uncertainty ; causal inference ; fantasy football

Abstract

A novel set of metrics is developed to measure the skills of an owner of a fantasy football (NFL) team. Owner skills are decomposed into managing and coaching components: managing refers to the assembling of the roster while coaching refers to the game-day selection of first-team players. The methodology demonstrates two general strategies that are effective but not commonly practiced in dealing with Big Data problems. First, the seemingly complete dataset of historical decisions and outcomes is expanded to simulate counterfactual scenarios; second, a complete counterfactual analysis provides a rigorous estimation of the causal effect of managing and coaching decisions on outcomes. When a decision backfires, a fantasy team owner often blames a player's performance for not living up to past standards; using realized outcomes to score prior decisions is also common in Big Data analysis of observational events data. This research illustrates that decisions made in the face of uncertainty should be evaluated based on possible outcomes, rather than the realized outcome. A byproduct of this approach is a method to quantify the effect of luck on the outcome.