The Practical Aspects of Doing Statistics on Large Data Sets

• Activity Number: 385
• Type: Roundtables
• Date/Time: Tuesday, August 6, 2013 : 12:30 PM to 1:50 PM
• Sponsor: Section on Statistical Computing
• Author(s): Joseph Rickert*+
• Keywords: Big data ; Parallel computing ; externam memory algorithms ; R programming ; mapreduce

Abstract

"Data scientists," "predictive analytics," "Big Data." Political strategists, health care providers, big league baseball, and your local "big box" retailers have developed insatiable appetites for analyzing large data sets. Entire industries have grown up around technologies such as Hadoop and MapReduce promising astonishing insights from "crunching" large amounts of data. Hyperbole aside, what are the practical issues, computational challenges, and client expectations facing statisticians attempting serious work with very large data sets? This roundtable will attempt to illuminate the landscape of issues and challenges by exploring two themes: (1) Are we really in a new era of statistical inference where the powerful "small sample" innovations developed in the 20th century are no longer sufficient? How does one get a feel for a data set for which you can't plot the data and even calculating a median might be painful or impossible? (2) What algorithms and computational environments are available for working with more than a hundred million observations, and what are their limitations? What do statisticians need to know about parallel algorithms? Can SAS and R keep up?