Section on Statistical Computing

• Activity Number: 486
• Type: Invited
• Date/Time: Wednesday, August 1, 2012 : 10:30 AM to 12:20 PM
• Sponsor: Section on Statistical Computing
• Author(s): Saptarshi Guha*+
• Companies: Mozilla
• Keywords: RHIPE ; mapreduce ; R ; Hadoop

Abstract

The R and Hadoop Integrated Processing Environment (RHIPE) is a merger of the R Statistical Computing Project and the Hadoop Distributed Computing Platform. RHIPE enables the data analyst to compute over extremely large data sets with the MapReduce programming model using the R language. RHIPE ensures that user does not leave the R console: writing of R code, job submission and monitoring and reading in results are all done within the R console. Socorro is a Mozilla service for collecting, processing and displaying Firefox crash reports. Consumers of the service had questions regarding the correct sample size and the number of unseen crash report signatures (a unique ID to identify the type of crash). RHIPE was used to answer the questions related to the crash report data that runs into the hundreds of gigabytes stored across 70 computers. In addition, we talk about the Telemetry Project, a system to collect remote data from Firefox 7 onwards. We used RHIPE to divide the data into samples using simple random sampling without replacement and combine (numerically and visually) the results of statistical methods across these subsets.