Large-scale Parallel Statistical Forecasting Computations in R

• Activity Number: 348
• Type: Contributed
• Date/Time: Tuesday, August 2, 2011 : 10:30 AM to 12:20 PM
• Sponsor: Section on Statistical Computing
• Author(s): Murray Stokely and Farzan Rohani*+ and Nate Coehlo and Eric Tassone
• Companies: Google Inc. and Google Inc. and Google Inc. and Google Inc.
• Keywords: R ; distributed computing ; forecasting ; simulation ; statistical computing ; timeseries

Abstract

We demonstrate the utility of massively parallel computational infrastructure for statistical computing using the MapReduce paradigm for R. This framework allows users to write computations in a high-level language that are then broken up and distributed to worker tasks in Google datacenters. Results are collected in a scalable, distributed data store and returned to the interactive user session. We apply our approach to a forecasting application that fits a variety of models, prohibiting an analytical description of the statistical uncertainty associated with the overall forecast. To overcome this, we generate simulation-based uncertainty bands, which necessitates a large number of computationally intensive realizations. Our technique cut total run time by a factor of 300. Distributing the computation across many machines permits analysts to focus on statistical issues while answering