What Would a Statistician Do with 10 Seconds on a Super Computer?

• Activity Number: 116
• Type: Topic Contributed
• Date/Time: Monday, August 4, 2014 : 8:30 AM to 10:20 AM
• Sponsor: Section on Bayesian Statistical Science
• Author(s): Douglas Nychka*+
• Companies: NCAR
• Keywords: Climate ; Rmpi ; Spatial data ; parallel computation

Abstract

The statistical problems of large climate datasets, the flexibility of R, and the architectures of current supercomputers has motiviated a different paradigm for data analysis for problems that are amenable to running in parallel. As motivation consider the spatial analysis of daily observed temperature fields for the North America and for the past 30 years. Typically each daily field, at least initially, can be analyzed separately by a single thread running R. The idea is initiate a large number of processors (cores), e.g. 10000, to work on each day separately. With this number of cores and for 30 years the execution time is comparable to the analysis of single day. Part of the value for statisticians is that most of this process can be handled within R using the Rmpi package. In this talk we give some examples of a practical implementation on the Yellowstone supercomputer supported by the National Center for Atmospheric Research and experience with creating a gridded data product based on historical temperature observations.