Emulation of Global 3D Spatio-Temporal Temperature: A Distributed Computing Approach to Model One Billion Data Points

• Activity Number: 445
• Type: Topic Contributed
• Date/Time: Wednesday, August 6, 2014 : 8:30 AM to 10:20 AM
• Sponsor: Section on Statistics and the Environment
• Author(s): Stefano Castruccio*+ and Marc G. Genton
• Companies: KAUST and King Abdullah University of Science and Technology
• Keywords: big data ; space-time statistics ; distributed computing ; sphere

Abstract

The use of statistical models rather than more time-consuming systems of partial differential equations is a promising avenue for providing simple, fast, and easy-to-use tools for decision making. The main challenge of fitting a statistical model to climate model output in space and time is the amount of data: covariance matrices describing interactions between all data points easily become too large to store in RAM memories, thus making data reduction or model simplification necessary. In this presentation, we show how the gridded geometry of the data and the use of computational facilities with a large memory and a large number of processors allow for emulation of global three-dimensional temperature fields over time without data reduction. By using designed and dedicated computational facilities and a spectral statistical model specifically defined to exploit parallelization, it becomes possible to fit models with data of unprecedented size. In this presentation, we show how it is possible to fit a non-trivial model to a data set of one billion data points with a covariance matrix comprising of 10^18 entries.