Developing Systems for Real-Time Streaming Analysis

• Activity Number: 307
• Type: Invited
• Date/Time: Tuesday, July 31, 2012 : 10:30 AM to 12:20 PM
• Sponsor: JCGS-Journal of Computational and Graphical Statistics
• Author(s): Sarah Michalak*+
• Companies: Los Alamos National Laboratory
• Keywords: Streaming Data ; Massive Datasets ; Real-Time Methods ; Radio astronomy

Abstract

Sources of streaming data are proliferating and so are the demands to analyze and mine streaming data in real time. Statistical methods frequently form the core of real-time analysis and therefore statisticians increasingly encounter the challenges and implicit requirements of real-time systems. This talk describes a broad process for development and implementation of streaming algorithms, beginning with exploratory data analysis in a flexible computing environment, followed by concrete specification of a computational algorithm and its initial implementation, and culminating in successive improvements to computational efficiency and throughput. This sequential development embodies a collaboration between statisticians, domain scientists, and the computer engineers developing the real-time system. The process is illustrated in the context of a radio-astronomy challenge to mitigate adverse impacts of radio frequency interference in searches for high-energy impulses from distant sources.