Dynamic Statistical Inference in Massive Datastreams

• Activity Number: 398 - Recent Advances in Streaming Data Analytics
• Type: Invited
• Date/Time: Wednesday, August 10, 2022 : 10:30 AM to 12:20 PM
• Sponsor: International Chinese Statistical Association
• Author(s): Lilun Du and Changliang Zou and Zhenke Wu and Jingshen Wang*
• Companies: HKUST and Nankai University and University of Michigan, Ann Arbor and UC Berkeley
• Keywords: streaming data; multiple testing

Abstract

Modern technological advances have expanded the scope of applications requiring analysis of large-scale datastreams that comprise multiple indefinitely long time series. There is an acute need for statistical methodologies that perform online inference and continuously revise the model to reflect the current status of the underlying process. In this manuscript, we propose a dynamic statistical inference framework--named dynamic tracking and screening (DTS)--that is not only able to provide accurate estimates of the underlying parameters in a dynamic statistical model but also capable of rapidly identifying irregular individual streams whose behavioral patterns deviate from the majority. Concretely, by fully exploiting the sequential feature of datastreams, we develop a robust estimation approach under a framework of varying coefficient models. The procedure naturally accommodates unequally-spaced design points and updates the coefficient estimates as new data arrive without the need to store historical data. A data-driven choice of an optimal smoothing parameter is accordingly proposed. Furthermore, we suggest a new multiple testing procedure tailored to the streaming environment.