Dynamic Clustering Methods for Analyzing Complex Systems

• Activity Number: 585
• Type: Contributed
• Date/Time: Wednesday, August 3, 2011 : 2:00 PM to 3:50 PM
• Sponsor: Section on Statistical Learning and Data Mining
• Author(s): Shawn Mankad*+ and George Michailidis and Andrei Kirilenko
• Companies: University of Michigan and University of Michigan and Commodity Futures Trading Commission
• Keywords: clustering ; spatiotemporal ; signal extraction ; online algorithm ; complex system ; finance

Abstract

Driven by advances in computer technology, businesses and organizations are increasingly able to collect data streams that capture the behavior of components in complex systems. Suppose we observe noisy snapshots, at the component level, of a complex system over time. Specifically, we observe the same features for each component at each time point. A challenging issue is to identify and interpret significant component behaviors and interactions. Further, in many cases it is impractical to store and organize the full data for retrieval and analysis. Thus, methods that do not require the full data and operate given the input of information up to a point in time are essential to the analysis of large systems. Such systems are encountered in biology, economics, transportation, among others. To gain insight into these interactions, we develop a dynamic biclustering method based on the plaid model. We introduce a regularization framework that smooths the model parameters over time, allowing us to identify persistent groups (components) and the critical features that separate them consistently over time. We illustrate this methodology with an application on electronic trading market data.