ANALYZING LARGE SCALE RECURRENT EVENT DATA with a DIVIDE-AND-CONQUER APPROACH

• Activity Number: 404 - Novel Methods for High-Dimensional and Large-Scale Survival Data
• Type: Topic Contributed
• Date/Time: Wednesday, August 5, 2020 : 1:00 PM to 2:50 PM
• Sponsor: WNAR
• Author(s): Jerry Cheng*
• Companies: New York Institute of Technology
• Keywords: Big data; Divide-and-conquer ; Meta-analysis; Recurret event data; Frailty models

Abstract

Currently in analyzing large scale recurrent event data, there are many challenges such as memory limitations, unscalable computing time, etc. In this research, a divide-and-conquer method is proposed using parametric frailty models. Specifically, the data is randomly divided into many subsets and the maximum likelihood estimator from each individual data set is obtained. Then a weighted method is proposed to combine these individual estimators as the final estimator. It is shown that this divide-and-conquer estimator is asymptotically equivalent to the estimator based on the full data. Simulation studies are conducted to demonstrate the performance of this proposed method. This approach is applied to a large real data set of repeated heart failure hospitalizations