Fast Precision Matrix Estimation by Partial Correlation Screening

• Activity Number: 488 - Hypothesis Testing When Signals Are Rare and Weak
• Type: Invited
• Date/Time: Wednesday, August 2, 2017 : 10:30 AM to 12:20 PM
• Sponsor: Section on Risk Analysis
• Author(s): Jiashun Jin* and Zheng Ke and Shengming Luo
• Companies: Carnegie Mellon University and University of Chicago and Carnegie Mellon University
• Keywords: network ; data collection ; data cleaning ; simplex structure ; PCA ; degree corrected mixed membership model

Abstract

We propose Partial Correlation Screening (PCS) as a new row-wise estimation approach to estimating sparse precision matrices. To estimate each row of the precision matrix, PCS employs a fast Screen and Clean algorithm, where it iteratively uses the partial correlation as the screening statistic. We apply PCS to two gene microarray data sets ($(p, n) = (8491, 181)$ and $(10237, 157)$), where PCS outputs a reasonable estimate of the precision matrix in $6.3$ and $9.6$ minutes, respectively. Higher Criticism Thresholding (HCT) is a modern a adaption of Fisher's LDA. Combining HCT with either PCS or the glasso gives rise to two new classifiers, HCT-PCS and HCT-glasso. For both data sets, the classification errors of HCT-PCS are much smaller than those of the HCT-glasso, suggesting PCS have a more reasonable estimate of the precision matrix than the glasso. Compared to more popular classifiers such as SVM and Rand Forest, HCT-PCS not only is competitive in classification errors, but also enjoys interesting theoretical properties the other two methods do not have.