Online Program Home
  My Program

Abstract Details

Activity Number: 80 - Inference Methods for High-Dimensional and Complex Data
Type: Contributed
Date/Time: Sunday, July 30, 2017 : 4:00 PM to 5:50 PM
Sponsor: Section on Statistics in Imaging
Abstract #322855
Title: Adaptive Large-Scale Testing under Heterogeneity Sparsity
Author(s): Xiang Lyu* and Guang Cheng
Companies: Purdue University and Purdue
Keywords: large-scale test ; heterogeneity sparsity ; minimax optimality ; adaptive ; de-sparsified lasso
Abstract:

Massive data often consists of a growing number of potentially heterogeneous sub-populations. This fact violates the assumption of i.i.d. observations in popular large-scale methods, e.g., divider-and-conquer. In this paper, we propose a testing procedure for detecting heterogeneity in the framework of high dimensional linear models. A new sparsity arises in this problem, termed as heterogeneity sparsity. In theory, we prove that our large-scale test procedure is asymptotically consistent, and minimax optimal in the sense that it can consistently detect departure from null of a magnitude that no other tests could improve. In addition, the test is adaptive to unknown heterogeneity sparsity. An interesting phenomenon is revealed that, to ensure heterogeneity detection consistency, if either of model dimensionality and the number of sub-populations is large, the other should not be relatively too small. We name this phenomenon as bless of massive data. These theoretical results hold, in particular, when model dimensionality grows exponentially fast and the number of sub-population diverges. As a by-product, a consistent estimator of heterogeneity sparsity is proposed.


Authors who are presenting talks have a * after their name.

Back to the full JSM 2017 program

 
 
Copyright © American Statistical Association