Distributed Estimation and Inference with Statistical Guarantees

• Activity Number: 149
• Type: Invited
• Date/Time: Monday, August 1, 2016 : 10:30 AM to 12:20 PM
• Sponsor: International Chinese Statistical Association
• Author(s): Heather Battey and Jianqing Fan* and Han Liu and Junwei Lu and Ziwei Zhu
• Companies: Imperial College London and Princeton and Princeton and Princeton and Princeton
• Keywords: Divide and Conquer ; Wald Test ; Likelihood Ratio Test ; Debiasing ; High-dimensional Linear Model ; Refitting

Abstract

This paper studies hypothesis testing and parameter estimation in the context of the divide and conquer algorithm. In a unified likelihood based framework, we propose new test statistics and point estimators obtained by aggregating various statistics from k subsamples of size n/k, where n is the sample size. In both low dimensional and high dimensional settings, we address the important question of how to choose k as n grows large, providing a theoretical upper bound on k such that the information loss due to the divide and conquer algorithm is negligible. In other words, the resulting estimators have the same inferential efficiencies and estimation rates as a practically infeasible oracle with access to the full sample. Thorough numerical results are provided to back up the theory.