Sparsity in Modern High-Dimensional Statistics

• Activity Number: 264
• Type: Other
• Date/Time: Tuesday, July 31, 2012 : 8:30 AM to 10:20 AM
• Sponsor: ASA
• Author(s): David Donoho*+
• Companies: Stanford University

Abstract

We'll cover the ideas, methods, and applications driven by ``Sparsity of significant effects''. Ideas: starting from needle-in-a-haystack problems, where many different noisy effects are observed, but only a small fraction can be expected to be truly significant, we will progress to regression models in which there are many potential predictors of which most are useless but a few may be useful, and discriminant analysis, where useful features can be rare and weak. We will delineate the detection boundary, where the effects are so weak or so rare that no method can detect them. Methods: we start with soft and hard thresholding and Bayesian conditional means, the False Discovery Rate controlling and Higher Criticism approaches to dealing with needle-in-a-haystack and rare-weak models, and we discuss LASSO and various generalizations for selecting sparse models out of many predictors. Applications: In astronomy, we mention planet finding and searching for nonGaussianity in the cosmic microwave background. We discuss compressed sensing, and its applications in medical imaging, Radar signal processing, and various branches of spectroscopy. We end with a examples in genomics and computational biology.