Identification of Signal, Noise, and Indistinguishable Subsets in High-Dimensional Data Analysis

• Activity Number: 585
• Type: Topic Contributed
• Date/Time: Wednesday, August 7, 2013 : 2:00 PM to 3:50 PM
• Sponsor: Section on Statistical Learning and Data Mining
• Author(s): X. Jessie Jeng*+
• Companies: North Carolina State University
• Keywords: Signal detection ; False positive control ; False negative control ; Multiple testing ; Variable screening

Abstract

Motivated by applications in high-dimensional signal detection where strong signals often stand out easily and weak ones can be inseparable from the noise, we develop a statistical framework to provide a novel categorization of the data into the signal, noise, and mixed subsets. The categorization can allow researchers working with real applications to design more appropriate follow-up studies for data in different subsets and allocate their resources more efficiently. We propose an efficient data-driven procedure to identify the three subsets. Theoretical study shows that under certain conditions, only signals are included in the identified signal subset and the rest of the signals are included in the identified indistinguishable subsets with high probability. Moreover, the proposed procedure adapts to the unknown signal intensity, so that the identified indistinguishable subset shrinks with the true indistinguishable subset when signals become stronger. The procedure is examined and compared with methods based on FDR control using Monte Carlo simulation, and applied successfully in a real-data application to identify genomic variants with different signal intensity.