Can Replicate-Based Methods Be Used in Variance Estimation for Cut-Point Estimators Derived Through ROC Analysis?

• Activity Number: 669
• Type: Contributed
• Date/Time: Thursday, August 2, 2012 : 10:30 AM to 12:20 PM
• Sponsor: Section on Survey Research Methods
• Author(s): Dan Liao*+ and Phillip Kott and Sarra Lorraine Hedden
• Companies: RTI International and RTI International and SAMHSA
• Keywords: Fay's BRR ; serious mental illness ; cutoff value ; bootstrap ; jackknife

Abstract

The Mental Health Surveillance Study (MHSS) uses data from clinical interviews administered to a sub-sample of adult respondents from the National Survey on Drug Use and Health (NSDUH) for estimating the prevalence of serious mental illness (SMI). First, probabilities of having SMI are estimated for each adult NSDUH respondent based a logistic regression fit to the MHSS sample. Then, a Receiver Operating Characteristic (ROC) analysis classifies NSDUH respondents as either having or not having SMI based on the estimated probabilities. A "cut-point" estimator of prevalence results from using those classifications. Due to the discrete property of ROC classification, the asymptotic variance of cut-point estimator cannot be linearized, and a replication method of variance estimation is needed. We discuss the theoretical reasons why Fay's balanced repeated replication (BRR) is superior to rival replication methods in this context. We then evaluate the results of a simulation study using the MHSS respondent sample to see whether Fay's BRR actually produces nearly unbiased variance estimates for the estimated SMI prevalence among all adults and among the Hispanic subpopulation.