40 – Survey Weighting, Imputation, and Estimation
Variance Estimation for Combined Probability and Nonprobability Samples
Michael Yang
NORC at the University of Chicago
Survey researchers have proposed three general approaches to estimation from nonprobability samples: quasi-randomization, superpopulation modeling, and doubly robust (Valliant, 2020). Through case studies and Monte Carlo simulations, the authors have evaluated some commonly used estimation methods associated with these approaches (Ganesh et al., 2017; Yang, et al. 2018, 2019; Mulrow, et al. 2020). Our empirical evaluations show that these methods tend to produce comparable point estimates, but estimates under two of these methods, Propensity Weighting (quasi-randomization) and Small Area Modeling (doubly robust), exhibit superior properties in terms of bias reduction, mean squared error, and confidence interval coverage. Focusing on these two methods, we expand our earlier simulations to explore variance estimation methods. Like our earlier evaluations, the simulation data was generated to mimic the coverage bias exhibited by opt-in online samples for some key characteristics. Our objective is to explore practical variance estimation solutions to guide practitioners who use nonprobability samples but may not have the resources to carry out elaborate variance estimation procedures. Our approach is to simulate Jackknife variances under Propensity Weighting and Small Area Modeling and compare with naïve variances or design variances where we assume that the combined probability and nonprobability sample is a probability sample.