Respondents: Who Art Thou? Comparing Internal, Temporal, and External Validity of Survey Response Propensity Models Based on Random Forests and Logistic Regression Models

• Activity Number: 606
• Type: Contributed
• Date/Time: Wednesday, August 7, 2013 : 2:00 PM to 3:50 PM
• Sponsor: Survey Research Methods Section
• Author(s): Trent Buskirk*+ and Brady West and Anh Thu Burks
• Companies: Nielsen and Institute for Social Research and Nielsen
• Keywords: Random Forests ; Survey Response Propensity Models ; Address Based Sampling ; , Internal, Temporal and External Validity ; Logistic Regression ; Ancillary Data

Abstract

Address based sampling (ABS) provides quantitative researchers with a vast array of ancillary information that can be appended to the sampling frame at the block-group level for virtually every sampling unit. This study is based on a national probability sample of over 600,000 addresses randomly selected from an ABS frame for a media survey, where over 40 demographic and behavioral variables from the 2000 Census and other sources have been appended. With these data,we derive two survey response propensity models using random forests and logistic regression with principal components. The internal, temporal and external validity for both models in terms of predicting whether or not a selected household responded to the survey is evaluated. The internal validity was assessed using an iterative series of hold-out samples; temporal validity was assessed by applying the models to a second national random sample selected during the same year as the test sample using the same sampling design. Finally, external validity was assessed by applying both of the models to a small national probability sample selected over a year later from a broader sampling frame with the same target population.