Mode Effects in American Trends Panel: A Closer Look at the Person-Level and Item-Level Characteristics

• Activity Number: 34
• Type: Contributed
• Date/Time: Sunday, August 9, 2015 : 2:00 PM to 3:50 PM
• Sponsor: Survey Research Methods Section
• Author(s): Stanislav Kolenikov* and Kyley McGeeney and Scott Keeter and Courtney Kennedy
• Companies: Abt SRBI and Pew Research Center and Pew Research Center and Abt SRBI
• Keywords: mode effect ; online panel ; mixed model ; total survey error ; American Trends Panel ; multimode survey

Abstract

American Trends Panel is a probability panel with RDD recruitment developed by by Pew Research Center and Abt SRBI. Panel surveys have been conducted on different modes in different waves, including web for most panel members, and mail or phone for those who do not have access to the Internet. We analyze the results of the July 2014 wave (Wave 5) that included a comprehensive, large-scale mode-of-interview experiment that randomly assigned respondents to telephone and web modes, with approximately 1,500 respondents in each mode. To quantify the contributions to the mode effects of the different question characteristics in the 75-question instrument, we build a cross-classified mixed model with effects of person and question characteristics to identify the properties of survey questions that make them susceptible to mode effects, as well as the demographic groups that tend to exhibit mode effects. We discuss how the decomposition of the total survey error and explained variance helps identifying the properties of the questions that are associated with the mode effects, such as question format, topic, and the potential impact of social desirability.