Use of Paradata to Assess the Quality and Functionality of the American Community Survey
Marcy C. Davis
U.S. Census Bureau
Rachel Horwitz
U.S. Census Bureau
Jennifer Guarino Tancreto
U.S. Census Bureau
Mary Frances Zelenak
U.S. Census Bureau
Many of the recent discussions of paradata have focused on survey operations and nonresponse weighting adjustments. In this capacity, paradata have been used to optimize call back schedules, and interviewer observations have been used to supplement information from non-responding households. However, using paradata generated by the respondents has not been given as much attention. Collecting this type of information can lead to a better understanding of how a respondent interacts with and understands a survey, as well as provide researchers with tools to reduce measurement error. As part of the April 2011 American Community Survey (ACS) Internet Test, the Census Bureau inserted JavaScript code into every page of the online instrument to capture a vast amount of paradata, along with respondent answers. Specifically, all clicked links were captured (navigational buttons as well as radio buttons, check boxes, help, etc.), along with timestamps, field values, errors rendered, invalid logins, timeouts, and logouts. These data were collected to help evaluate the quality of the ACS Internet instrument by making sure respondents were using the instrument as expected. Additionally, these paradata can help identify problematic screens or other issues with the instrument that might impact data quality. This paper analyzes the paradata collected during the 2011 ACS Internet Test. Specifically, it discusses the authentication procedures, problematic screens (as determined by errors, use of help, and breakoffs), response times, and data quality indicators. It will discuss the potential problems with the instrument, things that worked well, and plans for future research based on the findings from this test.
