NAEP Questionnaire Pre-Testing Methods: Establishing Standards for Survey Question Pre-Testing in Large-Scale Assessments (303646)

The National Assessment of Educational Progress (NAEP) measures what grade 4, 8, and 12 students in the United States know and can do in various subject areas, ranging from Reading and Mathematics to Technology and Engineering Literacy. In addition to the subject-area assessments, NAEP administers survey questionnaires to students, teachers, and school administrators to put the assessment results into context. NAEP survey questionnaires have recently undergone substantial innovations with changes to the constructs captured and the measurement, design, and reporting approaches. All NAEP survey questions undergo an elaborate multi-year development process including small-scale cognitive interviews, large-scale pilots, and various expert reviews. During this process, each new or revised question is pre-tested in one-on-one interviews with trained interviewers before being administered in a pilot assessment. While cognitive interview techniques are widely established with adult populations, the NAEP populations (especially 4th-graders) pose additional challenges. This presentation will give an overview of the established pre-testing methods in NAEP to ensure that survey questions are valid, nonintrusive, and appropriate for the diverse NAEP student populations. We will especially highlight recent innovations for increased standardization of pre-testing methods, such as improvements to verbal probing techniques, extensions to the data capture and analyses based on eye-tracking, and utilization of cognitive interview for systematic item version comparisons. The presentation will give an overview of the procedures developed and established for NAEP and showcase the mixture of qualitative and quantitative methodology in evaluating item performance prior to large-scale piloting. We will draw on data from several large cognitive interview studies with several hundred participants conducted since 2014 and present best practices recommended for other testing programs.