Error Variance Estimation in Linking Educational Survey Test Results Through Population Ability Distribution Matching

• Activity Number: 226 - Recent Developments in Statistical Methodology for the National Assessment of Educational Progress
• Type: Topic Contributed
• Date/Time: Monday, July 31, 2017 : 2:00 PM to 3:50 PM
• Sponsor: Social Statistics Section
• Author(s): Xueli Xu* and John Donoghue
• Companies: Educational Testing Service - Princeton, NJ and ETS
• Keywords: NAEP ; sampling error ; linking error ; jackknife ; moderation linking

Abstract

The National Assessment of Educational Progress (NAEP) is an educational survey measuring academic proficiency of 4th, 8th and 12th graders. Prior to 2015, NAEP assessments used paper and pencil format (paper-based assessment PBA). Starting in 2015, an experiment was conducted using computer delivery (digital-based assessment, DBA). Comparisons of the two modes are of keen interest. This study evaluates two jackknife-based approaches to estimating standard error (SE) of reporting statistics. DBA and PBA are administered to nationally representative samples, but differ in scale of measurement. Based on randomly equivalent groups, DBA linked (linearly transformed) to PBA. A key concern is quantifying the SE for reporting statistics, such as subgroup means, in the presence of this kind of linking. A simulation study compares two methods of estimating SE. In option 1, the usual SEs are obtained, ignoring the effect of linking. In option 2, a modified SE explicitly considers the effect of linking. Option 1 usually underestimates the error variance for DBA subgroup means; while option 2 gives strikingly better estimates for both within-year reporting statistics and trend comparisons.