Apply Resampling to Test for Differences in Educational Outcomes for Paired Cohorts Observed Over Several Years

• Activity Number: 256
• Type: Contributed
• Date/Time: Monday, August 10, 2015 : 2:00 PM to 3:50 PM
• Sponsor: Section on Statistical Education
• Author(s): William Goodman*
• Companies: University of Ontario Institute of Technology
• Keywords: Educational Evaluation ; Resampling ; Paired Summary Data ; Unbalanced Two-Factor Tables ; Statistical Literacy

Abstract

Evaluative reports of educational outcomes often compare results for different cohorts of students, distinguished by treatment or start date, observed over several academic years. A typical example compares average grades of traditional students at a university with those of students in some alternative pathway. Graphically or by a table, the two cohorts' grade averages are matched by their corresponding academic years, and apparent differences between groups are noted. Clear methods, however, are not presented in the literature to test the significance of such differences between cohorts as may appear. Comparing the cohorts' grade averages, overall, ignores the available by-year pairings. Yet it is groups, not individuals, that are paired; so paired t-tests, etc., fail to account for differing group n's and variances. A possible solution is an unbalanced ANOVA test, starting from summary data for each cohort-year combination; but commercial software does not support this. This paper presents and validates a simple resampling-based method for conducting the analysis described here. It then explores how this technique can be generalized to other applications.