The Occasionally High Type I Error Rate of the Mantel-Haenszel Statistic Used in Detecting Differential Item Functioning and an Empirical Bayes Solution

• Activity Number: 162
• Type: Topic Contributed
• Date/Time: Monday, August 2, 2010 : 10:30 AM to 12:20 PM
• Sponsor: Social Statistics Section
• Author(s): Sandip Sinharay*+ and Neil J. Dorans
• Companies: Educational Testing Service and Educational Testing Service
• Keywords: Differential item functioning ; Empirical Bayes ; Mantel-Haenszel ; Educational statistics ; psychometrics

Abstract

The Mantel-Haenszel (MH) procedure (Mantel & Haenszel, 1959) is a popular method for estimating and testing a common two-factor association parameter in a 2x2xK table. Holland (1985) described how to use the procedure to detect differential item functioning (DIF) in educational/psychological tests. Wang, Bradlow, Wainer, and Muller (2008) showed that the MH procedure often found DIF where there was none (that is, had a high Type I error rate) for easy items. They showed that their simulation-based Bayesian procedure for DIF detection overcomes this problem. We discuss a simpler solution of the problem, the use of the empirical Bayes procedure of Zwick, Thayer, and Lewis (1999), which is computationally much simpler than the procedure of Wang et al. We show, using results from real and simulated data, that the use of the empirical Bayes procedure leads to an acceptable Type I error rate.