Multimodal Likelihoods in Educational Assessment: Will the Real Maximum Likelihood Score Please Stand Up?

• Activity Number: 294
• Type: Contributed
• Date/Time: Tuesday, August 5, 2003 : 2:00 PM to 3:50 PM
• Sponsor: Section on Statistical Computing
• Author(s): Werner Wothke*+ and George Burket and Li-Sue Chen and Furong Gao and Lianghua Shu and Richard J. Patz and Aviva Lev-Ari
• Companies: CTB/McGraw-Hill and CTB/McGraw-Hill and CTB/McGraw-Hill and CTB McGraw-Hill and CTB/McGraw-Hill and CTB/McGraw-Hill and CTB/McGraw-Hill
• Keywords: maximum likelihood ; multi-modal likelihood ; function maximization ; itemrResponse theory ; educational testing ; efficient computing

Abstract

It has long been know that psychological and educational tests, due to guessing of multiple choice questions, may exhibit multimodal or flat likelihood functions, or both, of a respondent's ability. These conditions can introduce uncertainty and bias to ability estimation in production environments when standard Newton maximization is employed. This paper evaluates the performance of several smart (and not so smart) maximization methods, including initial (grid) searches probing the function slopes, simulated annealing, exhaustive likelihood evaluations, and two implementations of the standard Newton algorithm. In extensive studies, involving several hundred thousand records of both generated and real data, the algorithms were evaluated with respect to precision and speed. Three methods, exhaustive search, simulated annealing, and grid search followed by Newton steps, all yielded ML estimates at the required precision. The algorithm combining grid search with Newton iterations was found to be faster by more than an order of magnitude.