Hierarchical Models for Assessing Reliability in Teacher Selection Instruments

• Activity Number: 252
• Type: Contributed
• Date/Time: Monday, August 10, 2015 : 2:00 PM to 3:50 PM
• Sponsor: Social Statistics Section
• Author(s): Patricia Martinkova* and Dan Goldhaber
• Companies: Institute of Computer Science AS CR/University of Washington and University of Washington
• Keywords: inter-rater reliability ; intraclass correlation ; mixed-effect models ; latent variable models

Abstract

Inter-rater reliability describes the extent to which there is consistency amongst two or more raters in assigned scores. Lower reliability implies a higher standard error of measurement, wider confidence intervals, lower correlations with other variables, and less powerful hypothesis tests. In scientific research, measurement is often fully crossed or nested to allow for straightforward estimates of inter-rater reliability. However, in organizational research, the data structure is often hierarchical and designs deviate substantially from the ideal. Models underlying reliability estimation are formally equivalent to mixed-effect/variance components models, and can be estimated by restricted maximum likelihood methods or by Bayesian methods such as Markov Chain Monte Carlo. Inter-rater reliability can be also addressed in the context of latent variable models. In this work, we use and compare these approaches for estimation of reliability of teacher selection instruments. We generally find low levels of reliability, though this overall reliability varies according to whether the measure is between or within schools and based on the type of position to which applicants are applying.