JSM 2015 Online Program

Online Program Home
My Program

Abstract Details

Activity Number: 252
Type: Contributed
Date/Time: Monday, August 10, 2015 : 2:00 PM to 3:50 PM
Sponsor: Social Statistics Section
Abstract #316570
Title: Hierarchical Models for Assessing Reliability in Teacher Selection Instruments
Author(s): Patricia Martinkova* and Dan Goldhaber
Companies: Institute of Computer Science AS CR/University of Washington and University of Washington
Keywords: inter-rater reliability ; intraclass correlation ; mixed-effect models ; latent variable models

Inter-rater reliability describes the extent to which there is consistency amongst two or more raters in assigned scores. Lower reliability implies a higher standard error of measurement, wider confidence intervals, lower correlations with other variables, and less powerful hypothesis tests. In scientific research, measurement is often fully crossed or nested to allow for straightforward estimates of inter-rater reliability. However, in organizational research, the data structure is often hierarchical and designs deviate substantially from the ideal. Models underlying reliability estimation are formally equivalent to mixed-effect/variance components models, and can be estimated by restricted maximum likelihood methods or by Bayesian methods such as Markov Chain Monte Carlo. Inter-rater reliability can be also addressed in the context of latent variable models. In this work, we use and compare these approaches for estimation of reliability of teacher selection instruments. We generally find low levels of reliability, though this overall reliability varies according to whether the measure is between or within schools and based on the type of position to which applicants are applying.

Authors who are presenting talks have a * after their name.

Back to the full JSM 2015 program

For program information, contact the JSM Registration Department or phone (888) 231-3473.

For Professional Development information, contact the Education Department.

The views expressed here are those of the individual authors and not necessarily those of the JSM sponsors, their officers, or their staff.

2015 JSM Online Program Home