Unified Latent Class Modeling of Score and Rank Data Applied to Grant Panel Review

• Activity Number: 164 - Social Statistics Speed Session
• Type: Contributed
• Date/Time: Tuesday, August 10, 2021 : 10:00 AM to 11:50 AM
• Sponsor: Social Statistics Section
• Author(s): Michael Pearce* and Elena Erosheva and Steven Gallo
• Companies: University of Washington and University of Washington and American Institute of Biological Sciences
• Keywords: latent class modeling; preference learning with uncertainty; rank data; peer review; Mallows' distribution; EM algorithm

Abstract

Motivated by the setting of peer review, we develop a latent class framework to simultaneously model panel members’ scores and partial ranks for a collection of proposals. Review panels consist of subject matter experts, with two assigned reviewers for each proposal. Except for conflicts of interest, each panel member scores each proposal. Assigned reviewers reveal their scores before discussion; other panel members score during discussion; everyone ranks their top proposals at the end. Our approach combines binomial distributions to model scores and Mallows’ distributions to model ranks through common parameters for each latent class to identify reviewer consensus groups and provide group-specific final rankings of the proposals that incorporate uncertainty. Using a computationally efficient EM algorithm, we measure the strength of consensus within each group, quantify the effect of peer pressure from assigned reviewer scores on the remaining panelists, and discuss how inconsistencies between scores and ranks impact the final estimated rankings. We demonstrate the model and performance of the proposed estimation procedure on both simulated and real data.