Refinement: Measuring Informativeness of Ratings in the Absence of a Gold Standard

• Activity Number: 277 - ASA Student Paper Competition Winners
• Type: Topic-Contributed
• Date/Time: Wednesday, August 11, 2021 : 1:30 PM to 3:20 PM
• Sponsor: Government Statistics Section
• Author(s): Sheridan Lloyd Grant* and Marina Meila and Elena Erosheva and Carole Lee
• Companies: University of Washington and University of Washington and University of Washington and University of Washington
• Keywords: Ratings; Entropy; Peer Review; Decision-Making

Abstract

We propose a new metric for evaluating the informativeness of a set of ratings from a single rater on a given scale. This metric, refinement, is particularly useful when there is no widely accepted gold standard for the quality or utility of the decision outcomes produced by ratings.

We use the tension between entropy as a measure of information and the tendency of raters to round to develop an information-theoretic measure of the refinement of a set of ratings---Entropic Refinement---as well as two secondary measures. A mathematical analysis of the three measures reveals that only the first, which directly measures the information content of the ratings, possesses properties appropriate to a refinement metric. In particular, Entropic Refinement can be directly formulated as the information content of scores that are “in competition” with one another in the rounding process. We finish by analyzing refinement in real-world grant-review data from the American Institute of Biological Sciences, finding evidence that merit ratings---which score the overall scientific quality of the proposal---are more refined than criterion ratings that measure specific aspects of proposals.