Assessing Research Outcomes, Impact, and Innovativeness Through Senior Scientist Review: An Alternative Approach to Bibliometrics

• Activity Number: 234 - Bayesian Conditional Models and Updates
• Type: Contributed
• Date/Time: Monday, July 31, 2017 : 2:00 PM to 3:50 PM
• Sponsor: Section on Bayesian Statistical Science
• Author(s): Kassie Fronczyk and Justin Mary*
• Companies: Institute for Defense Analyses and Institute for Defense Analyses
• Keywords: Bayesian ; Bayesian multi-rater model ; survey methodology ; Program Evaluation ; Biomedical Research Evaluation

Abstract

IDA's Science and Technology Policy Institute (STPI) was tasked with evaluating the National Institutes of Health's (NIH) high risk, high reward funding mechanism intended to launch promising young scientist's careers and promote innovative research. One part of this evaluation was to compare the extent to which awardee research funded through this high innovation (HI) mechanism was more risky, creative, innovative, and impactful than research produced by a matched-comparison group of awardees funded through the traditional R01 mechanism (Matched R01s).

HIs and Matched R01s were compared using a senior scientist review, whereby experts provided assessments of research outputs for both groups. Seventy-eight experts were each given three sets of journal articles by 87 awardees and asked multiple questions about the impact, innovativeness, and creativity of the papers. These quantitative assessments were analyzed with a Bayesian multi-rater model, and resulting inference concludes that HIs had better overall assessments from senior scientists than the Matched R01s.