|
Abstract:
|
The Center for Scientific Review at the National Institutes of Health has recently begun conducting short stakeholder surveys to assess the utility of study section meeting experience. Respondent sentiment is being evaluated on Peer Review related issues using Likert type scales and an open text box for general comments. In early stages of this effort, the number of respondents providing general comments was small enough to manually establish a taxonomy of categories and evaluate the specific polarity of each comment. In subsequent studies, as the population increased to over 10,000 respondents, manual content analysis became infeasible. To analyze thousands of textual comments, innovative computational linguistics is being used to automate the process of capturing and categorizing stakeholder responses as well as assessing the sentiment expressed in these responses. Initially, a set of linguistic models were trained using manually annotated text fragments from the first pilot study. Then, the predictive accuracy and reliability of the model was refined using data from subsequent studies. Results are being used to identify areas of success and improvement in peer review quality.
|
