|
Abstract:
|
Many millions of dollars are allocated and awarded to proposals based on reviewer ratings. Several biases may occur in the process: 1) not all reviewers can review all proposals, so the combination of reviewers assigned affects the scores; 2) where each review occurs in the reviewer's schedule may affect its rating; and 3) the same combination of reviewers may review a number of proposals confounding reviewer and proposal effects. In assigning reviewer schedules, we even the periods, e.g., early/middle/late, over which each proposal is reviewed and maximize pairs of reviewers reviewing proposals. In the analysis, we use a main effects model to separate consistent reviewer bias and the proposal. We simulate the resulting rankings and report the uncertainty of the rankings. These are simple statistical techniques, but they have provided a good basis for discussions and decisions allocating resources for the U.S. Fish and Wildlife Neotropical Grant Program over the last six years.
|
