Skip to main content
Article
Low agreement among reviewers evaluating the same NIH grant applications
Proceedings of the National Academy of Sciences of the United States of America
  • Elizabeth L. Pier, University of Wisconsin-Madison
  • Markus Brauer, University of Wisconsin-Madison
  • Amarette Filut, University of Wisconsin-Madison
  • Anna Kaatz, University of Wisconsin-Madison
  • Joshua Raclaw, West Chester University of Pennsylvania
  • Mitchell J. Nathan, University of Wisconsin-Madison
  • Cecilia E. Ford, University of Wisconsin-Madison
  • Molly Carnes, University of Wisconsin-Madison
Document Type
Article
Publication Date
3-20-2018
Abstract

Obtaining grant funding from the National Institutes of Health (NIH) is increasingly competitive, as funding success rates have declined over the past decade. To allocate relatively scarce funds, scientific peer reviewers must differentiate the very best applications from comparatively weaker ones. Despite the importance of this determination, little research has explored how reviewers assign ratings to the applications they review and whether there is consistency in the reviewers’ evaluation of the same application. Replicating all aspects of the NIH peer-review process, we examined 43 individual reviewers’ ratings and written critiques of the same group of 25 NIH grant applications. Results showed no agreement among reviewers regarding the quality of the applications in either their qualitative or quantitative evaluations. Although all reviewers received the same instructions on how to rate applications and format their written critiques, we also found no agreement in how reviewers “translated” a given number of strengths and weaknesses into a numeric rating. It appeared that the outcome of the grant review depended more on the reviewer to whom the grant was assigned than the research proposed in the grant. This research replicates the NIH peer-review process to examine in detail the qualitative and quantitative judgments of different reviewers examining the same application, and our results have broad relevance for scientific grant peer review.

Publisher
National Academy of Sciences
DOI
10.1073/pnas.1714379115
Citation Information
Elizabeth L. Pier, Markus Brauer, Amarette Filut, Anna Kaatz, et al.. "Low agreement among reviewers evaluating the same NIH grant applications" Proceedings of the National Academy of Sciences of the United States of America Vol. 115 Iss. 12 (2018) p. 2952 - 2957 ISSN: 0027-8424
Available at: http://works.bepress.com/joshua-raclaw/2/