The Impact of CIHR Reforms on Peer Reviewer Workload

Canadian Institutes of Health Research

December 2016

Executive Summary

This analysis used data from post-competition reviewer surveys to assess whether CHIR funding reforms impacted the time required to review applications. On a per-application level, post-reform competitions required less time to assess proposals than the pre-reform OOGP competitions. The post-reform competitions have also required less total review time and less total time from each reviewer than the pre-reform competitions.

Purpose

The purpose of this analysis is to assess what effects, if any, reforms to CIHR funding programs had on the time required to review funding applications.

Data

Data for time spent reviewing was self-reported by peer reviewers and was collected in post-competition surveys.

The competitions included in this analysis were:

• Pre-reforms – 201209MOP, 201303MOP, 201309MOP, 201403MOP
• Post-reforms – 201409FDN, 201509FDN, 201603FDN

Analysis

This work sought to assess two elements of peer review time:

1. How much time did it take, on average, for a reviewer to review one application pre and post reforms (i.e. the time per application)
2. Given that a reviewer was usually responsible for reviewing multiple applications, how much total reviewer time was required pre and post reforms (i.e. the time per reviewer).

Time per Application

A peer reviewer required, on average, 7.9 hours to review a single application in the OOGP competitions included in this analysis (Table 1). This total consisted of an average of about 3.5 hours reading the application, a little over 2 hours writing their review, and over 2 hours in meetings to discuss that application. Note that, to enable comparisons between different peer review phases, which may or may not involve travel to a meeting, travel time is excluded from the analysis herein. However, based on survey responses, each reviewer spent an average of almost 8 hours travelling for the OOGP, which breaks down to approximately 49 minutes per application reviewed.

The CIHR reforms created competitions with multiple review stages, each requiring different tasks from the peer reviewers. Consequently, different surveys were utilized to capture the time spent reviewing applications for each. Surveys for Stage 1 of the Foundation and Project competitions and Stage 2 of the Foundation competition asked about time spent reading, looking up additional information related to the applications, writing their review, reading other reviewers’ preliminary reviews, participating in online discussion, and completing a ranking of their assigned applications. Surveys for the Final Assessment Stage of the Foundation and Project competitions asked about time spent reading the Stage 2 application reviews, consulting the Stage 2 grant applications, looking up additional information related to the applications, assigning applications to ‘yes’ and ‘no’ bins and writing comments to justify these assessments, reading other Final Assessment Stage reviewer comments on ResearchNet, and reviewing the Final Assessment Stage ranking to prepare for the Final Assessment Stage committee meeting.

In the two Foundation competitions analyzed, it took an average of almost 3.5 hours to review an application in Stage 1, and about 5 hours in Stage 2 (Table 2). Reviewers took approximately 5 hours to review an application in Stage 1 of the 2016 Project competition.

In the two Foundation competitions analyzed, reviewers reported spending an average of 1.5 hours performing their Final Assessment Stage review tasks (Table 3). For the 2016 Project competition, these tasks took reviewers an average of almost 3 hours per application.

The fact that post-reform competitions utilized multiple stages, with varying strategies used to limit those applications proceeding to the next stage according to the competition, complicates the interpretation somewhat. An application that advanced to the final stage of the two Foundation competitions required an average of 9.7 hours, which is greater than the 7.9 hours required to review an application in the OOGP competitions. However, all applications in the OOGP competitions required an average of 7.9 hours to review, whereas applications in the Foundation competitions that did not advance past the first or second stage required an average of 3.4 hours or 8.2 hours of review time, respectively. To assess whether implementing multiple stages decreased review time requirements, Table 4 displays weighted averages of the time required to review an application for the pre-reform and post-reform competitions.

In the Project competition, for example, 3,713 of the 3,813 applications submitted did not make it past Stage 1 and thus only required 4.8 hours to review. The 100 applications that proceeded to the FAS stage required the 4.8 hours needed in Stage 1 as well as the 2.7 hours required in the FAS stage for a total of 7.5 hours per application. This results in a weighted average for the Project competition of 4.9 hours per application. Based on the data in Table 4, the implementation of CIHR funding reforms resulted in a decrease in the average amount of time required to review a single application. 

Time per Reviewer

Although the CIHR funding reforms have decreased the amount of review time required to assess a single application, reviewers are recruited and asked to review a package of applications. It is possible, then, that the decreased review time for a single application may not have translated to a decrease in the overall demand being placed on reviewers.

Table 5 displays summary data from reviewer surveys for the OOGP, Foundation, and Project competitions regarding the number of applications they reported being assigned to review. Stage 2 of the Foundation competition is the only post-reform stage where the mean number of applications assigned was lower than the pre-reform OOGP standard, however the mean number assigned for Stage 1 of the Project competition was only slightly higher than the OOGP number (10.1 vs. 9.8).

Total reviewer burden for each stage of the review process was calculated by multiplying the average number of applications assigned to each reviewer against the average time required to review a single application (taken from Table 4). Reviewers reported being asked to spend an average of 77 hours reviewing applications for the OOGP competition (Table 6). This total review time decreased to averages of about 44 hours for Stages 1, 2, and 23 hours for the Final Assessment Stage of the Foundation competitions. Likewise, reviewers were asked to spend an average of 48 and 42 hours reviewing applications for Stage 1 and the Final Assessment Stage of the Project competition.

The total reviewer time required for each stage was calculated by multiplying the time required by each reviewer against the total number of reviewers for that stage. These values were summed to determine the reviewer workload for a competition. The OOGP competitions required, on average, over 75,000 hours of reviewer time. Post reforms, the Foundation competitions have required approximately 28,000 hours of reviewer time with the 2016 Project competition having required almost 58,000 hours of reviewer time (Table 6).

To compare the three competitions, the total reviewer burden was divided by the average number of applications submitted to the competitions. The OOGP competitions required 30.3 hours of reviewer time per application submitted (Table 6). Post reforms, the two Foundation competitions required 24.8 reviewer hours per application, whereas the Project competition requirements have decreased to 15.1 hours per application.

Conclusions

Based on self-reported peer reviewer survey data:

• The time required to review a single application was lower at every stage in post-reform competitions than in the pre-reform OOGP competitions.
• The multi-stage review process implemented in the reforms resulted in longer total times to review an application to the final stage in the Foundation competition and did not significantly change the total time required to review an application to the final stage of the Project competition, however:
  • The post-reform process of winnowing out applications at successive stages resulted in an overall decreased average total review time per application.
  • Less total reviewer time was required in post-reform competitions than in the pre-reform OOGP competitions analyzed.
  • Both the Foundation and Project funding competitions required less average time per reviewer than the OOGP competitions.

Limitations of the Analysis

1. Conclusions were drawn based on self-reported time-spent data.

   Surveys were administered post-competition, meaning reviewers had to recall their experience in order to answer questions. Recall bias is a well-known confounder in the collection of survey data^{Footnote 1}, and the results can only be trusted to the extent that we trust the reviewers’ ability to remember and report accurately. Additionally, survey respondents were self-selected and not randomly sampled. This makes it difficult to generalize these results to the entire review cohort.

2. Different surveys were used pre and post reforms.

   • Pre-reform surveys asked peer reviewers to report on time spent reading applications, writing reviews, meeting, and traveling.
   • Post-reform surveys asked peer reviewers about time spent reading applications, looking up additional information related to the applications, writing reviewers, reading other reviewers’ reviews, participating in discussions, and completing rankings of their assessments.

   The implication of these differences is that the ‘Total Time’ spent reviewing sums a different set of elements for each competition, which could in turn affect the reviewer’s perception of the total time required to review an application. Therefore, we suggest that conclusions drawn from the results herein be made with this reservation in mind.

3. Some survey questions asked reviewers to report time spent per application, and other questions asked to report total time.

   Because of this, the data had to be manipulated prior to combining the individual activities into a total time. In order to calculate time spent on a per-application basis, the ‘total time spent meeting’ for OOGP reviewers had to be divided by the number of applications reviewed to obtain a ‘per-application’ number. This number may have been different than what we would have obtained had we asked participants to report the time they spent per application meeting about their assessments.

4. Analysis involving extrapolation of survey results to entire competitions (Tables 4 and 6) did not incorporate variance into the calculations.

   Without taking variance into account, statistical conclusions cannot be reached using these numbers. The multiplication and division of variances was beyond the scope of this analysis, and these numbers were presented for consideration.

Methodological Notes

• Extreme outliers in the survey data were identified using the Tukey Method (1.5 x IQR) and were excluded from further calculations.
• Post-reform surveys included responses from Stage 1 and 2 Reviewers, and from Final Assessment Stage Reviewers.
• The total number of reviewers and the total number of applications reviewed for the OOGP competitions was obtained from SSRE data.
• The total number of reviewers and the total number of applications reviewed at each stage of the Foundation and Project competitions was obtained from the ‘Reviewer Workload Analysis’ slide decks prepared by Ioana Dobre

Date modified:

2017-05-30