Designing for the Future: The New Open Suite of Programs and Peer Review Process – Questions and Answers - Mechanics

• Rationale
• Architecture
• Mechanics
• Transition
• College of Reviewers
• Next Steps

1. How will CIHR be matching applications to expert reviewers?

Matching applications to appropriate reviewers is essential to implementing application-focused review. We are currently assessing various IT-based tools to assist with the matching processes. These tools are intended to support human judgment and experience in validating matching assignments. We envision that external experts will help support this process. A number of pilots are being designed to ensure these IT-based tools are effective and are appropriately integrated with expert guidance.

2. What was CIHR's rationale for including a remote (virtual) screening process?

CIHR is committed to implementing application-focused review, which is intended to match the most appropriate expert reviewers to individual applications in order to improve the quality and fairness of reviews. Given the number and different types of applications CIHR receives through its Open programs, application-focused review can only be logistically feasible and successful if supported by internet-assisted technology. The remote (virtual) screening process was proposed as a means to bring together the appropriate expertise to inform peer review, gain cost-effective access to a broader base of expertise (including international experts), reduce biases that occur in face-to-face discussions, and reduce the burden of travel demands imposed on peer reviewers' time.

3. What types of experts will be part of the face-to-face interdisciplinary committee(s) and how will they be selected?

The interdisciplinary committee(s) will consist of recognized leaders with a broad understanding of CIHR and the Canadian and international health research landscapes. CIHR envisions there will be one committee for the Foundation Scheme, and several committees for the Project Scheme. Project committees will be formed based on the different types of applications received in each competition.

These committees will assess "grey zone" applications from a portfolio perspective rather than an individual application perspective.

4. Will applicants have access to reviewer comments?

All applicants will receive structured feedback from each assigned reviewer at each stage of the multi-stage competition process.

5. Why are the adjudication criteria listed in Annexes A and B still in early draft form?

CIHR is currently working with a variety of stakeholders to further develop and refine details for the proposed adjudication criteria. Discussions are ongoing, and continue to shape the evolution of the proposed Project/Foundation Scheme adjudication review criteria.

6. It appears that the current OOGP committees will no longer exist in the new design. What are they being replaced with?

Given the number and diversity of applications CIHR receives, CIHR is moving towards an application-focused peer review process that will match applications to individual reviewers with the most appropriate expertise. However, CIHR values the work of its peer review committees, and recognizes the need to maintain face-to-face discussions as part of the new peer review process. It is CIHR's intention to make more judicious use of face-to-face committee meetings by using broader, multidisciplinary committees at the final assessment stage of the multi-stage competition process. Committee discussions will focus on "grey zone" applications, with particular emphasis on applications with large variances in independent reviewer rankings.

7. What role will Knowledge Users play in grant reviews? [ new ]

CIHR expects to include Knowledge users in the review of its applications to the new open funding schemes. Knowledge Users will be an important expertise pool in the College of Reviewers and have been given a high priority in the development of the College of Reviewers recruitment strategy. A strong Knowledge Users base will be critical to ensure that CIHR assigns the right mix of expertise in the review of CIHR applications; especially as CIHR expects to attract a diversity of programs and projects at any stage: discovery, demonstration, validation and/or application of health-related knowledge.

8. Can applicants nominate at least one expert reviewer to review their grant? [ new ]

Similar to the process in place for our current programs, CIHR will continue to offer applicants the opportunity to suggest reviewers (domestic or international) in their applications. These suggested reviewers are considered in the reviewer identification and assignment process. In the future, we recognize that the reviewers suggested by applicants may or may not be registered College members. In the latter case, applicants may use the College of reviewer nomination process to put forward their expert reviewer nomination. It is CIHR's intent to have a process in place to capture and fast track nominations made through the application process so that staff may have timely access to the appropriate reviewers.

9. Will reviewers still write reviews or just tick boxes? [ new ]

Reviewers will be required to provide written comments, in addition to alpha ratings, for each sub-criterion for each assigned application. Writing a formal, fully-articulated long review will not be required, although partaking in this exercise may be helpful to some as preparation for submitting formal comments to CIHR. In the spirit of reducing peer reviewer burden, reviewers will only be asked to provide their expert perspective on the key strengths and weaknesses.

10. How will the ranking of individual reviews be tabulated if the total number of applications assigned to a reviewer varies? [ new ]

The new adjudication process will be founded on relative quality as opposed to absolute scores. The relative ranking is applied at the individual reviewer level on their own pool of assigned applications. This approach is intended to address differences in reviewer behavior across disciplines, and provide greater consistency in peer review. Rankings from all assigned reviewers for each application will be consolidated into a final competition ranking. At the outcome of the competition, the top-ranked applications across a broad base of health-related research will be selected for funding.

11. As a new investigator I find the committee meetings helpful as a learning opportunity. What will happen to this opportunity when the committees are no longer in place? [ new ]

Because CIHR has not had any formal or comprehensive approach to the orientation and development of reviewers, the majority of reviewer learning experiences have been in the committee setting.

Through the College of Reviewers, new reviewers will have the opportunity to develop their peer review skills through an orientation and development program which will include mentorship opportunities as well as training modules targeting new reviewers. The mentorship program will allow new reviewers to be paired with Mentors from similar research areas/disciplines. The details of the mentoring program are under development and could involve, e.g., a new reviewer shadowing the review steps of a Mentor

The remote review stage of the new peer review process for the new schemes will offer a on-line discussion platform for interaction with other reviewers, which will include access to the reviews of other reviewers assigned to the same applications. CIHR anticipates that new reviewers will build their peer review skills and learn from other reviewers in such fora.

Furthermore, strategic competitions will continue to use panels/committee structure and would provide opportunities for new reviewers to learn in this setting.