Reforms to CIHR’s Open Suite of Programs and Peer Review Process: Knowledge Synthesis Grant Pilot: Questions and Answers

What is the Knowledge Synthesis Grant pilot?

The Knowledge Synthesis Grant pilot will use the Fall 2013 Knowledge Synthesis Grant competition to pilot and test the validity and functionality of key design elements related to application and peer review processes for the new CIHR Project Scheme. This pilot introduces a new structured application process and a new structured review process (including new structured review criteria, a new rating and ranking process, remote review, and an online tool that would enable reviewers to discuss applications).

For more information on the reforms to CIHR’s open suite of programs, including the proposed pilots, please see Designing for the Future, The New Open Suite of Programs and Peer Review Process.

Why is CIHR conducting this pilot?

The objectives of the pilot are: (1) to test key components of the proposed Project Scheme in a live grant competition, and (2) to ensure that the principles behind the Synthesis Grant and other open Knowledge Translation programs are effectively integrated into the new Project Scheme design.

Will the current application process for the Knowledge Synthesis program change?

Yes. While applicants will still apply using the ResearchNet platform, the application will be structured to align with the Project Scheme’s proposed review criteria. Results from the pilot will help to ensure that this new template is appropriate for the Project Scheme.

The application process is described in detail in the “How to Apply” section of the Knowledge Synthesis Grant: Fall 2013 competition funding opportunity.

Will this pilot competition require an integrated Knowledge Translation approach?

Yes. An integrated Knowledge Translation approach is required to meet the objectives of the Knowledge Synthesis program. This program requires a partnership between researchers and knowledge users who work together to shape the research process. The importance of including this approach is communicated in the Knowledge Synthesis Grant: Fall 2013 competition funding opportunity as well as in the instructions to reviewers, Interpretation Guidelines for the Fall 2013 Knowledge Synthesis Grant Competition Adjudication Criteria.

How will the current review process for the Knowledge Synthesis Grant program change?

The review process will change significantly. There will be five different reviewers assigned to each application at Stage 1 and Stage 2: two academic reviewers, two knowledge user reviewers, and one reviewer who may be either a researcher or knowledge user.

Both a structured application process and a structured review process will be introduced, together with a new adjudication scale. Expert reviewers will conduct their reviews remotely at Stage 1, and will be supported by an asynchronous online discussion tool to inform their independent assessments. Reviewers will rate and rank their assigned applications and provide a written report. CIHR will consolidate all individual reviewer rankings into an average ranking for each application and provide a combined ranking list for the competition. Stage 2 review will involve a face-to-face committee meeting supported by electronically enabled technologies. The multi-disciplinary committee comprised of researcher and knowledge-user reviewers, will be responsible for integrating the results of the Stage 1 reviews. The discussion will focus on determining which applications to fund, with particular emphasis on applications with large variances in independent reviewer rankings. This committee will make final recommendations on which applications to fund, by creating a rank list of the applications.

Will there be any changes to the evaluation criteria?

Yes. The previous merit review criteria will be replaced with the new structured criteria established for the Project Scheme. Instructions to reviewers will emphasize the importance of assessing each criterion, and include guidance relevant to the objectives of the Knowledge Synthesis program. For further information see the Interpretation Guidelines for the Fall 2013 Knowledge Synthesis Grant Competition Adjudication Criteria.

Reviewers will not assign a separate score for scientific merit and potential impact as they have previously done in merit review. These concepts have been integrated into the new Project Scheme criteria. It is expected that the inclusion of both researcher and knowledge user reviewers will achieve a comprehensive review of the factors critical for success of the proposal.

For more information on the adjudication criteria, please see Designing for the Future, The New Open Suite of Programs and Peer Review Process, Project Scheme: Overview of the Mechanics.

How will this pilot impact the outcome of my Knowledge Synthesis Grant application?

The pilot will not compromise the fairness and transparency of the review process. All applications to the competition will receive the same standard of high quality review.

Will the results of the pilot be shared with the research community?

Yes. CIHR is committed to sharing the findings of the reforms pilots with the research community. Details about the pilot will be released once the competition is completed and the findings are analyzed.