Proposed Open and Peer Review Reform Questions and Answers

1. Why is CIHR developing a new Open Suite of Programs?

CIHR recognizes the importance of investigator-initiated research, and its role in sustaining the Canadian health research enterprise. We have initiated a process to design a new Open Suite of Programs and peer review system to ensure the long-term sustainability of CIHR's contribution to the Canadian health research enterprise to remove barriers, and to enable researchers from all pillars to improve CIHR's ability to deliver on its mandate.

Consultations with CIHR's health research community have shown that there are challenges with mechanics of the existing programs, specifically with CIHR's competition and peer review processes. These challenges were consistent with observations by the International Review Panel and by CIHR's own internal assessments.

Researcher and stakeholder communities continue to challenge us to streamline and strengthen CIHR's funding programs, to continue to reduce program complexity, and to reduce the need for researchers to spend large amounts of time writing and applying for grants to support their research programs.

Also all too common are researcher concerns that CIHR's current peer review committee structure and peer reviewers fail to adequately accommodate research across all of CIHR's pillars, new and evolving areas of research, and paradigm-shifting research. Our peer review system and processes strain CIHR's capacity to recruit peer reviewers from a broad base of expertise to ensure that the right expertise is engaged in reviewing the spectrum of grant applications received. At the same time, growing application pressure and the complexity of many applications have meant that potential peer reviewers increasingly express reluctance to volunteer for the heavy workload.

2. How many researchers is CIHR currently supporting? How many will be supported in the design?

CIHR currently funds some 14,000 researchers and trainees, which include a mix of nominated principal investigators, principal investigators and co-applicants. Of the approximately 9,000 researchers funded through the current Open Suite of Programs, approximately 3,000 are unique nominated principal investigators. In designing the new Schemes, Science Council emphasized the importance of continuing to support a similar number of principal investigators.

3. What evidence did CIHR consider in this design?

In developing this new design for the Open Suite of Programs, CIHR took into consideration the literature on research funding and peer review; examined national and international research funding programs; and engaged in thoughtful reflection on what new design elements would best work for health researcher and stakeholder communities and for CIHR. We have listened closely to our stakeholders, and based on our own analyses and the existing evidence, have identified a number of changes that could transform the current Open Suite of Programs to better deliver on the full spectrum of CIHR's mandate.

In reviewing what evidence is available, CIHR found that there is not a large base to draw upon in this area. Through designing and implementing the new Open Suite of Programs, there is an opportunity to contribute to the evidence base for funding program and peer review design. A research plan will be developed in the coming months.

Annex 1 of the design discussion document. lists the sources that CIHR used to inform the proposed design of the new Open Suite of Programs.

4. How will the new Open Suite of Programs be structured?

CIHR is considering a suite of two separate, yet complementary funding schemes:

1. A Foundation/Programmatic Research Scheme, which will provide long-term support both to established investigators with demonstrated track records of success, and new/early career investigators with excellent training and early-career productivity, to pursue innovative, high impact programs of health research; and, enable integrated knowledge translation. Flexible, long-term funding will also provide top researchers with the opportunity to pursue novel and/or emergent avenues of health research with a less frequent requirement for grant renewals.
2. A Project Scheme, which will provide support for defined projects with a beginning, a middle and definite end point that capture the best original, innovative and/or impactful ideas across the spectrum of health and health system research and knowledge translation. This scheme is open to both established and promising new/early career researchers.

It should be noted that through the years, although the Open Operating Grants Program had been set up as a project-based mechanism, researchers have been creating programs of research using this mechanism by coordinating a series of individual Project grant proposals and renewals.

The programmatic research and project-based funding mechanisms are not intended to be viewed as part of a continuum. They are two distinct funding streams that support two different approaches to funding health research: through programs of research that build towards long-term health research goals; and, through projects that answer specific health research questions and have a defined end point. CIHR believes that both types of funding schemes are required to meet its needs, and will provide applicants with the flexibility to choose the most appropriate type of mechanism for the type of research being conducted and their personal research style, irrespective of research domain and/or pillar. The intent is to merge as many of the existing small, purpose-built funding mechanisms as possible, including the Open Operating Grants Program, into these two Schemes.

5. If I only have one grant today does that exclude me from the Program Scheme?

No, the Foundation Scheme is about supporting people. It is about providing longer-term support to investigators with a demonstrated track record of success. There is a cadre of excellent researchers in Canada that come back to CIHR year after year for funding and are consistently successful. We want to reduce their burden of writing grants and increase the time dedicated to doing research and translating results. We want to give them the freedom to create, change, and redirect their efforts as required. We want them to be supported by their institutions with the necessary resources and protected time to do research. We also want them to mentor and develop the next generation of researchers. The assessment criteria in this scheme would be based on the caliber of the applicants and the vision articulated for the proposed program of research. For some researchers, this will be a big change – we are asking you to think about your work programmatically. We will ask reviewers to be less focused on your specific methodology for a specific project and more on your overall approach for a series of research endeavors – some of which won't be defined at the time of application.

6. Can teams apply for funding under the Foundation/Programmatic Research Funding Scheme?

CIHR recognizes that research is increasingly being conducted by groups of researchers, and has heard concerns over the eligibility of applications with multiple leads. The current thinking on this scheme is that it should be flexible enough to support both individuals and groups of researchers who are committed to work together for a long period of time on a program of research. CIHR is currently in the process of determining how to best assess these types of applications within the context of the proposed competition process for the Foundation/Programmatic Research Scheme.

7. Can Foundation/Programmatic Research grant holders also apply for Project grants?

In the current model, principal Foundation/Programmatic Research grant holders (i.e., Nominated Principal Investigators and other lead Principal Investigators peer reviewed in the competition) would not be able to apply for a Project grant.

8. How will the proposed design help reduce applicant burden?

The longer duration and larger value of grants awarded through the Foundation/Programmatic Research Scheme is intended to reduce applicant burden and give greater flexibility to investigators. Successful applicants would not need to write multiple research proposals to competitively fund their research programs, nor would applicants need to apply for funding as frequently.

To further reduce applicant burden, applications will be shorter and more focused on the relevant information needed to support decision-making. CIHR is considering implementing multi-phased competition processes for both schemes. The intent is to focus reviewer attention on specific criteria at different points in the process. This would be supported by structured review to minimize inconsistent and inappropriate application of review criteria and to improve transparency of the review process. Both multi-phased competition processes and structured review will help manage applicant and reviewer burden by reducing the number of applicants who move on to full application and by reducing the length of time it will take to review applications at each stage.

9. How do you define new/early career investigators?

In the context of the proposed design, CIHR defines a new/early career researcher as an applicant who has either never applied before to CIHR, or whose last degree ended five years or less before the original competition date.

10. What kind of support will be available for new/early career investigators?

Both project-based and programmatic funding schemes are being designed to include features to support new/early career investigators.

CIHR's Project Scheme is considering focusing the first stage of review on the quality of the idea with limited information about the track record of the applicant. This would remove some biases/barriers (real or perceived) for new/early career investigators.

For the Foundation/Programmatic Research Scheme, CIHR is considering a specific stream to support new/early career investigators with excellent training and early-career productivity to ensure that these researchers have an opportunity to build promising programs of research and knowledge translation.

11. What do you mean by institutional commitment in the Foundation/Programmatic Research Scheme?

Institutions play a critical role in supporting health research and share with CIHR a common interest in supporting research success. Recently, we have heard from the research community that there is a need for institutions to provide more support for the conduct of research. Concerns over teaching loads and protected time for research varied within and across academic institutions, as well as across different pillars of research.

To ensure that researchers are able to successfully execute their programs of research, CIHR's new Foundation/Programmatic Research Scheme would require applicants to secure a formal commitment from their institution to provide significant support. This support may include providing the necessary time release, infrastructure, resources, knowledge translation support, training, salary and career development support to investigators over the duration of the Foundation/Programmatic Research grant awarded. CIHR recognizes that there are growing demands on the resources available to Institutions, and that there may be significant costs associated with providing these types of support. CIHR will need help from each institution to identify what constitutes a sufficient amount of support for the Foundation/Programmatic Research and Project schemes, and how this support can be integrated into CIHR's funding scheme design.

12. What do you mean by integrated knowledge translation? Is it mandatory in the new schemes?

According to the CIHR definition, in integrated KT, stakeholders or potential research knowledge users are engaged in the entire research process. Researchers and research users work together to shape the research process by collaborating to determine the research questions, deciding on the methodology, being involved in data collection and tools development, interpreting the findings, and helping disseminate the research results. This approach, also known by such terms as collaborative research, action-oriented research, and co-production of knowledge, should produce research findings that are more likely be relevant to and used by the end users.

As part of the new design for the Open Suite of Programs, both the Foundation/Programmatic Research and Project funding schemes would encourage and researcher to pursue an integrated KT approach to their work where it makes sense and could increase the potential for impact.

13. Where does training fit within the two schemes?

Both the Foundation/Programmatic Research and Project Schemes are will undoubtedly support the training and mentoring of promising trainees, much as the Open Grants Program does today. However, applications to the Foundation/Programmatic Research Scheme will be required to include a comprehensive mentoring and training plan that will be evaluated in peer review.

For their part, CIHR's direct training programs (Frederick Banting and Charles Best Canada Graduate Scholarships - Master's and Doctoral Awards Programs; Doctoral Foreign Study Award Program; Vanier Canada Graduate Scholarships Program; CIHR Fellowships Program; and Banting Postdoctoral Fellowships Program) are not changing, and will continue to be a part of CIHR's open research funding strategy to support a sustainable pipeline of talented new health researchers into the health research enterprise.

14. How will the two schemes support high-risk/high-reward research?

Feedback from the research community suggests that current committee structures, favour established approaches. In an environment where only a small proportion of applications are funded, there is less incentive and comfort to accept riskier, unproven areas of research. These factors lead to a conservative peer review system.

As part of the design considered for the new Open Suite of Programs, CIHR will be building in features for project-based and programmatic funding schemes to address this challenge. The Project Scheme attempts to remove track-record bias from Stage 1 of the review process.

Conversely, one of the benefits of the Foundation/Programmatic Research Scheme is sufficient, stable and long-term funding. This would allow researchers to pursue high risk and innovative ideas without the constant threat of grant renewal.

15. Will there be an opportunity for Foundation/Programmatic Research grant holders to request funding increases mid-stream?

A mid-term review process to evaluate progress is being considered. Budgetary adjustments, if needed, may be considered as part of the review. Foundation/Programmatic Research grants are intended to support the direct costs of research and do not include a salary support component.

16. How will the proposed design reduce peer review burden?

In designing the new Open Suite of Programs, CIHR aims to implement design elements that would reduce the overall time a reviewer spends reviewing, discussing, and providing feedback on an application. To achieve this, CIHR is considering a number of design elements:

1. A multi-phased competition process that involves a two-stage screening process prior to face-to-face review. Stage 1 applications are short, and would be designed to be reviewed in a relatively short period of time. Peer reviewer burden will also be actively managed by controlling the number of applicants invited to participate in Stage 2, where a more detailed application is required.
2. Structured review criteria would provide peer reviewers with clearly defined review criteria and relevant application information in order to evaluate success. This would help reduce the amount of time reviewers spend analyzing an application package, and providing free form feedback to applicants.
3. Conducting screening reviews and conversations in a virtual space (internet-assisted discussions) would help reduce reviewer burden by alleviating the need for reviewers to travel to Ottawa, freeing time to conduct research or engage in other activities. This model also would also facilitate international expert review by supporting cost-effective access to international research leaders.

As part of a multi-phase competition process, we want to be thoughtful about how and when to use face-to-face committee meetings. We know that today, a lot of committee time is spent discussing applications that everyone agrees should be funded or that everyone agrees have fatal flaws. There is, however, always a "grey zone" – where reviewer views are varied for a number of different reasons. The proposal is to leverage the strengths of structured review and available technology to encourage early recognition of outstanding applications and triage of non-competitive applications. The intent is to concentrate face-to-face discussions in fewer, broader, more multi-disciplinary committees on these "grey zone" applications.

Each design element highlighted has the potential to reduce the burden felt by applicants and peer reviewers.

17. How many competitions per year will there be for both schemes?

Details about the competition timing for the Foundation/Programmatic Research Scheme are still being discussed. Current thinking suggests that there would be one Foundation/ Programmatic Research competition per year and one or two Project competitions per year.

18. Will CIHR continue to use face-to-face committee meetings?

It is CIHR's intention to make more judicious use of face-to-face committee meetings as a mechanism to integrate the results of remote reviews and determine the final recommendation for funding. The primary focus of face-to-face committee reviews would be on the applications that require further discussion, especially with respect to rankings near the funding cut-off (i.e. the grey zone). We know that today, a lot of committee time is spent discussing applications that everyone agrees should be funded or that everyone agrees have fatal flaws. There is, however, always a "grey zone" – where reviewer views are varied for a number of different reasons. The proposal is to leverage the strengths of structured review and available technology to encourage early recognition of outstanding applications and triage of non-competitive applications. The intent is to concentrate face-to-face discussions in fewer, broader, more multi-disciplinary committees on these "grey zone" applications.

19. Why is CIHR considering an increased use of technology in peer review?

With the recent advances in technology and social media, there is an opportunity to use technology to support enhancements to the peer review process, including the facilitation of application-focused review. CIHR is looking to bring reviewers together in a virtual space supported by mechanisms that facilitate internet-based discussions ("virtual peer review") similar to what the National Institute of Health's Centre for Scientific Review is currently implementing. Reviews, for the most part, would be individual. However, reviewers would be given an opportunity to discuss, justify and exchange perspectives to inform the substance of their final individual reviews of applications. Using this technology, CIHR could gain broader access to the required expertise, including international experts, and bring together multiple perspectives to inform peer review decisions.

As part of a multi-phase competition process, we want to be thoughtful about how and when to use face-to-face committee meetings. We know that today, a lot of committee time is spent discussing applications that everyone agrees should be funded or that everyone agrees have fatal flaws. There is, however, always a "grey zone" – where reviewer views are varied for a number of different reasons. The proposal is to leverage the strengths of structured review and available technology to encourage early recognition of outstanding applications and triage of non-competitive applications. The intent is to concentrate face-to-face discussions in fewer, broader, more multi-disciplinary committees on these "grey zone" applications.

20. How will CIHR transition current grant holders to the new schemes?

CIHR's Transition Plan for phasing in the new Open Suite of Programs and proposed Peer Review enhancements is currently in its early stages. It is difficult to plan out the transition strategy until the design is finalized. However, CIHR is committed to ensuring that the transition to a new Open Suite of Programs occurs with minimal disruption.

Work is underway to model various transition scenarios related to the implementation of the Foundation/Programmatic Research / Project Schemes, the phasing out of existing program competitions, and the management of ongoing grantees. Integrated in the transition planning is the development of a robust monitoring and evaluation system to ensure continuous quality improvement for the new system, to which CIHR is committed.

21. Will CIHR conduct pilots or phase-in the changes in progressive steps?

Yes, CIHR will, as part of its transition planning, introduce changes in a managed fashion. This will be important, not only for a smooth transition for the research community, but also for piloting some of the more complex changes. Such pilots will serve to evaluate, demonstrate and/or validate aspects of the proposed design prior to full implementation. This managed approach reflects the feedback received to date from the community and forms an integral part of CIHR's Research Plan aimed at building the evidence base for program design and peer review models. Both a Transition plan and the Research Plan are under development.

22. What is the implementation timeline?

In June, we will provide the community with a summary of the feedback received through the engagement process, including how we are planning to use the feedback to inform the design. No firm design decisions will be announced until the fall of this year. Applicants and reviewers would be provided with a minimum of one year from the time of the announcement of changes to the first competition launch to prepare. This means that the first funded researchers under the new schemes would be announced at the earliest at some point in 2014-15.

23. How will these changes influence other health research funders?

As a major federal funder of health research in the country, CIHR must take appropriate action to ensure the long-term sustainability of its contributions to the health research enterprise.

CIHR recognizes the proposed changes may also have an impact on other health research funding groups/agencies. We are currently working with funding partners to further discuss the implications of the changes and the next steps needed to ensure minimal disruption to the health research funding landscape.