CIHR Peer Review Manual For Fall 2013 Knowledge Synthesis Grant Applications

Other format

Table of Contents

Section I – Introduction

CIHR has chosen the Fall 2013 Knowledge Synthesis Grant competition to pilot some of the reforms proposed to the Open Suite of Programs. As part of CIHR's second strategic plan, Health Research Roadmap: Creating innovative research for better health and healthcare reforms to the Open Suite of Programs and Peer Review are underway to address a number of challenges identified by CIHR's research and stakeholder communities. As described in Designing for the Future: The New Open Suite of Programs and Peer Review Process, these reforms are intended to ensure the long-term sustainability of the Canadian health research enterprise.

The design of the new Open Suite of Programs sets clear objectives for CIHR's investigator-driven grants program. This includes integrating CIHR's current suite of Open funding mechanisms into a simpler system (Project Scheme and Foundation Scheme) that is flexible enough to accommodate today's changing landscape of health research.

The Project Scheme is designed to capture ideas with the greatest potential for important advances in health-related knowledge, the health care system, and/or health outcomes, by supporting projects with a specific purpose and defined endpoint. The best ideas may stem from new, incremental, innovative, and/or high-risk lines of inquiry or knowledge translation approaches. It is CIHR's intent to capture all of these ideas in the Project Scheme.

The Project Scheme is expected to:

  • Support a diverse portfolio of health-related research and knowledge translation projects at any stage, from discovery to application, including commercialization;
  • Promote relevant collaborations across disciplines, professions, and sectors;
  • Contribute to the creation and use of health-related knowledge.

The Project Scheme will be supported by a two-stage competition and application-focused review process. Stage 1 focuses on assessing a project's concept and feasibility, which is founded on selecting projects with a sound and important idea supported by a feasible plan of execution. Stage 2 focuses on the selection of applications close to the funding cut-off, also known as “grey zone” applications.

The success of the transition to the new Open Suite of Programs and peer review processes rests on our ability to efficiently and effectively pilot and test the functionality of peer review design elements. CIHR will use the Fall 2013 Knowledge Synthesis Grant competition to pilot Open reforms design elements.

The objectives of the pilot are to test components of the proposed Project Scheme in a live grant competition, to evaluate open reforms design elements and enhancements to the peer review process, and to ensure that future knowledge synthesis grant applications with an integrated knowledge translation approach can effectively be integrated into the new Project Scheme designs.

The following design elements of the proposed Project Scheme will be tested:

  1. Increased number of reviewers per application
  2. Structured application and review
  3. Multi-stage review
  4. Remote review and asynchronous online discussion
  5. New rating scale and ranking system
  6. Face-to-face committee meeting (Stage 2)
  7. New e-enabled technologies during the face-to-face meeting

Piloting peer review design elements will allow CIHR to adjust and refine processes and systems in order to best support applicants and reviewers. CIHR will collect feedback from applicants, institutions, and peer reviewers through surveys. Lessons learned from this pilot will help refine the design of the new Open funding schemes and peer review processes. It is our intent to share our findings with the research community, and contribute to the body of literature on peer review.

Section II – Policies and Principles of Peer Review

1. Purpose of the Manual

On behalf of CIHR, we would like to thank you for agreeing to serve as a peer review committee member for the Fall 2013 Knowledge Synthesis Grant Competition. The success of the peer review process is made possible by dedicated people like yourself who generously give of their time and expertise, and your efforts are greatly appreciated by CIHR and the scientific community.

The peer review process is described in detail in this manual. It is essential that committee members read and become familiar with this Manual and the Funding Opportunity. The purpose of this manual is to:

  • provide information on CIHR's objectives, governance and policies;
  • outline the roles and responsibilities of peer review committee members;
  • define the policies and procedures for peer review of applications; and
  • describe the peer review process that will be used in this competition.

For detailed regulations concerning all aspects of CIHR funding programs, please refer to the Grants and Awards Guide.

If you require further clarification on any subject in this manual or other instructional documents, please contact Anne-Marie Poulin, Program Coordinator (at 613-948-2899; anne-marie.poulin@cihr-irsc.gc.ca).

2. Peer Review at CIHR

The mandate of the Canadian Institutes of Health Research (CIHR) is as follows:

“To excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened health care system.”

The purpose of peer review is to ensure excellence in the research funded by CIHR. The peer review system also ensures accountability, not only to the Government of Canada and the Canadian taxpayer – the source of CIHR funding – but to the research community at large. Peer review is carried out by experts that encompass all four pillars of health research (Biomedical, Clinical, Health Systems and Services, and Population and Public Health).

Peer review is overseen by CIHR's Science Council (SC), which governs all aspects of research-related decision making. SC provides scientific leadership and advice to Governing Council (GC) on health research and knowledge translation (KT) priorities and strategies, and recommends investment strategies in accordance with CIHR's 5-year Strategic Plan. The approval of funding opportunities for all research and knowledge translation initiatives is an integral part of SC's responsibilities.

3. Principles of Peer Review

3.1 Confidentiality

Review Committee members, External Reviewers and Observers must abide by the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations (COIC), and ensure that:

  1. all documentation and information that the CIHR entrusts to Review Committee members, External Reviewers and Observers is maintained in strict confidence at all times. It must be used only for the purpose for which it was originally collected--namely, to review applications and make funding recommendations as applicable;
  2. review documentation is stored in a secure manner to prevent unauthorized access. It must be transmitted using secure techniques and when it is no longer required, it must be destroyed in a secure manner. Any loss or theft of the documentation must be reported to CIHR; and
  3. all enquiries or representations received by Review Committee members, External Reviewers or Observers about an application or its review must be referred to CIHR staff. Review Committee members, External Reviewers or Observers must not contact the Applicants for additional information or disclose matters arising from the Review Process to the Applicants.

Additional requirements for Review Committee members and Observers:

  1. Review deliberations are confidential. Comments made by Review Committee members during the review of applications and the conclusions of the Committee's review must never be discussed or disclosed with individuals not involved in the Review Process unless required by legislation or the courts.
  2. The identity of successful Applicants and the details of the grants/awards must remain confidential until a decision is made by the Funding-decision Authority and officially announced to the Applicants and the public. The identities of unsuccessful or ineligible Applicants are not made public and must not be divulged unless required by legislation or the courts.
  3. During the meeting, observers must be as unobtrusive as possible to minimize disruption and must not remove from the meeting room written notes or documentation related to reviewer assignments, ratings or reviewer comments on applications.

3.2 Conflict of Interest

CIHR must make every effort to ensure not only that its decisions are fair and objective, but also that they are seen to be so. According to the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations (COIC):

A person may be considered for membership on a Review Committee unless he/she:

  1. has Funding-decision Authority for the funding opportunity;
  2. is identified as the principal Applicant on an application to be reviewed by that committee; or
  3. is currently ineligible to apply for and/or hold funds from the Funding Organizations, or any other research or research funding organization worldwide for reasons of breach of policies on responsible conduct of research--such as ethics, integrity or financial management policies, unless otherwise agreed by the Responsible Authority.

Reviewers are considered to have a conflict of interest with an application if they:

  • are a relative or close friend, or have a personal relationship with the applicants;
  • are in a position to gain or lose financially/materially from the funding of the application;
  • have had long-standing scientific or personal differences with the applicants;
  • are currently affiliated with the applicants' institutions, organizations or companies — including research hospitals and research institutes;
  • are closely professionally affiliated with the applicants, as a result of having in the last six years:
    • frequent and regular interactions with the applicants in the course of their duties at their department, institution, organization or company;
    • been a supervisor or a trainee of the applicants;
    • collaborated, published or shared funding with the applicants, or have plans to do so in the immediate future; and/or,
  • feel for any reason unable to provide an impartial review of the application.

All committee members (Chair, Scientific Officer, reviewers, etc.) are subject to the same conflict of interest guidelines. CIHR staff and the Chair are responsible for resolving areas of uncertainty.

All committee members must read and agree to abide by the COIC policy prior to viewing any application information.

As you are aware, managing conflicts of interest is very important from CIHR's perspective, but we also don't want the review process to suffer because of the overly strict application of our policy. As such, we do allow flexibility for members to manage their own conflicts, particularly in situations where their relationship with the individual may be through a large network or other environments where true interactions are limited. Given that applications are becoming more multi-disciplinary and with increasing numbers of applicants, the conflict of interest criteria are most important for Principal applicants. When asked how to determine conflicts with co-applicants, the criteria we recommend applying most strictly are if you have been an applicant (with true interactions, such as a Principal applicant) on a grant or published with the individual in the past 6 years. As the most important overall guideline, we advise that if you think you would feel uncomfortable seeing the person and not sharing the information you know (or alternatively if you think you can't remain unbiased), then we ask that a conflict be declared. If you are from the same institution or department but don't really know or work with the person, we would not consider that you automatically need to declare this as a conflict.

Please advise the committee coordinator as soon as possible:

  • If you discover you have a conflict of interest with an application which you have not already identified;
  • If you are unsure whether your situation constitutes a conflict of interest.

If you require further clarification or would like to discuss any concerns, please contact Anne-Marie Poulin, Program Coordinator (at 613-948-2899; anne-marie.poulin@cihr-irsc.gc.ca).

3.3 Fairness

Success of the peer review system is critically dependent upon the willingness and ability of all committee members to be fair and reasonable; to exercise rigorous scientific judgment; and to understand, and take into account in a balanced way, the particular context of each application. Reviews are provided to the applicant without prior editing by CIHR staff, and CIHR does not take responsibility for their content. An applicant will not accept that your review is fair if it contains comments that could be construed as sarcastic, flippant, arrogant, or inappropriate in any way. Conversely, a constructive review, which includes helping the applicant by pointing out strengths and weaknesses that could be repaired in a resubmission, will help to convince a disappointed applicant that you provided a fair assessment of the proposal.

4. Policies Impacting Peer Review

4.1 International Collaboration & Global Health Research

As stated in the CIHR Act, one of the ways CIHR fulfills its mandate is by “pursuing opportunities and providing support for the participation of Canadian scientists in international collaborations and partnerships in health research.” As a result, CIHR accepts applications for research to be carried out in, or in collaboration with applicants based in, other countries. The international nature of the research should not be a factor in the scientific assessment of the proposal, beyond how it relates to the feasibility of the proposed research and the quality of the research question or concept. Reviewers should also not be influenced by the funding obtained or requested for the international components when recommending a budget for the Canadian component(s). For detailed information on applying for funding with an international partnership component, please refer to the subsections titled Global Health Research and International Collaborations in the Grants and Awards Guide.

For detailed information on how CIHR supports international collaborations and global health research, please view Internationalization of CIHR Funding Policy and Program Tools.

4.2 Knowledge Translation

Knowledge translation is integral to CIHR's mandate and falls into two main categories, end of grant KT and integrated KT. With both categories of knowledge translation, CIHR expects researchers to disseminate their findings and facilitate their translation into improved health, more effective products or services, and/or a strengthened healthcare system. Note that the costs of dissemination are eligible expenditures in all CIHR grants.

For end of grant KT, many means of dissemination exist and the onus is on the researcher to select the most appropriate vehicle for the intended knowledge-user audience to ensure maximum impact. When the primary knowledge users are researchers, dissemination of results through the publication of articles in high quality and accessible journals is appropriate, although other strategies that increase awareness of the results and facilitate their application may also be appropriate. When knowledge-user audiences outside the research community should be informed of specific research findings, dissemination plans with more ambitious goals and comprehensive strategies are expected. With integrated KT, stakeholders or potential research knowledge users are engaged in the entire research process and the research is directed at producing solutions to issues or problems the stakeholders/knowledge users have identified. Please consult About Knowledge Translation for more information.

4.3 Open Access

As of January 1, 2013, researchers awarded funding from CIHR are required to adhere with the following responsibilities:

  • ensure that all research papers generated from CIHR funded projects are freely accessible through the Publisher's website or an online repository within 12 months of publication;
  • deposit bioinformatics, atomic, and molecular coordinate data into the appropriate public database (e.g. gene sequences deposited in GenBank) immediately upon publication of research results;
  • retain original data sets for a minimum of five years (or longer if other policies apply); and
  • acknowledge CIHR support by quoting the funding reference number in journal publications.

Please consult the CIHR Open Access Policy.

4.4 Gender, Sex and Health Research

Applicants are encouraged to demonstrate the use of gender and sex-based analysis in applications. Gender and sex-based analysis is an approach to research which systematically inquires about biological (sex-based) and sociocultural (gender-based) differences between women and men, boys and girls, without presuming that any such differences exist. The purpose of this line of inquiry is to promote rigorous health research which expands understanding of health determinants in both sexes and results in improvements in health and health care. For more information on how peer reviewers can assess whether gender and/or sex are appropriately integrated into CIHR applicants' proposed research designs, please refer to Integrating Gender and Sex in Health Research: A Tool for CIHR Peer Reviewers.

4.5 Official Language Minority Communities

Federal agencies are required to take positive measures to ensure the support and recognition of minority language communities in Canada. For CIHR, this means an obligation to promote health research in these communities. For further information, please refer to the initiative on Official Language Minority Communities. Research proposals in these areas should still be subject to the same rigorous peer review process as any other application. However, the justification for promoting health research in minority language communities should not be a factor in the assessment.

4.6 Publications and Productivity

An important evaluation criterion in all grant programs is the expertise and experience of the applicant(s). A key factor in assessing this criterion is the productivity of the applicant(s), as determined by the quality and impact of contributions to the field. When assessing the quality of publications, peer review committees should focus on the quality of a publication's content and NOT simply the number of publications nor the quality or impact factor of journals. In the case of multi-authored publications or other collaborative work, applicants are advised to describe their contribution and reviewers should assess the specific contribution of the applicant to the work.

CIHR funds researchers in many health-related areas, and the forms of research publications can vary greatly among disciplines. In addition to the more traditional peer-reviewed journals, health researchers also publish in books, monographs, memoirs or special papers, review articles, conference/symposia proceedings and abstracts, government publications, etc. Some fast-moving research fields, such as some areas of computing science, genetics or microelectronics, use special means to reach the target audience quickly. Communications, quick-print reports, letters and electronic distribution of pre-prints are important vehicles for disseminating research results. All such contributions should be treated equally when assessing quality and impact, and reviewers should not regard certain types as "second class" or "grey literature."

When assessing productivity, reviewers should also be sensitive to legitimate delays in research and dissemination of research results. Some circumstances make it impossible or undesirable for researchers to publish important results of their research prior to applying for CIHR support. For instance, the time required to complete a monograph may exceed the time available between consecutive applications, or the protection of intellectual property may require a delay in publication. Research productivity may also vary as a result of personal circumstances, such as pregnancy or early child care, administrative leave, disability, elder care, etc., whether or not a formal leave of absence is taken. Applicants are advised to clearly and fully describe any circumstances that affect the dissemination of research results in their application. Peer reviewers must be sensitive to the impact of these circumstances on the level of productivity, while ensuring that the quality of the research remains competitive.

Section III - Peer Review

5. Peer Reviewer Recruitment

The Fall 2013 Knowledge Synthesis peer review committee consists of a Chair, Scientific Officer, peer reviewers (consisting of academic researchers and knowledge users), and CIHR staff. Individual committee members are selected for their research excellence, as reflected by their ability to obtain continued extramural peer-reviewed funding, and for their breadth of knowledge and maturity of judgment. For more details, please refer to the Peer Review Membership Guidelines. For this competition, reviewers will be recruited to participate as expert reviewers in Stage 1 or as multi-disciplinary reviewers in Stage 2. For more details, please refer to the Procedure for Selection of Peer Review Committee Members. Peer review committee members have clear roles and responsibilities to ensure the effective functioning of the peer review process. For more details see Appendix 1.

6. Peer Review Process Overview

Peer review for the Fall 2013 Knowledge Synthesis grants competition will be conducted using a multi-stage approach. Stage 1 review will be conducted remotely by expert reviewers including both researchers and knowledge users, and supported by an internet-assisted platform that will enable communication among reviewers in a virtual space. Reviewers will rate and rank their assigned applications and provide a written report. CIHR will consolidate all individual reviewer rankings into an average ranking for each application and provide a combined ranking list for the competition. Stage 2, the final assessment stage, will involve a face-to-face discussion by a number of multi-disciplinary reviewers. These reviewers will be responsible for integrating the results of the Stage 1 reviews. The discussion will focus on determining which applications to fund, with particular emphasis on applications with large variances in independent reviewer rankings. This committee will make final recommendations on which applications to fund, by creating a final rank list of the applications.

Applications will be funded from the top down in order of final ranking as far as the budget will allow.

Figure 1. Two-stage competition process for the Knowledge Synthesis pilot competition

Figure 1 long description

For more details, access the Interactive Learning Lesson: Overview of the Knowledge Synthesis Competition.

7. Initializing the Review Process

7.1 Assigning Applications

All eligible applications received by the appropriate deadline date are entered into the competition. Applications are required to be complete at the time of submission; otherwise they may be withdrawn from the competition.

After the list of applications is compiled, committee members are given access to the application summaries to declare any conflicts of interest and indicate their level of expertise.

The Chair and Scientific Officer, together with CIHR staff, review the responses and assign the applications to committee members. Together, they are responsible for ensuring the committee is equipped with the appropriate expertise necessary to ensure a fair review. The final authority for the assignment of applications rests with CIHR.

For the Fall 2013 Knowledge Synthesis Grant competition, in Stage 1 and Stage 2, each application will be assigned to five reviewers, with no less than 10 and no more than 20 applications assigned to each reviewer. Applications will be assigned to two academic and two knowledge user reviewers based on available expertise and the fifth reviewer type will be either an academic or a knowledge user.

Readers and external Reviewers will not be assigned for this competition.

All reviewers are then given access to their assigned full applications four to six weeks before the deadline for submitting final reviews and rankings.

7.2 Access to Applications

Copies of Applications

As part of CIHR's “Go Green” campaign, paper copies of applications are no longer sent to reviewers. Applications can be accessed electronically through ResearchNet after you have completed the conflicts and ability to review task and have received your assignments.

7.2.1 Security

All materials used in the peer review process must be stored in a secure manner to prevent unauthorized access. They must be transmitted using secure carriers and technologies. When they are no longer required, all material related to peer review must be destroyed using a secure method or returned to CIHR for destruction. Any loss or theft of materials related to peer review must be reported to your committee coordinator immediately. Please see the Guide on Handling Documents Used in Peer Review for further details. Contact your committee coordinator if you require assistance.

7.2.2 Structured Application Format

Applicants will submit their research proposal using the new structured application format including character limits for each criterion section (i.e. font size of 12 point, six lines per inch, no condensed type or spacing). Due to technical limitations, CIHR does not systematically verify the formatting of each application received. If you feel that the formatting of any of the proposals assigned to you may not follow CIHR's guidelines for font size, line spacing and/or margins, please contact the committee coordinator as soon as possible, prior to the meeting so CIHR can determine whether or not the application conforms to the guidelines. More information regarding CIHR's formatting guidelines can be found in the document entitled Acceptable Application Formats and Attachments.

7.2.3 Application Attachments

CIHR allows freedom in the amount of material that can be attached in the Application attachments (up to 30 MB) to accommodate the varying needs of different applications. However, CIHR will not oblige peer reviewers to read excessive or superfluous information. Applicants are advised that the research proposal should depend only upon references, tables, figures and letters of support and collaboration and peer reviewers are under no obligation to read additional materials, beyond those named above. Peer reviewers may exercise their judgment as to which additional materials are required for review of the application (see Knowledge Synthesis Grant: Fall 2013 - ResearchNet "Application" Phase Instructions for more information).

8. Peer Review Supporting Information

It is the responsibility of all peer reviewers to familiarize themselves in advance of conducting their review with the details of the funding opportunity, adjudication criteria, interpretation guidelines, and the peer review process in general.

The following resources are available:

8.1 Documents

  1. Knowledge Synthesis Grant: Fall 2013 Funding Opportunity.
  2. Interpretation Guidelines for the Fall 2013 Knowledge Synthesis Grant Competition Adjudication Criteria
  3. Knowledge Synthesis – Tips for success
  4. A Guide to Knowledge Synthesis
  5. Synthesis Resources
  6. About KT
  7. CIHR's Guide to Knowledge Translation Planning
  8. CIHR Guide to Writing Letters of Support

8.2 Interactive Learning Lessons:

The following lessons are currently available on the Interactive Learning Lessons page:

  • Overview of the Knowledge Synthesis Competition
  • Application Process
  • Asynchronous Online Discussion Tool

The following lessons will be available in the coming weeks:

  • Understanding the Adjudication Criteria
  • Stage 1 Review Process
  • Stage 2 Review Process

8.3 Webinar Orientation Sessions

  • Stage 1 Reviewers: October – November 2013
  • Stage 2 Reviewers: January 2014

Note: Precise dates of sessions will be communicated to reviewers in fall 2013.

9. Step by Step Guide to Peer Review

9.1 Stage 1 Review:

Stage 1 reviewers are required to complete the following review tasks on ResearchNet:

9.1.1 Complete Preliminary Reviews
9.1.2 Discuss Applications Online (Optional)
9.1.3 Complete Application Ranking and Submit Final Reviews

For more details, access the Interactive Learning Lesson: Stage 1 Review Process

9.1.1 Complete Preliminary Reviews:

The preliminary review for each assigned application is conducted on ResearchNet by completing the following:

  • Read the applications.
  • Rate the applications by assigning a rating for each adjudication sub-criterion using the A, B, C, D and E adjudication scale (see Adjudication Scale).
  • Provide up to half a page of written justifications with comments on the strengths and weaknesses for each sub-criterion.
  • Assess the requested level of budget support.
  • Identify applications with “Special Attentions”.
  • Edit your reviews and resave as often as desired. Each time you save, a rank ordered list of your assigned applications is created.
  • The reviewer does not have to break ties prior to the online discussion.
  • Submit preliminary reviews respecting assigned deadlines.
  • As reviews are submitted, the reviews will be available to the other reviewers assigned to the application.

9.1.1.1 Stage 1 Adjudication Criteria

For more details see the: Adjudication Criteria in this document or access the interactive learning lesson: Understanding the Adjudication Criteria

Criterion 1 – Concept

1.1 Quality of the Idea
1.2 Importance of the Idea

Criterion 2 – Feasibility

2.1 Approach
2.2 Expertise, Experience and Resources

Interpretation guidelines for the adjudication criteria can be found in the document Interpretation Guidelines for the Fall 2013 Synthesis Grant Competition Adjudication Criteria.

9.1.1.2 Stage 1 Adjudication Scale

Reviewers are asked to assign an alpha rating to each equally weighted sub-criterion using the adjudication scale A, B, C, D, E. For more information, see the Adjudication Scale.

9.1.1.3 Complete Budget Assessment

Review the requested budget level and justification. A detailed item-by-item scrutiny by the reviewer is not expected. Reviewers may approve the budget as is or provide an adjusted amount. For more details refer to Budget and Term Determinations and the document entitled Interpretation Guidelines for the Fall 2013 Synthesis Grant Competition Adjudication Criteria.

9.1.1.4 Flag Application Issues to CIHR

These issues are not to be considered as criteria for evaluation, except if they impact scientific quality of the application. For detailed regulations concerning these issues, please refer to the CIHR Grants and Awards Guide.

  1. Eligibility: Reviewers should raise any concerns with respect to whether the Principal Applicant(s) and their affiliated institutions meet the criteria to receive CIHR funding.
  2. Ethics: Responsibility for ensuring that all research meets ethical standards is delegated to the local institution by CIHR. Ethics forms are not required as part of the application. However, the reviewer may flag specific issues, such as the use of human subjects, animals, human tissues or hazardous material, or research that appears to involve Aboriginal people, if they feel they have not been adequately addressed.
  3. Human pluripotent stem cell research: Applications involving the use of human stem cells and likely to be funded will also be reviewed by the Stem Cell Oversight Committee (SCOC). Applicants are instructed to check the relevant box in the section entitled “Certification Requirements”, but it is essential that this be verified by committee members
  4. Budget justification: If the reviewer cannot properly assess the budget request because of an unclear justification by the applicant, the issues should be brought to the attention of CIHR staff for follow up before funds are released, if the application is funded.
  5. Section 56 of the federal Controlled Drugs and Substances Act: All research proposals that are subject to Section 56 of the Controlled Drugs and Substances Act are required to have an exemption from Health Canada. Committee members should flag such applications to CIHR staff at the meeting who will follow up before funds are released, if the application is funded

For more information please refer to the Pending Grants and Awards page in the Grants and Awards Guide.

9.1.1.5 Submit Preliminary Reviews on ResearchNet

Preliminary reviews may be saved at any time. When the assigned preliminary reviews are complete they may be submitted to CIHR individually or all at once. It is important to respect all assigned deadlines.

9.1.2 Discuss Applications Online

The Online discussion tool is a means to engage communication between the reviewers. All reviewers assigned to the application may view and participate in the discussion once they have submitted their preliminary reviews. The identity and the assessment of every reviewer for each application will be shown to all reviewers of the application. The decision to discuss an application is at the discretion of the reviewer(s). The Chair, Scientific Officer or CIHR staff may prompt the initiation of discussion or provide input where necessary to ensure that CIHR policies and procedures are followed.

This tool is not intended as a means to provide direct feedback to the applicant(s) and discussions will not be made available in the applicant's decision package.

Preliminary reviews (ratings, written justifications (strengths and weaknesses) and ranking of the application) will be shared with the other reviewers assigned to the applications once the preliminary reviews are submitted. Reviewers who have not submitted their preliminary reviews by the end of the discussion period will not be able to participate in the discussion and will have a status of pending.

During and after the discussion, reviewers may update their ratings and reviews. This is important since all comments to applicants should be noted in the structured assessment.

9.1.2.1 Using the Online Discussion Tool

Using the online discussion is an optional step in the process and reviewers are not required to initiate or participate in online discussions. However, CIHR encourages reviewers to share their perspectives and discuss discrepancies in their reviews with the other assigned reviewers.

The asynchronous online discussion tool will only be opened once every reviewer has submitted their preliminary reviews for a given application or after the pre-assigned date in the system if not all preliminary reviews have been submitted. Once the tool is open, reviewers will be able to see any preliminary review that has been submitted for an application.

The following statuses will help the reviewers to navigate the online discussion function in ResearchNet:

  1. Pending: The reviewers cannot begin the discussion. They must wait for the other reviewers to submit their preliminary ratings.
  2. Start Discussion: Allows the reviewers to be the first to post a comment.
  3. Active: The discussion has begun.
  4. Active with star symbol (*): A reviewer posted a new comment since the last time the discussion was accessed.
  5. Closed: All reviewers have submitted their final ratings and reviews.
  6. Suspended: The online discussion is closed until further notice by CIHR.
  7. Notify Reviewer: Notifies reviewers if other reviewers are online and available for discussion.

Once a comment is posted, the reviewers cannot delete or edit this comment. Comments can be posted for the attention of CIHR staff, however, it should be noted that these comments will be visible by the other reviewers. This option may be useful for flagging issues for CIHR attention. Should they prefer, reviewers may send an e-mail message directly to CIHR staff. A notification will be sent daily to advise reviewers of any new posts on the online discussion board. Once the discussions are complete, reviewers may edit the alpha ratings and written justifications and save their reviews as often as desired.

Note: Applicants will not receive a summary of the on-line discussion. Reviewers may update their reviews at any time during the on-line discussion period if they deem it necessary.

Once satisfied, the reviewer confirms their reviews. After confirming reviews, the reviewer can no longer edit their reviews.

For more details, access the interactive lesson: Asynchronous Online Discussion Tool.

9.1.3 Complete Application Ranking and Submit Final Reviews to CIHR:

As each review is confirmed, an initial ranked list of his/her assigned applications is generated. Applications are ranked based on their ratings assigned during the preliminary review exercise with the highest ranked applications listed first.

  • Reviewers must break ties,
  • Reviewers can change the application rank order position up or down as the original rank order position will remain visible as a reference point.
  • Once satisfied, the reviewer confirms that the rankings are “Final” and the system locks the rankings. The reviewer is no longer able to edit the rankings.
  • The reviewer submits the reviews and rankings for all their assigned applications to CIHR
  • CIHR will consolidate all individual reviewer rankings into an average ranking for each application.
  • It is important for the reviewers to submit their reviews and rankings on ResearchNet by the assigned deadline.
  • A consolidated ranking list will be computed for the competition.
  • All applicants will receive feedback from each of the 5 assigned reviewers.
  • Stage 1 decisions will not be released to applicants and institutions. Funding decisions will only be made public after Stage 2 for this competition.
  • Approximately 50% of the applications are expected to be recommended for Stage 2 review.

Please contact your committee coordinator, Anne-Marie Poulin, Program Coordinator (at 613-948-2899; anne-marie.poulin@cihr-irsc.gc.ca) if you have questions or encounter any technical issues.

9.2 Stage 2 Review:

The final assessment for Stage 2 review will be a face-to-face multidisciplinary committee meeting, with different reviewers from Stage 1. The multi-disciplinary reviewers will discuss the ranking list of Stage 1 applications and will be responsible for integrating and validating the results of the Stage 1 review. This committee will focus on competitive (highly ranked) applications with special attention to any applications with high variance, and will make final recommendations on which applications to fund within the allocated budget.

Stage 2 reviewers are required to complete the following review tasks on ResearchNet:

9.2.1 Complete Pre-Stage 2 Committee Meeting Activities
9.2.2 Attend the Face-to-Face Committee Meeting

Once the reviewers receive notice, they will be asked to log on to ResearchNet to declare their conflicts.

Each application will be assigned to five reviewers, with no less than 10 and no more than 20 applications assigned to each reviewer. Applications will be assigned to two academic researchers, two knowledge users and the reviewer type for the 5th reviewer will be randomly assigned.

For more details, access the interactive lesson: Stage 2 Review Process

9.2.1 Pre-Stage 2 Committee Meeting Activities

Reviewers will receive an assessment package which includes:

  • Access to all applications that are submitted to Stage 2 including their Stage 1 reviews (written strengths and weaknesses, ratings and budget recommendations) along with an assessment of the variance of reviewers' rankings.
  • A cross section, through random assignment of 10-to 20 Stage 2 applications
  • The available budget and anticipated success rate of the competition

Stage 2 reviewers will be responsible for reading their assigned stage 1 reviews prior to the face to face committee meeting and to consult the other application materials if necessary. Access to all applications is provided as an option.

  • Reviewers will review their assessment packages and the rank order list of the competitive applications reviewed in Stage 1.
  • Reviewers will be required to assign the applications that they are responsible for into either a yes (recommended for funding) or no (not recommended for funding) bin. The number of yes/no assigned will be based on the budget available and anticipated success rate of the competition.
  • Reviewers are encouraged to provide written comments to justify their recommendations.
  • The reviewers will be able to save and submit their results to CIHR respecting the assigned deadline.
  • The reviewers will have 3 to 4 weeks to complete the reviews.
  • The Chair of the committee is expected to review all Stage 2 assessment packages and recommendations from the committee prior to the meeting. CIHR staff will prepare a schedule for the face-to-face committee meeting in consultation with the Chair and Scientific Officer.

9.2.2 Attend the Face-to-Face Committee Meeting

The Chair and CIHR Deputy Director will begin the committee meeting by reviewing the recommendations from the Stage 2 reviewers (generated from the pre-Stage 2 meeting yes/no decisions) with the committee.

The Chair and CIHR staff will provide guidance on how to discuss the applications. Any committee member who has a conflict of interest with an application (as defined in the Conflict Of Interest section) must not take part in the discussion of that application. For face-to-face meetings, committee members in conflict must leave the room before the application is discussed. The Chair and CIHR staff are responsible for monitoring conflicts and for resolving areas of uncertainty.

The Chair moderates the discussion and the Scientific Officer records meeting notes.

If an application is discussed, the committee meeting discussion proceeds as follows:

  • Any committee members in conflict must leave the room.
  • The Chair asks one of the assigned reviewers to summarize the reviews of the application and any issues that need to be discussed by the committee.
  • Other reviewers assigned to that application are asked to provide further information concentrating on points of agreement or disagreement, and elaborating points not already addressed.
  • The Chair leads the discussion of the application by all committee members;
  • The Scientific Officer reads back the Scientific Officer notes, capturing the key elements of the discussion to be considered when ranking the application;
  • After discussion, the reviewers may vote to move the applications up, down or stay by using an automated voting device (all votes will be secret).
  • The majority of votes obtained for a given action will be deemed as the outcome for that application. For example, up one position, down one position and/or remain at the same position.
  • The application will not move if there is a tie in votes for a given movement and the applications can only move in increments of one place at a time (up or down). The application can be discussed and voted on as often as the committee desires and may continue to vote as long as there is movement. The committee may also come back to an application as necessary. Discussion usually stops when votes lead to no movement.
  • This process continues until the committee reaches consensus on a final rank position for the application.

9.2.3 Budget and Term

The appropriateness of the budget and the term of support may be discussed if necessary.

9.2.4 Flag Application Issues to CIHR

Any concerns in the following areas should have been flagged by the Stage 1 reviewers but Stage 2 reviewers may discuss and, if necessary, flag any issues at the meeting for CIHR staff to address.

These issues are not to be considered as criteria for evaluation, except if they impact scientific quality of the application. For detailed regulations concerning these issues, please refer to the CIHR Grants and Awards Guide.

  1. Eligibility: Reviewers should raise any concerns with respect to whether the Principal Applicant(s) and their affiliated institutions meet the criteria to receive CIHR funding.
  2. Ethics: Responsibility for ensuring that all research meets ethical standards is delegated to the local institution by CIHR. Ethics forms are not required as part of the application. However, the reviewer may comment on specific issues, such as the use of human subjects, animals, human tissues or hazardous material, or research that appears to involve Aboriginal people, if they feel they have not been adequately addressed.
  3. Human pluripotent stem cell research: Applications involving the use of human stem cells and likely to be funded will also be reviewed by the Stem Cell Oversight Committee (SCOC). Applicants are instructed to check the relevant box in the section entitled “Certification Requirements”, but it is essential that this be verified by committee members.
  4. Budget justification: If the peer review committee cannot properly assess the budget request because of an unclear justification by the applicant, the issues should be brought to the attention of CIHR staff for follow up before funds are released, if the application is funded
  5. Section 56 of the federal Controlled Drugs and Substances Act: All research proposals that are subject to Section 56 of the Controlled Drugs and Substances Act are required to have an exemption from Health Canada. Committee members should flag such applications to CIHR staff at the meeting who will follow up before funds are released, if the application is funded.

For more information please refer to the Pending Grants and Awards page in the Grants and Awards Guide.

9.2.5 Conclusion of Review

The committee concludes its work after the discussions of applications has been completed and the committee has agreed on a final ranking list of all applications with recommendations for funding.

9.2.6 End of Committee Meeting Review

An important component of any peer review committee meeting is the final review of the committee's effectiveness and functioning, and a discussion of policy issues that may have arisen in the course of its deliberations. This discussion is important as it provides an opportunity for CIHR staff to address any concerns of the committee members and for staff to record feedback on the piloted peer review process. Feedback received during this discussion and through surveys will help refine the design of the new Open funding schemes and peer review processes.

9.3 After the Meeting

As soon as possible after the peer review committee meeting, CIHR staff generates a funding proposal based on committee recommendations, to be reviewed by the Chief Scientific Officer (CSO) and the Chief Financial Officer (CFO). The CSO and CFO consider the funding recommendation in light of criteria established by the CIHR Science Council (SC) and submit their recommendations to SC for final approval. A list of successful applicants is posted on the Funding Decisions Notifications page as soon as it is available.

Applicants are informed of the results of the competition once the SC has approved the grants to be funded. All applicants are sent a Notice of Decision, indicating whether or not their proposal was approved, and if approved, with what budget, which may or may not coincide with that recommended through peer review. The Notice of Decision will normally be released (on ResearchNet and by mail) within three weeks following the SC meeting.

Applications that have been flagged for special attention and followed up by CIHR staff (see Section 9.2.4) are withheld as “pending”. The applicant will be notified if further information is required. The additional information may be discussed by CIHR staff and peer review committee members if necessary prior to a final decision regarding funding.

10. Evaluating Grant Applications

10.1 Types of Applications

For the Fall 2013 Knowledge Synthesis Grant competition, applications may be new proposals, or resubmissions of an unsuccessful application. All application types are evaluated together “on a level playing field” and the same criteria and process are applied to all.

10.2 Adjudication Criteria

The review of the application should be structured to justify the overall rating based on the following equally weighted criteria.

Criterion 1 – Concept

1.1 Quality of the Idea

  1. Are the overall goal and objectives of the project well-defined and clear; with distinct outputs that support advances in health-related knowledge, health research, health care, health systems, and/or health outcomes?
  2. Is the rationale of the project idea sound logical and valid?

1.2 Importance of the Idea

  1. Is the proposed contribution(s) of the project well-defined, clear and significant with respect to advancing health-related knowledge, health research, health care, health systems and/or health outcomes?

Criterion 2 – Feasibility

2.1 Approach

  1. Are the approaches and methods appropriate to deliver the proposed output(s) and achieve the proposed contribution(s)?
  2. Are the timelines and related deliverables of the project realistic?
  3. Does the proposal identify potential challenges and appropriate mitigation strategies?

2 2. Expertise, Experience and Resources

  1. Does the applicant(s) bring the appropriate expertise and experience to lead and deliver the proposed output(s) and achieve the proposed contribution(s)?
  2. Is there an appropriate level of engagement and/or commitment from the applicant(s)?
  3. Is the environment (academic institution and/or other organization) appropriate to enable the conduct and success of the project?

Interpretation guidelines for the adjudication criteria can be found in the document Interpretation Guidelines for the Fall 2013 Synthesis Grant Competition Adjudication Criteria.

For more details, access the interactive lesson: Understanding the Adjudication Criteria: Fall 2013 Knowledge Synthesis Grant Competition

10.3 Structured Review Work Sheet

Reviewers complete their ratings and reviews in ResearchNet using a structured review format. The structured review work sheet is designed to ensure that the reviewer addresses each adjudication sub-criterion and has access to the relevant section of the structured application as reference. The goal of this process is to reduce the burden on the reviewers, and to provide consistent reviews to the applicants.

10.4 The Adjudication Scale

To ensure consistency, all Stage 1 reviewers must adhere to a common adjudication scale. It is particularly important that the full scale be used. To facilitate this, the following scale and descriptors are provided:

  1. For this sub-criterion, the application excels in most or all relevant aspects. Any short-comings are minimal.
  2. For this sub-criterion, the application excels in many relevant aspects, and reasonably addresses all others. Certain improvements are possible.
  3. For this sub-criterion, the application excels in some relevant aspects, and reasonably addresses all others. Some improvements are necessary.
  4. For this sub-criterion, the application broadly addresses all relevant aspects. Considerable improvements are required.
  5. For this sub-criterion, the application fails to provide convincing information and/or has serious inherent flaws or gaps.

11. Budget and Term Determinations

CIHR's objective is to provide the funds needed to allow approved research to be carried out effectively. While not being factored into the overall rating of an application, CIHR values the experience and perspectives of reviewers in estimating overall budget requirements of the proposed project. The appropriate budget is very much a matter for judgment by the peer reviewer.

The applicant has been asked to indicate his/her budget using pre-set incremental levels. A precise valuation of all line items in the budget justification is not expected or provided. A detailed item-by-item scrutiny by the reviewer is not expected.

Reviewers are asked to consider the proposed budget of the project, and provide a justified recommendation as to whether the budget should be:

  • Accepted, as described; or
  • Adjusted to the level of $X per annum.

The maximum amount allowable for a knowledge synthesis grant is $100,000 and the maximum amount allowable for a scoping review is $50,000.

If the peer reviewer feels that the budget is not adequately justified or explained in order to assess the request appropriately, the stage 1 reviewer or the stage 2 committee may request a follow-up by CIHR staff. If the proposal is approved for funding, funds will not be released until the budget justification concerns are resolved.

Consideration of the budget should include the following factors:

11.1 Research Staff

Research staff (research associates, research assistants, technicians, etc.) should be determined by the actual needs for the techniques and work required for the research. The salary scales put forward by the institution should be followed. Some institutions require non-discretionary benefits packages for staff (e.g. supplementary medical and dental insurances). These are considered eligible expenses on grants. Salaries for applicants (Principal Investigators or Co-Investigators) cannot be paid from the grant, except in the case of research associates and trainees.

Graduate students may be hired as research personnel on a grant. In general this is on a part-time basis, i.e., hourly. This situation is to be distinguished from a graduate student receiving a stipend from a grant (see below), in which case the work done is part of the training of the student and constitutes the thesis or comparable academic requirement.

11.2 Knowledge User Participation

Release time allowance may be requested to facilitate the participation of a knowledge user in the proposed research program. The eligible cost is limited to $50,000 per annum per individual per grant. Persons receiving release time allowances are not considered CIHR employees. Funds for research time allowance(s) must be justified in the budget section of the grant application.

The following conditions must be met for an individual to receive release time allowance, with documentation provided to the institution administering the grant:

  • The individual for whom the release time allowance is provided must:
    • be a knowledge-user applicant on the grant whose primary responsibilities do not include an expectation to engage in research (i.e., as part of their regular employment); and
    • have their organization's approval for the research time on the project that would justify the allowance; and
    • have their organization certify that they are engaged in the activities for which funds are being disbursed.
  • Release time allowance requests will not be considered for the following:
    • Individuals with salaried academic research appointments;
    • The cost of teaching time to allow the individual to engage in research;
    • Supplementing or replacing part of the salary of a researcher on sabbatical from their main appointment.

11.3 Trainees

The assessment of requests for support of trainees should take into account the quality of the training environment and the track record of the applicant in training young researchers.

As of September 15, 2011, the three federal granting agencies' existing regulations on minimum and maximum stipend levels paid from grants will no longer apply. The agencies will also no longer restrict researchers from using some of their grant money to provide supplements to scholarship holders.

The removal of tri-agency regulations will offer professors more financial flexibility to pay students and postdoctoral fellows at levels that reflect the variations in costs between departments and between regions. It should be noted that stipend levels will still be governed by applicable collective agreements and labour legislation. Also, many universities have established minimum support packages, varying by department, where stipend support can be assembled from teaching assistantships, research funds, university graduate scholarships, external scholarships, and bursaries.

11.4 Material, Supplies, Services and Travel

A budget should be established to include all non-personnel requirements for funds (materials and supplies, consumables, services, etc.).

In general, CIHR grant funds may be used to cover only the direct costs of research and may not be used for indirect costs (e.g., library, heat and light, office furniture, regular telephone rates etc.). See the Tri-Agency (CIHR, NSERC & SSHRC) Financial Administration Guide for further details. If a budget request includes amounts for what may seem to be indirect costs or “overhead,” these need to be justified in terms of their direct contribution to the research. For example, a departmental “tax” to cover costs other than research expenses (e.g., library acquisitions, graduate student stipends, secretarial pool, etc.) is not allowable. The latter should normally be covered by the institution. When in doubt, such costs should be flagged for CIHR staff to follow up.

11.5 Equipment

Research Equipment: Any item (or interrelated collection of items comprising a system) of nonexpendable tangible property, having a useful life of more than 1 year and a cost of $2,000 or more, which is used wholly or in part for research.

Note that all three conditions must be met for an item to be considered equipment:

  1. nonexpendable tangible property; and,
  2. useful life of more than 1 year; and,
  3. a cost of $2,000 or more.

To be considered materials or supplies an item must meet only one of the following conditions:

  1. expendable tangible property; or,
  2. useful life of 1 year or less; or,
  3. a cost of less than $2,000.

For example, a laptop computer that costs less than $2,000 would be considered as materials or supplies even though it is a nonexpendable tangible item with a useful life of more than one year.

11.6 Overlaps with other Funding Sources

Peer reviewers are asked to recommend budgets for grant applications irrespective of other sources of funds received or applied for, i.e., reviewers should not reduce recommended budgets to take into account potential overlap.

11.7 Term of Support

The term of support for both knowledge synthesis applications and scoping review applications is up to one year.

Appendix I - Roles and Responsibilities of Committee Members

General Responsibilities for all Committee Members

  • Must agree to abide by CIHR's Confidentiality and Conflict of Interest Policy through ResearchNet.
  • Conflicts of interest with applications are declared on ResearchNet
  • Ensure that all review materials personally used are handled safely and disposed of according to the document "Guide on Handling Documents used in Peer Review".
  • Become familiar with CIHR policies and procedures described in this Peer Review Manual and posted on ResearchNet. It is also important to be familiar with the objectives of the funding opportunity
  • Review Peer Review Supporting Information to become familiar with the peer review process.
  • Complete Surveys as required.

Chair and Scientific Officer

Before Stage 1 Review

  • Suggest names to CIHR of potential reviewers.
  • Work with CIHR staff to assign applications and to select reviewers for each application. When the Chair is in conflict, the Scientific Officer is responsible for the reviewer assignment.
  • Work with CIHR staff to manage conflicts of interest of reviewers.

Stage 1 Review – Preliminary Review

  • Fulfill an oversight role and do not rate applications or submit preliminary reviews.

Stage 1 Review - Online Discussions

  • The Chair with the help of the Scientific Officer act as moderators to ensure a positive, constructive, fair-minded environment in which applications are discussed during the online discussion.
  • Ensure that the online discussion functions smoothly, effectively and objectively, in accordance with CIHR's policies and procedures during the online discussion period and alerts CIHR to any issues.

Stage 1 Review – Complete Application Rankings

  • Works with CIHR staff to determine which applications will proceed to Stage 2 Review.

Stage 2 Review - Pre-Meeting Activities

  • Work with CIHR staff to review the tabulated binning outcome list and to determine a review sequence or agenda for the Stage 2 committee meeting.

Stage 2 Review – Attend the Face-to-Face Committee Meeting

  • Chair provides opening remarks to the committee.
  • Chair makes introductions and explains the meeting process.
  • Chair ensures that all committee members in conflict with an application leave the room before discussion begins.
  • Appoints a delegate as Chair or Scientific Officer when either leaves because of conflict of interest. Whenever possible, the same individual should not occupy both roles.
  • Ensures the involvement of the entire committee in discussions.
  • Scientific Officer summarizes the discussion of each application discussed by the committee in written notes.
  • Ensure that the allotted budget is respected.
  • Ensures that specific concerns of ethics and other CIHR requirements are addressed and any discussion is recorded in the Scientific Officer notes.
  • Ensure that the committee agrees on a final ranking list and finalizes their selection of applications for the funded grants.

At the end of the Committee Meeting, the Chair:

  • Sets aside adequate time to receive feedback from the committee members regarding the effectiveness and functioning of the committee.

After the Committee Meeting, the Chair:

  • Provides feedback about the meeting process, dynamics and effectiveness in a short (one page) written report or e-mail to be submitted to the CIHR Deputy Director within one week after the committee meeting.

Note: All communications with the peer review committee from the Chair or Scientific Officer must include the CIHR Deputy Director and the Program Delivery Coordinator to ensure consistency with CIHR policies.

Stage 1 Reviewers

Complete Preliminary Reviews

Discuss Applications Online (Optional)

  • Initiate discussion of applications in ResearchNet
  • Participate in discussions of applications in ResearchNet
  • Edit ratings and revise reviews if desired
  • Confirm reviews

Complete Application Ranking and Submit Reviews

  • Review ranked list of assigned applications
  • Break ties or adjust ranked list of assigned applications
  • Submit reviews and ranked list of assigned applications to CIHR by the assigned deadline.

Stage 2 Reviewers

Complete Pre-Stage 2 Committee Meeting Activities

  • Review assigned application assessment packages
  • Allocate applications into fundable and not fundable bins
  • Provide comments.

Attend Face-to-Face Committee Meeting

  • Participate in the discussion of applications.
  • Participate in voting applications up down or stay to determine a final ranking list of applications.

Appendix II: Sequence of Steps for the Stage 2 Committee Meeting

  1. Welcome and Introductions

    Members are welcomed by the Chair, CIHR Deputy Director and Staff. After introductions around the table, the Chair, CIHR Deputy Director and Staff will address the committee providing opening remarks and objectives for the meeting.

  2. Agenda for the meeting

    The Chair and Deputy Director begin the committee meeting by reviewing the tabulated binning outcome list, based on yes/no decisions from the pre-stage 2 meeting activities submitted before the committee meeting.

    The Chair and CIHR staff provide guidance on how to discuss the applications i.e.: every application, the 10 closest to the cut-off, etc.

  3. Conflict of Interest:

    Before a specific application is discussed in detail, members in conflict of interest must leave the room.

    The Chair and CIHR staff are responsible for monitoring conflicts and for resolving areas of uncertainty.

  4. Reviewers:
    • provide summary of the reviews of the application
    • identify any issues that need discussion (e.g. large variation in ranking)
  5. Other Reviewers:
    • raise additional issues
  6. Committee discussion of applications should focus on:
    • factors important in ranking
    • differences of view between reviewers
  7. Scientific Officer reads SO notes to the committee:
    • summary of discussion
  8. Chair Discusses the Voting Process

    The Chair and CIHR staff provide guidance on the mechanics of the voting process.

  9. Individual rankings:
    • members may vote to:
      • move the application up by one position,
      • down by one position or
      • stay the same.
    • votes are confidential
    • The committee may vote as often as desired as long as there is movement up or down.
  10. Budget:
    • Any budget issues may be raised for discussion.
  11. Term of grant
  12. Issues to be flagged:
    • eligibility
    • ethics
    • human stem cells
    • environmental impact
    • budget justification
  13. Scientific Officer reads final notes (including budget comments) for review/modifications/additions by committee.

Supplemental content (right column)