Guide for Reviewers - CIHR Fellowships
Other format
Table of Contents
- Guide for Reviewers - CIHR Fellowships
- What's New?
- Introduction
- Peer Review Principles
- Step by Step Instructions
- Step 1: Read this Guide and the Funding Opportunity
- Step 2: Review Your Assigned Applications
- Step 3: Complete Survey Questions
- Step 4: Submit Preliminary Ratings and Reviews
- Step 5: Discuss the Application with the Other Reviewers
- Step 6: Review/Edit Your Ratings and Comments
- Step 7: Submit Final Ratings and Comments
- Annex 1: Evaluation Criteria
- Annex 2: Reviewer Worksheet
- Annex 3: Guide to Locating Information
Guide for Reviewers - CIHR Fellowships
1) What's New?
As part of CIHR's continuous commitment to improve peer review, and in the context of the Health Research Roadmap, CIHR is enhancing the peer review process for the Fellowship program. The enhancements include:
- Increasing the number of reviewers per application to three;
- Submitting preliminary assessments by reviewers prior to discussion;
- Facilitating the discussion of applications via online chat;
- Submitting final assessments at the end of the review process;
- Eliminating the re-review and third review process;
- Eliminating the calibration exercise, which included the benchmark application; and,
- Allowing reviewers to inform on their ability to assess applications.
2) Introduction
On behalf of CIHR, we would like to thank you for agreeing to serve as a reviewer. The success of the peer review process is made possible by dedicated people like yourself who generously give their time and expertise. Your efforts are greatly appreciated by CIHR and the scientific community.
The peer review process is described in detail in this Guide for Reviewers and in the Peer Review Manual for Salary & Training Award Applications.
It is essential that reviewers read and be familiar with these documents and, most importantly, the objectives of the Funding Opportunity.
The purpose of these documents is to provide information on CIHR's objectives, governance and policies; to outline the roles and responsibilities of reviewers; and to define the policies and procedures for peer review of applications.
For detailed regulations concerning all aspects of CIHR funding programs, please see the Grants and Awards Guide.
3) Peer Review Principles
- Avoid conflict of interest;
- Respect the confidentiality of applications;
- Respect deadlines.
4) Step by Step Instructions
- Step 1: Read this Guide and the Funding Opportunity
- Step 2: Review your Assigned Applications:
- Become knowledgeable of the evaluation criteria
- Read the applications
- Rate the applications
- Write comments supporting your ratings
- Flag issues for CIHR's attention
- Step 3: Complete survey questions
- Step 4: Submit preliminary reviews (ratings and comments)
- Step 5: Discuss the application with the other reviewers
- Step 6: Review/edit your ratings and comments
- Step 7: Submit final ratings and comments
5) Step 1: Read this Guide and the Funding Opportunity
It is essential that reviewers read and be familiar with this document and the objectives of the Funding Opportunity.
6) Step 2: Review Your Assigned Applications
A) Become knowledgeable of the Evaluation Criteria
There are three evaluation criteria for the Fellowship program:
- Achievements and Activities of the Candidate
- Characteristics and Abilities of the Candidate
- Research Training Environment
It is important to note that expectations may differ by research area/discipline (e.g., publication productivity can vary while comparing a biomedical researcher, a clinician or a social scientist).
The evaluation criteria and their weightings can be found in Annex 1. Reviewers should be familiar with the evaluation criteria and the differences in weighting. ResearchNet will apply the weighting to your rating and as such, the weighting of each criterion should not influence your rating.
1. Achievements and Activities of the Candidate
The Achievements and Activities of the Candidate criteria include these four sub-criteria:
- Candidate's Plans - Training Expectations
- Proposed Research Project
- Honours, Awards and Academic Distinction
- Publications and Related Research Achievements
Reviewers should consider the following in their evaluation of the achievements and activities of the candidate:
- Assess the clarity and logic in the explanation of the candidate's plans for a research career and the relevance of the proposed training.
- Determine if the proposed project is adequate to the candidate given their education, experience and interests. Is the project the right balance of challenge, importance of the research question and feasibility in relation to the candidate's experience and training? Note: it is not the project per se that is being assessed. It is the project as an integral part of the candidate's development as a researcher.
- Assess the number, importance and breadth of the candidate's special distinctions relative to their education, training and work experience. Also determine if the distinction is relevant to their research and if the recognition is regional, national or international.
- Note the length of time required to complete academic programs and any indications of special academic distinction.
- Look for evidence of achievements in research relative to opportunities to date. Bear in mind that opportunities to publish may vary according to research discipline and life course (e.g., time spent raising children).
- For publications, observe the number of co-authors and the position of the candidate's name in the authors list. Note that the importance of this position can vary depending on the discipline, etc. Also note the candidate's role in publications and their estimated percent contribution to the work, as well as the type of publication (article, chapter, book, etc.).
2. Characteristics and Abilities of the Candidate
To review the Candidate's characteristics and abilities, we ask you to take a look at the Sponsor Assessment forms. They should offer a perspective on the candidate from individuals who are familiar with their characteristics and abilities. Recognize that positive comments are common while negative ones are not. Read the supporting text carefully, taking note of the extent to which they justify the ratings.
To perform an accurate review of these assessments, the following should be considered:
- Do the detailed comments support the ratings outlined on the first page of the assessment?
- How long as the sponsor known the candidate?
- What is the relationship of the sponsor to the candidate?
3. Research Training Environment
Finally, in your review of the Research Training Environment, consider:
- Information on the supervisor's research experience, qualifications, honours and awards.
- The supervisor's publication record to get a sense of productivity, impact and collaboration (consider the different disciplines and their impacts on these).
- If the research environment, including space, facilities, and personnel support available is adequate. Review the information on grants currently held, noting the extent to which the supervisor was either listed as a principal or co-applicant for the funds. Get a sense of the resources available and the overall level of activity.
Review the supervisor's training record. Note for each listed trainee: the level of training, length of time with the supervisor, degree received (if applicable) and current position. Your assessment should take into consideration the career stage and discipline of the supervisor.
B) Read the Applications
Read all of your assigned applications before rating any of them. It is important to note that many candidates will likely be conducting research outside of your research specialty. We ask that you provide an overall assessment of the quality of the application. However, if you feel that your level of comfort of reviewing an application is unacceptably low, you should inform CIHR staff and they will assign the application to an alternate reviewer.
As you read the applications, be alert to unconscious bias related to gender, discipline or geographic location. Remember that:
- Career interruptions for child bearing and raising can influence opportunity for knowledge production, publications and related variables;
- Each discipline and environment offer different opportunities for research contributions, publication and other research related activities; and,
- The reputation of institutions should not affect your view of applicants or their research training environment.
To ensure that all applications are treated equally, reviewers are asked to base their evaluation only on the content of the application and not to complete any additional research (i.e. publications via PubMed, etc.). You are however free to consult published lists of journal impact factors when assessing the candidate's research accomplishments. Note that journal impact factors vary from one discipline to another and that they do not necessarily indicate the quality of individual articles.
As you examine each application, jot down notes to capture your impressions. The Reviewer Worksheet (Annex 2) provides a template that you could use. The worksheet is not submitted to CIHR.
C) Rate the Application
The raw ratings that you submit for each criterion on the 0.0 to 4.9 scale will be weighted automatically by CIHR in the calculation of an overall rating. Each criterion is described in details in Annex 1. CIHR uses a scale of 0.0 to 4.9 to rate applications with 4.9 being the highest possible rating. The purpose of the scale is to serve as a benchmark for peer reviewers as they rank the applications. In order to be considered for funding, an application must receive a rating of 3.5 or higher.
| Descriptor | Range | Outcome |
|---|---|---|
| Outstanding | 4.5 – 4.9 | May be Funded |
| Excellent | 4.0 – 4.4 | |
| Very Good | 3.5 – 3.9 | |
| Good | 3.0 – 3.4 | Not Fundable |
| Average | 2.0 – 2.9 | |
| Below average | 1.0 – 1.9 | |
| Not acceptable | 0.0 – 0.9 |
D) Write Comments Supporting Your Rating
Prepare brief comments that support your rating of the application. The objective of the written reviews is to provide constructive advice to applicant to assist them in improving the quality and efficiency of the proposed training.
- Keep it simple;
- Use familiar descriptors that align with your rating (i.e.: outstanding, excellent, very good, good…);
- Provide comments on the strengths and weaknesses only for each aspect of the application;
- Include justification for each topic, providing context and an explanation of your comments, if applicable;
- Be clear and concise, while expressing complete thoughts;
- Use objective and non-inflammatory language;
- Carefully avoid language that might be construed as sarcastic, flippant, arrogant, or inappropriate in any way.
E) Flag Issues for CIHR's Attention
Any concerns in the following areas should be brought to CIHR's attention for follow-up:
- Eligibility
- Ethics
- Human pluripotent stem cell research
- Budget justification
- Section 56 of the federal Controlled Drugs and Substances Act
Please report these concerns to CIHR staff via the online discussion tool or by email at Fellowships@cihr-irsc.gc.ca. Questions or interpretations of CIHR policy should not affect the review of an application and must not appear in any written assessment of an application. For detailed regulations concerning these issues, please see the Grants and Awards Guide.
7) Step 3: Complete Survey Questions
Your will be asked to answer the following mandatory questions for each application assigned to you.
- Rate your expertise in the applicant's proposed subject area.
- Rate your ability to review this application.
It is important that you take the time to answer these questions as your response will inform the development and enhancements of future adjudication process.
8) Step 4: Submit Preliminary Ratings and Reviews
Submit your reviews once you have reviewed, rated and written comments for every application assigned to you.
9) Step 5: Discuss the Application with the Other Reviewers
A) Accessing Other Reviewers Reviews and Online Discussion
Once your preliminary ratings and reviews are submitted, you will have access to the submitted ratings and written reviews from other reviewers. Read their reviews and identify topic of agreement and disagreement. Also compare your rating of the application with those of the other reviewers.
The online discussion function will only be available once all reviewers assigned to an application have submitted their preliminary ratings and written comments. A “pending” status will signal that the discussion cannot begin until all reviewers have submitted their reviews.
B) Using the Online Discussion
Once all reviewers assigned to the application have submitted their reviews, the online discussion function will become accessible. The following statuses will help you navigate the online discussion function in ResearchNet:
- Pending: You cannot begin the discussion. You must wait for the other reviewers to submit their preliminary ratings.
- Start Discussion: Allows you to be the first to post a comment.
- Active: The discussion has begun.
- Active with star symbol (*): A reviewer posted a new comment since the last time you accessed the discussion.
- Closed: All reviewers submitted their final ratings.
- Suspended: The online discussion is closed until further notice by CIHR.
Once a comment is posted, you cannot delete or edit this comment. Comments can be posted for the attention of CIHR staff; however it will be visible by the other reviewers. This option may be useful for flagging issues for CIHR attention.
A notification email will be sent daily to advise reviewers of new posts on the online discussion board.
10) Step 6: Review/Edit Your Ratings and Comments
At the end of the online discussion, you may modify your ratings and edit your comments. It is important that your written review reflect your final assessment of the application.
Reviewers are required to submit their final rating of all applications assigned to them, whether or not the review was modified following the online discussion period.
11) Step 7: Submit Final Ratings and Comments
Submit your reviews once you have finalized your ratings and written comments for all applications assigned to you.
Annex 1: Evaluation Criteria
1. Overview of the six evaluation criteria
The Fellowship evaluation criteria and weightings were established based on studies of the predictors of post-training research activity and through consultation with past Fellowship reviewers.
The ratings that you submit for each criterion on the 0.0 to 4.9 scale will be weighted automatically by ResearchNet in the calculation of an overall rating.
| Criteria | Weight for Post-PhD Candidates1 | Weight for Health Professional Candidates without PhD2 | Overall Weight |
|---|---|---|---|
| Achievements and Activities of the Candidate | |||
|
|||
| Candidate's Plans (Training Expectations) | 10% | 10% | 60% |
| Proposed Research Project | 10% | 10% | |
| Honours, Awards and Academic Distinction | 5% | 15% | |
| Publications and Related Research Achievements | 35% | 25% | |
| Characteristics and Abilities of the Candidate | |||
| Characteristics and Abilities | 20% | 20% | 20% |
| Research Training Environment | |||
| Research Activity, Resources and Mentorship | 20% | 20% | 20% |
| 100% | 100% | 100% | |
1a) Evaluating the Candidate's Plans
10% weight for all candidates
A) Working Definition
A description of the applicant's career intentions and proposal for achieving them.
B) What to Look For
Clarity and logic in the explanation of the candidate's plans for a research career and the relevance of the proposed training.
| Rating Range | Description |
|---|---|
| 4.5 - 4.9 | Outstanding Faultless depiction of research career intentions and relevance of the proposed training. Ideal career path. |
| 4.0 - 4.4 | Excellent Clear, convincing depiction of research career intentions and relevance of the proposed training. Highly appropriate career path. |
| 3.5 - 3.9 | Very good Very good depiction of research career intentions and relevance of the proposed training. Logical career path. |
| 3.0 - 3.4 | Good Reasonable depiction of research career intentions and relevance of the proposed training. Suitable career path. |
| 2.0 - 2.9 | Average |
| 1.0 - 1.9 | Below Average |
| 0.0 - 0.9 | Not Acceptable |
1b) Evaluating the Proposed Research Project
10% weight for all candidates
A) Working Definition
A carefully planned, systematic study aimed at clearly answering a question in health research.
B) What to Look For
The ideal project is one that is best for the candidate given their education, experience and interests. It is the right balance of challenge, importance of the research question and feasibility in relation to the candidate's experience and training.
Bear in mind that it is not the project per se that is being assessed. It is the project as an integral part of the candidate's development as a researcher.
| Rating Range | Description |
|---|---|
| 4.5 - 4.9 | Outstanding Extraordinary optimization of: challenge to the candidate, scientific importance and feasibility of completion during the Fellowship period. An ideal project that is faultlessly outlined. |
| 4.0 - 4.4 | Excellent Excellent optimization of: challenge, importance and feasibility. A highly suitable project that was superbly outlined. |
| 3.5 - 3.9 | Very Good Strong optimization of: challenge, importance, and feasibility. A very suitable project that was very clearly outlined. |
| 3.0 - 3.4 | Good Good optimization of challenge, scientific importance and feasibility. A suitable project that was well outlined. |
| 2.0 - 2.9 | Average |
| 1.0 - 1.9 | Below Average |
| 0.0 - 0.9 | Not Acceptable |
1c) Evaluating the Honours, Awards and Academic Distinction of the Candidate
5% Weight for Post-PhD applicants
15% for Health Professional without PhD applicants
A) Working Definition
Official recognition or prizes that signifying special qualities of the recipient. This includes accomplishments in terms of formal education and scholarships.
B) What to Look For
When assessing these variable and other achievements of the candidate, it is essential to take into consideration the career path the candidate has followed to date. Assess the number, importance and breadth of the candidate's special distinctions relative to their education, training and work experience. Note relevance to research and whether the recognition is regional, national or international. Note the length of time required to complete academic programs and any indications of special academic distinction.
| Rating Range | Description |
|---|---|
| 4.5 - 4.9 | Outstanding All aspects of the candidate's distinctions (their number, importance and breadth) indicate recognition of a very rarely encountered level of talent. |
| 4.0 - 4.4 | Excellent Several aspects of the candidate's distinctions (their number, importance or breadth) indicate recognition of superb talent. |
| 3.5 - 3.9 | Very Good At least one aspect of the candidate's distinctions (their number, importance or breadth) indicates recognition of talent. |
| 3.0 - 3.4 | Good The candidate's distinctions indicate an above-average performance. |
| 2.0 - 2.9 | Average |
| 1.0 - 1.9 | Below Average |
| 0.0 - 0.9 | Not Acceptable |
1d) Evaluating the Publications and Related Research Achievements of the Candidate
35% Weight for Post-PhD applicants
25% for Health Professional without PhD applicants
A) Working Definition
Contributions such as articles, chapters or books published (particularly peer-reviewed) as well as conference presentations, abstracts and evidence of practical impact such as patents or copyrights.
B) What to Look For
Evidence of achievements in research relative to opportunities to date. Bear in mind that opportunities to publish may vary according to research discipline and life course (e.g., time spent raising children).
For publications, observe the number of co-authors and the position of the candidate's name in the authors list (note that the importance of this position can vary depending on the discipline, etc.). Note the candidate's role in publications and their estimated percent contribution to the work, as well as the type of publication (article, chapter, book, etc.).
Try to get a sense of the entire body of work and its likely impact. Note the publication dates and relate them to the candidate's education and training. Consider the list of abstracts as an indication of conference presentation activities. Note the candidate's other professional activities. Also consider any patents, copyrights, policy documents, etc. to which the candidate contributed.
| Rating Range | Description |
|---|---|
| 4.5 - 4.9 | Outstanding All aspects of the candidate's publications and related research achievements (number, likely impact and breadth) indicate an extraordinarily productive and creative individual in their field of research/discipline. |
| 4.0 - 4.4 | Excellent Several aspects of the candidate's publications and related research achievements (number, likely impact or breadth) indicate excellent productivity and creativity in their field of research/discipline. |
| 3.5 - 3.9 | Very Good At least one aspect of the candidate's publications and related achievements (number, likely impact or breadth) indicate very good productivity or in their field of research/discipline. |
| 3.0 - 3.4 | Good There is evidence of greater than expected involvement in publication and related research activities in their field of research/discipline. |
| 2.0 - 2.9 | Average |
| 1.0 - 1.9 | Below Average |
| 0.0 - 0.9 | Not Acceptable |
2. Characteristics and Abilities of the Candidate
20% weight for all candidates
A) Working Definition
A perspective on the candidate provided by persons who are familiar with their characteristics and abilities.
B) What to Look For
Evidence from the sponsors that the candidate exhibits the characteristics and skills that correlate with career research achievement. Examine the sponsor's assessments, recognizing that positive comments are common while negative ones are not. Read the supporting text carefully, taking note of the extent to which they justify the ratings.
Look particularly for indications that the sponsors perceive the candidate as an investigative type, that is, someone whose thinking is critical, questioning, original and independent. Also look for indications that the sponsors perceive the candidate as both energetic and capable of being highly focused.
If the candidate has had an opportunity to conduct research, look for mention of creativity in setting research goals, designing experiments, developing new methodologies, interpreting findings and presenting results in writing.
| Rating Range | Description |
|---|---|
| 4.5 - 4.9 | Outstanding An outstandingly critical, original and independent thinker. Exceptionally focused, energetic and creative. An ideal role model for others. |
| 4.0 - 4.4 | Excellent A highly critical, original and independent thinker. Very focused, energetic and creative. Excellent potential for future research leadership. |
| 3.5 - 3.9 | Very Good Clearly a critical, original and independent thinker. Definitely focused, energetic and creative. Very good potential for success as an independent researcher. |
| 3.0 - 3.4 | Good Appears to be a critical, original and independent thinker. Seems to be focused, energetic and creative. Above average potential for a productive career in research. |
| 2.0 - 2.9 | Average |
| 1.0 - 1.9 | Below Average |
| 0.0 - 0.9 | Not Acceptable |
3. Evaluating the Research Activity, Resources and Mentorship in the Training Environment
20% weight for all candidates
A) Working Definition
Elements of the research environment that will contribute directly or indirectly to the quality of the candidate's research training experience.
B) What to Look For
Review information on the research experience, qualifications, honours and awards of the proposed supervisor. Examine the supervisor's contribution (e.g. publications, presentations, policy documents, etc.) record to get a sense of productivity; impact and collaboration (as noted above, please consider the different disciplines and their impacts on these).
Determine if the proposed research environment, including space, facilities, and personnel support available is appropriate. Review the information on grants currently held, noting the extent to which the supervisor was either listed as a principal or co-applicant for the funds. Get a sense of the resources available to support the candidate and the overall level of activity of the research environment.
Review the supervisor's training record. For each listed trainee, note the level of training, length of time with the supervisor, degree received (if applicable) and current position. Your assessment should take into consideration the career stage and discipline of the supervisor. Your expectations of mentoring by a recently-established investigator should differ from your expectations of mentoring by a long-established researcher.
| Rating Range | Description |
|---|---|
| 4.5 - 4.9 | Outstanding A vibrant, world-class research environment. Outstanding availability of research resources. Superb mentorship. |
| 4.0 - 4.4 | Excellent A highly active research environment. Excellent availability of research resources. First-rate mentorship. |
| 3.5 - 3.9 | Very Good A very active research environment. Very good availability of research resources. Strong mentorship. |
| 3.0 - 3.4 | Good An active research environment. Sufficient research resources available. Appropriate mentorship. |
| 2.0 - 2.9 | Average |
| 1.0 - 1.9 | Below Average |
| 0.0 - 0.9 | Not Acceptable |
Annex 2: Reviewer Worksheet
This template is strictly for your working notes and will not be filed with CIHR.
| Name of Applicant | |
| Application Number | |
| Background Information on the Candidate Refer to the CCV of the Candidate |
Degrees held or in progress: Time for completion of degree programs: Research experience: |
| Candidate's Plans Refer to the "Training Expectations" section of the ResearchNet task entitled "Application Details Attachments" |
Link between proposed and prior training: Career goals: |
| Proposed Research Project Refer to the following tasks on ResearchNet: Lay Abstract and Application Details Attachments (section entitled "Research Proposal Summary") |
Project: Expected duration of training: Suitability of project for applicant's career: |
| Honours, Awards and Academic Distinction Refer to the CCV of the Candidate |
|
| Publications and Related Research Achievements Refer to the CCV of the Candidate Note: Be aware of the variances between disciplines. |
Papers: Presentations: Other: Candidate's role: |
| Characteristics and Abilities Refer to sponsors' assessments |
Sponsor: Relationship to candidate: Notes: |
| Sponsor: Relationship to candidate: Notes: |
|
| Sponsor: Relationship to candidate: Notes: |
|
| Research Activity in the Training Environment Refer to the CCV(s) of the proposed supervisor(s) Note: Be aware of the variances between disciplines. |
Name of proposed supervisor: Qualifications: Publication activity: Impact: Collaborators: |
| Resources Available in the Training Environment Refer to the following tasks on ResearchNet: Space, Facilities and Personnel Support, CCV(s) of the proposed supervisor(s) |
Space, facilities and personnel support available: Research funding available to the supervisor: Other key resources: |
| Mentorship Record in the Training Environment Refer to the CCV(s) of the proposed supervisor(s) |
Trainees in the last five years - level and outcome: |
| General Notes on the Application | Overall impression: |
Annex 3: Guide to Locating Information
| Criteria | Section (s) Where to Find Information |
|---|---|
| Candidate's Plans |
Reason for Selecting a Foreign Training Environment (if applicable) |
| Proposed Research Project |
|
| Honours, Awards and Academic Distinction |
|
| Publications and Related Research Achievements | The “Contributions” section of the candidate's Academic CCV should list the following:
|
| Characteristics and Abilities |
|
| Research Activity, Resources and Mentorship in the Training Environment |
The Academic CCV of the supervisor (s) of the candidate should contain the following:
|
Supplemental content (right column)
- Modified: