CIHR Peer Review Guide for Training and Salary Awards

Table of Contents

  1. Introduction
  2. Policies and Considerations Impacting Peer Review
  3. Principles of Peer Review
  4. Peer Review Logistics
  5. Review Process
  6. Feedback
  7. Funding Decisions

1. Introduction

On behalf of CIHR, we thank reviewers for agreeing to serve as a peer review committee member. The success of the peer review process is made possible by dedicated people like you who generously give their time and expertise. Your efforts are greatly appreciated by CIHR and the scientific community. The purpose of this document is to provide information on the policies, procedures and process for the peer review of  training and salary awards applications.

It is important to note that some policies may not apply to all training or salary award programs.

Members of the Peer Review Committees are expected to review this document as they prepare for and conduct evaluations of applications. It is essential that members of a peer committee read and become familiar with the pertinent documentation associated with the Funding Opportunity for which they are reviewing. They are also expected to become familiar with:

2. Policies and Considerations Impacting Peer Review

The following are the policies guiding peer review:

2.1 International Collaboration (if applicable)

Applications focused on global health, or which include international collaborations can be eligible if outlined on a training or salary award program. Applications focused on the field of global health that demonstrate that the proposed research project has the potential to improve health in Canada and/or the global community are accepted and encouraged as more than ever before, the health of Canadians is deeply intertwined with the health of people around the world. Global health research is defined as any research with a focus on the health of people living in low- and middle-income countries, as well as research that aims to understand the systematic factors that shape health and are inherently global (e.g., globalization, equity, neglected diseases, and transnational risks). Global health is relevant to all pillars. Please consult International collaboration website and Global Health 3.0: CIHR's Framework for Action on Global Health Research for more information.

2.2 Knowledge Translation

Knowledge translation (KT) is integral to CIHR's mandate and falls into two main categories: end of grant KT and integrated KT. With both categories of knowledge translation, CIHR expects researchers and trainees to disseminate their findings and facilitate their translation into improved health, more effective products or services, and/or a strengthened healthcare system.

For end of grant KT, many means of dissemination exist and the onus is on the researchers to select the most appropriate vehicle for the intended knowledge-user audience to ensure maximum impact. When the primary knowledge users are researchers, dissemination of results through the publication of high quality articles in accessible journals is appropriate, although other strategies that increase awareness of the results and facilitate their application may also be appropriate. When knowledge-user audiences outside the research community should be informed of specific research findings, dissemination plans with more ambitious goals and comprehensive strategies are expected. With integrated KT, stakeholders or potential knowledge users are engaged in the entire research process and the research is directed at producing solutions to issues or problems the stakeholders/knowledge users have identified.

2.3 Open Access Policy on Publications

CIHR believes that greater access to research publications and data will promote the ability of researchers and knowledge users in Canada and abroad to use and build on the knowledge needed to address significant health challenges. Open access will promote accessibility to CIHR-funded research and will serve to increase the international visibility of Canadian research. CIHR grant and award recipients are reminded to adhere to the responsibilities outlined in the Tri-Agency Open Access Policy on Publications.

2.4 Sex- and Gender-Based Analysis (SGBA) and Health Research

CIHR expects that all applicants will integrate sex and gender into their research designs when appropriate. Sex- and gender-based analysis (SGBA) is an approach that systematically examines sex-based (biological) and gender-based (socio-cultural) differences between men, women (during infancy, childhood, adulthood, etc.) as well as gender-diverse people. The purpose of SGBA is to promote rigorous science that is sensitive to sex and gender and therefore has the potential to expand our understanding of health determinants for all people. Reviewers are asked to explicitly assess whether the integration of sex (as a biological variable) and/or gender (as a socio-cultural factor) is a strength, a weakness or not applicable to the proposal. Reviewers are also asked to comment on their assessment and to provide recommendations to the applicants on how they might improve the strength of their applications with respect to the integration of sex and/or gender. The SGBA section of the CIHR website provides helpful resources for applicants and peer reviewers alike, including CIHR's definitions for sex, gender, and SGBA, as well as information on applying SGBA to the development and assessment of research proposals.

2.5 Gender-neutral and gender-inclusive language in reviews

Reviewers are encouraged to adopt gender-neutral and gender-inclusive language in their reviews. In reviews, both written and spoken, gender-neutral and gender-inclusive language is more accurate and more respectful when discussing the science and the applicant.

When discussing the science remember that gender is non-binary. When research is meant to include all people, avoid binary statements like "men and women." Instead, consider phrases like "men, women, and gender-diverse people" or "people of all genders." In addition, be mindful of word choice. Below are a few examples of words that could be replaced with more gender-neutral and gender-inclusive terms:

  • "Female/male anatomy" with specific anatomical language (e.g., ovaries, uterus, testes)
  • "Mankind" with "humankind" or "people"
  • "Man-made" with "machine-made", "synthetic" or "artificial"

When referring to the applicant, use gender neutral pronouns or phrases. For example, use "they" or "the applicant," rather than "he" or "she". Remain mindful of word choice, as some words could be replaced with more gender-neutral and gender-inclusive terms:

  • "Chairman" with "Chair" or "Chairperson"
  • "Maternity/paternity leave" with "parental leave"

2.6 Official Languages

In order to ensure that positive measures are undertaken within CIHR's mandate to contribute to the vitality of English and French minority communities in Canada, CIHR:

  • Encourages the appropriate inclusion of official language considerations of English and French speaking Canadians, including those living in linguistic minority communities, in health research design, conduct, and application to improve health outcomes; and
  • Facilitates equitable access to its programs and services for health research stakeholders in official language minority communities.

2.7 San Francisco Declaration on Research Assessment

CIHR is a signatory of the San Francisco Declaration on Research Assessment (DORA), which "recognizes the need to improve the ways in which the outputs of scholarly research are evaluated". DORA recognizes that scholarly outputs include a broad range of contributions and impacts and are not limited to published journal articles.

Reviewers are to assess productivity and progress broadly by taking into consideration:

  • a range of contributions (e.g., research publications, reports, books, guidelines, datasets, code, tools, training and mentorship, volunteerism, community engagement, standards, software and commercialized products) and impacts (e.g., influence on policy and practice, health outcomes, societal outcomes, and distinctions-based, meaningful and culturally safe health research), and
  • the context of the applicant (e.g., leave history, career stage, area(s) of research, experiential knowledge, diverse career paths, family responsibilities, pandemic impact) and how it may have affected their progress.

Metrics such as number of publications and citations, and size/number of research grants should not be used in isolation to assess productivity. Reviewers should not use journal-based metrics (e.g., Journal Impact Factors) as surrogate measures of quality and impact of individual research publications. As stated in DORA, the "scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published".

3. Principles of Peer Review

The integrity of the peer review process relies on well-established principles and policies that ensure fair and effective evaluation and support CIHR’s mandate and objectives.

CIHR’s principles of review are:

  • Confidentiality;
  • Absence of conflict of interest;
  • Fairness; and,
  • Transparency.

3.1 Confidentiality

Confidentiality is information about a person that will not be disclosed directly or indirectly to anyone else without that person's prior expressed consent. The information provided by applicants in their applications is protected by the Privacy Act and is made available to external assessors for reviewing purposes only. The use of this information for any other purpose is a breach of the Privacy Act and could result in a CIHR investigation and/or report to the federal Privacy Commissioner's Office.

Committee members are not to approach or communicate in any way with applicants, or anyone outside of the committee, regarding any information relating to the review of a specific application or offer opinions on the applicant’s chances of success. In turn, applicants are not to contact committee members, including the Chair(s) and Scientific Officer(s) (if applicable), regarding the status of their applications (for example, ratings).

By law, applicants have access to their own application files. Therefore, all written materials used in evaluating an application are made available to the applicants when they are notified of CIHR's funding decision for a competition.

The identity of the reviewers will not be revealed to the applicants. However, a list of review committee members is typically published on the CIHR website 60 days after the approval of funding of a competition.

3.2 Absence of conflict of Interest

CIHR must meet the highest ethical and integrity standards in all that it does in order to continue to merit the trust and confidence of the research community, the government and the public. CIHR review committee members, external reviewers and observers must meet the highest standards of ethical behaviour to maintain and enhance public confidence in CIHR’s ability to act in the public’s best interest and for the long-term public good. Where a conflict arises between private and public interests, review committee members, external reviewers, and observers will be expected to take the necessary measures to ensure that the public interest is protected.

According to the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations (COIC), a conflict of interest means a conflict between a participant's (e.g., reviewers and observers) duties and responsibilities with regard to the review process, and a participant's private, professional, business or public interests. There may be a real, perceived or potential conflict of interest when the participant:

  • would receive professional or personal benefit resulting from the funding opportunity or application being reviewed;
  • has a professional or personal relationship with an applicant or the applicant's institution;
  • has a direct or indirect financial interest in a funding opportunity or application being reviewed; and/or
  • is currently under investigation for an alleged breach of Funding Organization policies.

A conflict of interest may be deemed to exist or perceived as such when committee members, reviewers or observers:

  • are applicants within the competition and have ability to bias or influence the process to the benefit of their application;
  • are a relative or close friend, or have a personal relationship with the applicants/supervisor(s);
  • are in a position to gain or lose financially/materially from the funding of the application;
  • have had long-standing scientific or personal differences with the applicants/supervisor(s);

  • are currently affiliated with the applicants' and/or supervisor(s) institutions, organizations or companies—including research hospitals and research institutes;

    Note: A reviewer is not automatically in conflict with an application if they are from the same institution as the applicant/supervisor(s), but do not know or work with the applicant/supervisor(s).

  • are closely professionally affiliated with the applicants/supervisor(s), as a result of having in the last six years:
    • frequent and regular interactions with the applicants/supervisor(s) in the course of their duties at their department, institution, organization or company;
    • been a supervisor or a trainee of the applicants;
    • collaborated, published or shared funding with the applicants/supervisor(s), or have plans to do so in the immediate future; or,
    • been employed by the institution, when an institution is the applicant/supervisor(s); and/or
  • feel for any reason unable to provide an impartial review of the application.

All committee members (Chairs, Scientific Officers, reviewers) and observers are subject to the same conflict of interest guidelines. CIHR reserves the right to resolve areas of uncertainty and to determine if a conflict exists.

All committee members must read and agree to abide by the Conflict of Interest and Confidentiality Agreement for Peer Reviewers and Peer Review Observers prior to viewing any application information.

3.3 Fairness

The success of the peer review system is critically dependent upon the willingness and ability of all committee members to:

  • exercise rigorous scientific judgment;
  • be impartial and reasonable;
  • understand and take into account, in a balanced way, the particular context of each application; and
  • provide a constructive, quality review which helps applicants by pointing out strengths and weaknesses that contributed to the application rating.

In programs where written reviews are required, these reviews are provided to the applicant without prior editing by CIHR staff, and CIHR does not take responsibility for their content. An applicant will not accept that the review is fair if it contains comments that could be construed as sarcastic, flippant, arrogant, or inappropriate in any way. Conversely, a constructive review, that helps the applicant by pointing out deficiencies that could be repaired in a resubmission, will help convince a disappointed applicant that a fair assessment of the proposal was provided.

3.4 Transparency

CIHR ensures transparency in the review process through a number of different mechanisms. All applications submitted to CIHR are evaluated by reviewers who provide an overall assessment of the application. This assessment is performed using the application requirements and review criteria established for each individual competition, as seen in the relevant funding opportunity details. In addition, for competitions involving a face-to-face peer review meeting, the Scientific Officer (SO) notes capture the committee discussion during the peer review deliberations. All reviews and SO notes, as applicable, are shared with applicants.

On its website, CIHR publishes the Peer Review Committee Membership Lists when appropriate, and lists all CIHR successful applications by competition through its Funding Decisions Database.

4. Peer Review Logistics

4.1 Review Types and Meeting Formats

Peer Review at CIHR is typically conducted using one of two fundamental types of review: structured review or unstructured review – although, in some instances, other methods of adjudication may be used where appropriate. The meeting format can be a face-to-face meeting, a teleconference and/or a virtual meeting (at-home) using the ResearchNet web platform.

Unstructured review means that the peer reviewers "weigh" the evaluation criteria as they see fit to determine a final score; it is commonly used by committees with face-to-face committee meetings.

For structured review, reviewers assign scores to a set of predetermined evaluation criteria, where the final score is generated automatically according to the weighting assigned to each criterion. The process is typically done entirely on ResearchNet.

Notwithstanding the type of review, the objective remains the same - to fund excellence. The following key points determine the type of review and meeting format used for a Funding Opportunity:

  1. Program objectives;
  2. Anticipated number of applications;
  3. Length of applications;
  4. Nature of the assessment required (i.e., weighted criteria, reaching consensus, ranking).

Reviewers are invited to consult the appropriate program-specific Peer Review Guide and/or program literature to find out what type of review will be used for their committee and the detailed step-by-step instructions of the process.

4.2 Committee Members

The committee membership will vary depending on the type of peer review. All committee members are selected for their research excellence, as reflected by their ability to obtain continued extramural peer-reviewed funding, and for their breadth of knowledge and maturity of judgment.

Committees, as a whole, should also satisfy the need to cover the range of research areas and/or perspectives for which the committee is responsible, to appropriately represent the Canadian health research community, to review in both official languages, and to allow for the logistics of conflict of interest and turnover of committee members.

For further details on committee membership, please consult the roles and responsibilities of committee members. Note that for committees that do not have any face-to-face committee meetings, all committee members assume the Reviewer role.

5. Review Process

All applications received by the appropriate deadline date are reviewed by CIHR staff for eligibility and compliance. Those deemed not eligible are withdrawn from the competition. Note that peer review activities are performed on ResearchNet, unless otherwise stated.

Step 1: Attending the Orientation Session

The objective of the orientation session is to familiarize the reviewers with the program, explain the peer review process and provide details on how to complete the review.

Step 2: Declaring Conflicts and Assignment of Applications

Reviewers cannot assess applications if they have a conflict of interest with the said application(s).

Depending on the type of review being completed, the reviewers will be given access to the applications to declare conflicts of interest and, if applicable, indicate their level of expertise prior to the assignments. Chairs, Scientific Officers and/or CIHR staff assign the applications to committee members. External members can also be solicited if required. However, the final authority for the assignments rests with CIHR.

Step 3: Evaluating Applications

All applications submitted to a funding opportunity are treated equally for evaluation; the same criteria and funding cut-offs are applied to all. The applications are evaluated in reference to adjudication criteria that are specific to each program. These criteria and how to review applications are listed in the funding opportunity details and in program-specific Reviewers’ Guide and/or program literature.

CIHR’s Training and Salary Awards peer review mechanisms commonly use a scale of 0.0 to 4.9 to rate applications, with 4.9 being the highest possible rating. Ratings are assigned qualitative descriptors (see table below) to help guide reviewers in their rating of applications. In order to be considered for funding, an application must receive a score of 3.50 or higher. Other rating scales may be used to support the objectives of a specific competition.

Descriptor Range Outcome
Outstanding 4.5 – 4.9 May be Funded
Excellent 4.0 – 4.4
Very good 3.5 – 3.9
Good 3.0 – 3.4 Not Fundable
Average 2.0 – 2.9
Below Average 1.0 -1.9
Not Acceptable 0.0 – 0.9

Depending on the program (thus the review type being used), the evaluation of applications can be completed in multiple stages.

Information provided in the application package must be self-contained. Hyperlinks and links to documents hosted on a Google drive (or other similar drive) should not be accessed in the review of applications.

To ensure that all applications are treated equally, reviewers are asked to base their evaluation only on the content of the application and avoid doing any additional research (e.g. looking for publications via PubMed).

In order to ensure that all applicants have exactly the same amount of space to write their research proposals, applicants are instructed to adhere to the acceptable application format requirements.

Unconscious bias

As reviewers read the applications, they should be alert to unconscious bias related to gender, cultural (including Indigenous and geographic) bias, age, language (including official language and minority communities bias) and institution and remember that:

  • Career interruptions for child bearing and raising can influence opportunity for knowledge production, publications and related variables;
  • Different disciplines, environments and diverse career paths offer different opportunities for research contributions (e.g., publication, influence on policy and practice, patents, knowledge mobilization activities, etc.);
  • The reputation of institutions should not affect the reviewer’s view of applicants or their research training environment;
  • It is important to take steps to mitigate bias in reviewers’ thought process about difference in culture (e.g. Indigenous Peoples of Canada);
  • Recognize Indigenous identity in a reconciliation, nation-to-nation framework; and,
  • A point should be made of respecting Canada’s linguistic duality by recognizing the value of research in French and the value of research on Francophone minority communities.

To learn more about bias in peer review, reviewers are encouraged to complete the related learning module.

Publications and Productivity

An important evaluation criterion in most training and salary awards funding programs is the excellence of the applicant(s). A key factor in assessing this criterion is the productivity of the applicant(s), as determined by the quality (e.g., distinctions-based, meaningful and culturally safe health research) and impact (e.g., influence on policy and practice, health and societal outcomes) of contributions to the field. When assessing publications, peer review committees should focus on the quality of a publication's content. In the case of multi-authored publications or other collaborative work, applicants are advised to describe their contribution and reviewers should assess the specific contribution of the applicant to the work. As mentioned above, CIHR signed the San Francisco Declaration on Research Assessment (DORA), which recognizes that scholarly outputs include a broad range of contributions and impacts and are not limited to published journal articles. For more information about DORA, please visit CIHR’s DORA webpage.

CIHR funds individuals in many health-related areas, and the forms of research publications can vary greatly among disciplines. In addition to the more traditional peer-reviewed journals, health researchers also publish in books, monographs, memoirs or special papers, review articles, conference/symposia proceedings and abstracts, government publications, etc. Some fast-moving research fields, such as some areas of computing science, genetics or microelectronics, use special means to reach the target audience quickly. Communications, quick-print reports, letters and electronic distribution of pre-prints are important vehicles for disseminating research results. All such contributions should be treated equally when assessing quality and impact, and reviewers should not regard certain types as "second class" or "grey literature."

When assessing productivity, reviewers should also be sensitive to legitimate delays in research and dissemination of research results. It is important that reviewers take into account the context of the applicant (e.g., health problems, family responsibilities, disabilities, trauma and/or loss, pandemic impact, area(s) of research, experiential knowledge and diverse career paths) and how it may have impacted the applicant’s performance.

Some circumstances make it impossible or undesirable for researchers to publish important results of their research prior to applying for CIHR support. For instance, the time required to complete a monograph may exceed the time available between consecutive applications, or the protection of intellectual property may require a delay in publication. Peer reviewers should consider the Special Circumstances document (if provided) as research productivity may also vary as a result of personal circumstances, including when gender, race, diversity, ability, sexuality, health disparities, educational access, etc., have played a role in shaping an individual’s path. Thus, reviewers should also take into account the career stage of the applicants to better assess and calibrate their set of applications.  Any information about interruptions to an individual’s academic career may be included, such as pregnancy or early childcare, administrative leave, disability, elder care, etc., whether or not a formal leave of absence is taken. Applicants are advised to clearly and fully describe any circumstances that affect the dissemination of research results in their application. Reviewers must be sensitive to the impact of these circumstances on the level of productivity, while ensuring that the quality of the research remains competitive.

Note: Some policies may not apply to all training or salary award programs and reviewers should refer to the program-specific Reviewers’ Guide or contact the committee coordinator for more information.

Step 4: Streamlining (if applicable)

For many programs involving a face-to-face meeting, less than one-third of the applications are ultimately funded; thus, it is important that committees focus their time and efforts on the most competitive applications to ensure that an accurate rank-order list is generated. To help support this goal, some programs may use a streamlining process to eliminate non-competitive applications, allowing peer reviewers more time to judge and discriminate between potentially successful applications and helping to ensure that the most deserving applications receive funding.

Step 5: Special Considerations

Budget and Term

The budget and term for training and salary award programs are predetermined and are stated in the Funding Opportunity details. Nevertheless, given that all programs have different objectives, some committees could require reviewers to comment on those aspects of the applications. Questions about the requested budget should not influence the rating of the application, unless they bear directly on the scientific merit. For further details, please refer to the budget and term details in the funding opportunity, at the beginning of the peer review process.

Special Attention Issues

Any concerns in the following areas should be flagged for CIHR staff to address, and should not be noted in the written comments. Note that some issues listed below may not apply to all training or salary award programs. Please contact the corresponding CIHR program staff for more information.

These issues are not to be considered as criteria for evaluation unless they impact on the scientific quality of the application, as detailed below.

  1. Eligibility: Reviewers should raise to CIHR staff any concerns with respect to whether the Principal Applicant(s) and their affiliated institutions meet the criteria specified in the Funding Opportunity to receive CIHR funding.
  2. Ethics: Responsibility for ensuring that all research meets ethical standards is delegated to the local institution by CIHR. Ethics forms are not required as part of the application. However, the reviewer may comment on specific issues, such as the use of human subjects, animals, human tissues or hazardous material, or research that appears to involve Aboriginal people, if they feel they have not been adequately addressed.
  3. Human pluripotent stem cell research: Applications involving the use of human stem cells and likely to be funded will also be reviewed by the Stem Cell Oversight Committee (SCOC). Applicants are instructed to check the relevant box in the section entitled “Certification Requirements”, but it is essential that this be verified by committee members.
  4. Budget justification: Issues related to the budget should be brought to the attention of CIHR staff if the peer review committee cannot properly assess the budget request because of an unclear justification by the applicant. CIHR staff will follow up before funds are released, if the application is funded.
  5. Section 56 of the federal Controlled Drugs and Substances Act: All research proposals that are subject to Section 56 of the Controlled Drugs and Substances Act are required to have an exemption from Health Canada. Committee members should flag such applications to CIHR staff at the meeting who will follow up before funds are released, if the application is funded.

Similarly, if you believe there may be a misrepresentation of information in an application or a breach of Agency policy, please inform us immediately so we can determine whether the issue needs to be addressed through the responsible conduct of research (RCR) process. For a list of potential breaches, please refer to the Tri-Agency Framework: Responsible Conduct of Research.

6. Feedback

An important component of the peer review process is the review of the committee’s effectiveness and functioning, and feedback on policy issues that may have arisen in the course of the process. This feedback provides an opportunity for CIHR staff to address any concerns of the committee members and for staff to record comments on the peer review process as part of CIHR’s ongoing efforts to maintain an effective and high-quality peer review system.

For committees that have a meeting, this occurs at the end of meeting deliberations. However, for committees that do not have a meeting, feedback should be communicated to the corresponding CIHR program staff by email.

7. Funding Decisions

Following peer review, CIHR staff generates a rank list based on the committee recommendations, to be reviewed by CIHR’s Vice-President, Research – Programs and Chief Financial Officer (CFO). Applications will be funded from the top down in order of ranking as far as the budget will allow. The Vice-President, Research - Programs and CFO consider the funding recommendation in light of criteria established by Scientific Council (SC) and submit their recommendations to SC. Following the recommendations from SC, CIHR's funding decisions are approved by the President. A list of successful applicants is posted on the Funding Decisions Notifications page.

Applicants are informed of the results of the competition via ResearchNet once the President has approved the list of applicants to be funded. All applicants are sent a Notice of Decision, indicating whether or not their application was approved. They will also receive a copy of all reviews, the Scientific Officers notes (if applicable) and, for the successful candidates, an Offer of Award that details the budget, term and conditions of funding.

Applications that have been flagged for special attention and followed up by CIHR staff are withheld as “pending”. The applicant will be notified if further information is required. The additional information may be discussed by CIHR staff and peer review committee members, if necessary, prior to a final decision regarding funding.

CIHR will not question scientific evaluations made to the best knowledge of Peer Review Committee members. It is CIHR’s policy to only review a committee decision in a situation where there would be a procedural error during the peer review process.

Date modified: