Design Discussion Document - Proposed Changes to CIHR's Open Suite of Programs and Enhancements to the Peer Review Process

March 2012

Table of Contents

• Message from CIHR's Science Council
• Executive Summary
• 1. Purpose of this Document
• 2. Introduction
• 3. Governing Council's Vision for a New Open Suite of Programs
• 4. Addressing Current Challenges through a New Program Design and Peer Review Processes
• 5. A New Open Suite of Programs
• 6. Transition
• 7. Conclusion
• References Cited
• Annex
• Footnotes

Message from CIHR's Science Council

CIHR has initiated a process to design a new Open Suite of Programs and peer review system that ensures the long-term sustainability of CIHR's contribution to the Canadian health research enterprise, removes barriers, and enables researchers from all pillars to improve CIHR's ability to deliver on its mandate. Science Council sees these reforms as an opportunity to take a holistic approach to designing a new Open Suite of Programs and peer review process, and to address multiple challenges at once.

In December 2010, Science Council established an executive Task Force, with representatives from all pillars of health research, to oversee the design and implementation of the changes. The reform process, thus far, has been completed in consultation with targeted members of CIHR's researcher and stakeholder communities. CIHR would like to thank all those who have helped guide the new design and discussion document to date. From the beginning, there has been an appetite from the broader researcher and stakeholder communities to be more engaged in this process. However, had broad consultations begun too early, those consultations would have lacked the design details needed to produce effective conversations. Conversely, by developing this discussion document, some may feel that CIHR is presenting a finished product. With this Design Discussion Document, CIHR's objective is to strike a balance between design detail and design flexibility.

This document represents CIHR's current thinking on the new design. It defines a set of funding mechanisms for the new Open Suite of Programs and a set of design parameters that are currently being considered. Work is underway to model various scenarios related to the implementation of these funding schemes, the phasing out of existing program competitions, and the management of ongoing grantees. CIHR is committed to monitoring the rollout of the new schemes as part of the implementation process and applying course corrections, if needed.

CIHR is also committed to ensuring that the transition to a new Open Suite of Programs occurs with minimal disruption, and your input continues to be a valued part of this process. Before further developing elements of design and moving forward with their implementation, CIHR wants to ensure that the changes being contemplated address the concerns that have been raised by its researcher and stakeholder communities.

We invite you to join us in an on-going, active, and productive discussion about the proposed new Open Suite of Programs and peer review enhancements over the coming weeks.

CIHR Science Council

Executive Summary

CIHR has a bold mandate: "to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health-care system." The challenge is to design funding mechanisms, and support a peer review system, that embrace and recognize the diversity of the research community required to meet this mandate across pillars, research areas, methodologies and approaches. These new funding mechanisms and peer review system must also strive to integrate the increasingly common multidisciplinary and researcher-knowledge user collaborations that underlie many of the advances of today and the future.

Feedback from CIHR's stakeholders and observations from the recent International Review Panel have made it clear that our current research funding programs and review models need to be streamlined to reduce program complexity, and to ensure that researchers spend less time writing and applying for multiple grants to support their research.

Our peer review system and processes fail to adequately accommodate research across all of CIHR's pillars, new and evolving areas of research, and paradigm-shifting research. They also fail to ensure that the right expertise is engaged in reviewing the spectrum of grant applications received. At the same time, growing application pressure, and the complexity of many applications, has meant that potential peer reviewers increasingly express their reluctance to volunteer for the heavy workload.

Changes Being Considered to meet the Challenges

Funding Mechanisms: People and Ideas

CIHR is working to develop a streamlined Open Suite of Programs that will address the full spectrum of CIHR's mandate, and alleviate key challenges with the current competition processes and peer review system. To this end, CIHR is considering two separate, but complementary, funding schemes:

• Foundation/Programmatic Research Scheme
• Project Scheme

The Foundation Scheme is about supporting people. It is about providing long-term support to investigators with a demonstrated track record of success. We want to reduce the time they spend writing grant applications, and leave them more time to conduct research. We want to give them the freedom to create, change, and re-direct their efforts, as required. We also want to give them more time to mentor and develop the next generation of researchers. The assessment criteria for this scheme would be based on the caliber of the applicant, the vision articulated for the proposed program of research, and the support provided to the applicant by their Institution. Applications under this scheme would, therefore, be focused on track-record and the overall approach to a series of research endeavours, rather than on project details or methodology.

CIHR recognizes the critical role that new/early career investigators play in creating a sustainable foundation for the Canadian health research enterprise. This is why we are proposing to have a separate stream in the Foundation Scheme for new/early career investigators – to ensure that new/early career investigators are assessed with their peers and are not competing for funding against established investigators.

The Project Scheme is about supporting ideas. It is meant to encourage researchers who want to explore specific ideas across the spectrum of health and health systems research and knowledge translation, to submit proposals for projects with a specific scope and defined timelines. Today, there are both real and perceived barriers in CIHR's programming; barriers that limit success for certain types of ideas. We want to remove these barriers to ensure there are opportunities for all types of health researchers to submit novel ideas. Applications for the Project scheme will be assessed based on the quality and originality of the idea.

Peer Review: The Mechanics

Many of the challenges with the current CIHR peer review processes relate to the limitations of CIHR's current committee structures, and the variability in peer review judgments between individual reviewers and committees. CIHR wishes to harness scientific expertise, and align peers with the objectives of its programs.

We are considering implementing multi-phased competition processes for both schemes. The intent is to focus reviewer attention on specific criteria at different points in the process. This would be supported by structured review to minimize inconsistent and inappropriate application of review criteria, and to improve transparency of the review process. Both multi-phased competition processes and structured review will help manage applicant and reviewer burden by reducing the number of applicants who move on to full application, and by reducing the length of time it will take to review applications at each stage.

We also want to maximize the use of face-to-face committee meetings. Too much of our current committees' time is spent discussing applications everyone agrees should be funded, or applications everyone agrees have fatal flaws. There is, however, always a "grey zone" where reviewer views are varied for a number of different reasons. We believe that the introduction of a two-phase screening process review will allow for early recognition of outstanding applications, will allow for screening of non-competitive applications, and will concentrate face-to-face discussions on applications that fall into the "grey zone".

We also want to improve the way applications are matched to reviewers to ensure that appropriate expertise is assigned to each application. This will help to avoid having to "force fit" applications into the standing committee structure. This will be aided by the establishment of a College of Reviewers that will facilitate access to appropriate expertise and provide the framework for mechanisms to recruit, train and reward reviewers for specific roles.

Modeling for Transition

CIHR's Transition Plan for phasing in the changes is still in its early stages; and, it is difficult to go into details until the design is finalized. However, we are committed to ensuring that the transition occurs with minimal disruption to researchers; and therefore, are committed to proceeding in a stepwise, incremental fashion. No researcher will have to relinquish his or her grants until the end of their term, and mechanisms will be set in place to allow easy transition between old and new systems. We also plan to develop a robust monitoring and evaluation plan to ensure that the new process generates improvements, when compared to the old.

The target is to announce the final design of the Open Suite of Programs and Peer Review Processes in the late spring of 2012 and launch the first competition of the new Open Suite in the spring of 2013. This would allow a minimum of one year from the announcement of changes to the first competition launch, and would provide sufficient time for the researcher community to prepare.

The Consultation Process

Excellence in research will always rest on the shoulders of individual inspiration, curiosity, and drive. CIHR needs to have an open program committed to investigator-initiated research to leverage this excellence. The proposed design for CIHR's new Open Suite of Programs and Peer Review is driven by recognition of this need.

The design is still evolving. This discussion document has been released to outline major design elements of the planned reforms. Before moving forward with the proposed reforms, it is important to ensure that the changes being contemplated effectively address the multiple issues and challenges that have been identified by researchers, stakeholders and CIHR.

Your feedback is important and will be considered, analyzed and used to inform design and implementation decisions going forward. We encourage you to share your perspectives to help us refine the new design, and transform the way health research is funded, by engaging us through our web-enabled discussion forum, or by submitting your comments/questions to: support-soutien@cihr-irsc.gc.ca.

1. Purpose of this Document

This document is intended to introduce CIHR's proposed design for its new suite of open research funding mechanisms and peer review processes. It is based on numerous discussions that have taken place with researchers, university delegates, peer review committee chairs and scientific officers, institutions, and CIHR partners about some of the challenges that currently exist with CIHR's Open Programs and with the peer review system. One of the messages received from all of CIHR's stakeholders was the importance of involving them in discussions early (before the design is finalized) so that they can:

1. Help inform the design,
2. Help inform the successful implementation of the changes, and
3. Make necessary preparations for the anticipated changes.

The new design for CIHR's Open Suite of Programs is still evolving. As CIHR continues to work on the design, CIHR would like to continue the dialogue with its researcher and stakeholder communities, which is the purpose of this document. This design discussion document provides an overview of the design elements that are being considered in CIHR's new Open Suite of Programs and peer review system.

As you read this document, consider the following questions:

• What are the strengths of the design that is being considered?
• What are the gaps in this design that CIHR should address to ensure a successful implementation?
• What challenges do you anticipate as a researcher/peer reviewer in adopting these changes?

Structure of the Document

Section 2 outlines CIHR's mandate and funding strategies, and provides an overview of CIHR's existing investigator-initiated funding mechanisms, collectively known as CIHR's Open Suite of Programs.

Section 3 describes the CIHR Governing Council's vision to ensure the long-term sustainability of CIHR's contributions to the health research enterprise.

Section 4 provides an overview of the current challenges with the existing Open Suite of Programs, and the design elements that are being considered to address these challenges.

Section 5 provides an overview of CIHR's proposed design of the new Open Suite of Programs, which includes two separate, but complementary funding schemes: a Foundation/Programmatic Research Scheme, and a Project Scheme. Details about these funding schemes, and the current thinking on the mechanics behind their respective competition processes, are provided.

Section 6 outlines some of the implications for transitioning from the existing Open Suite of Programs to the new Open Suite of Programs, and highlights some of the considerations CIHR must address to ensure a successful implementation. CIHR has made a commitment to give researcher and stakeholder communities ample time to adjust to any planned changes.

Section 7 outlines how you can provide CIHR with your feedback. As the purpose of this design discussion document is to open a dialogue with researcher and stakeholder communities, your feedback will be considered, analyzed, and used to inform the design moving forward.

Throughout the last year, Task Force has endeavored to rely, wherever possible, on objective evidence from the literature to develop the proposed changes. An examination of various journal articles, expert opinions, existing practices, and technical reports was undertaken to ensure the new design was informed by evidence. In reviewing what evidence is available, CIHR found that there is not a large base to draw upon in this area [1].

2. Introduction

Background

The Canadian Institutes of Health Research (CIHR) was established in 2000 with a mandate

"to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system".

The organization promotes a solutions-focused, multidisciplinary, and collaborative approach to health research through a unique structure that brings together researchers from across disciplinary and geographic boundaries through its thirteen Institutes. The full breadth of CIHR's mandate can be better appreciated as written in the CIHR Act.

Like most similar agencies around the world, to deliver on its mandate, CIHR has two fundamental strategies for funding research and its translation across the full spectrum of health: an investigator-initiated strategy, sometimes called a bottom-up strategy, that is open to all areas of health research, and a directed or top-down strategy that is targeted to address specific needs and gaps in health research and knowledge translation (Figure 1).

CIHR's top-down and bottom-up strategies and their objectives

Figure 1 long description

The top-down strategy accounts for approximately 30% of CIHR's annual expenditures, and is supported by a set of targeted programs and initiatives that address gaps in specific research areas and research communities, such as:

• Roadmap Signature Initiatives, which include the Canadian Epigenetics, Environment and Health Research Consortium, Personalized Medicine, Community-based Primary Health Care, Inflammation in Chronic Disease, Evidence-Informed Healthcare Renewal, and the International Collaborative Research Strategy for Alzheimer's Disease;
• Institute Strategic Initiatives, such as the Palliative and End-of-Life Care Initiative, and the Maternal Health Initiative; and
• Large/Pan-Institute Strategic Initiatives, such as the Regenerative Medicine and Nanomedicine Initiative; the HIV/AIDS Research Initiative; and the Drug Safety and Effectiveness Network Initiative.

These programs leverage existing strengths. CIHR has made significant strides in developing and implementing a strategic investment process to reduce the number of top-down strategic initiatives and to design them to achieve maximum impact. For more information on CIHR's new strategic investment planning process and the Roadmap Signature Initiatives, please visit Signature Initiatives.

The investigator-initiated, or bottom-up, strategy is supported by a suite of programs designed to capture excellence in research and knowledge translation in Canada. This suite is known as CIHR's Open Suite of Programs.

The Open Suite of Programs accounts for approximately 70% of CIHR's annual expenditures. It includes the Open Operating Grants Program, CIHR's largest funding mechanism, as well as a number of smaller funding mechanisms such as:

• Partnerships for Health System Improvement Program;
• Knowledge Synthesis Program;
• Knowledge to Action Program;
• Proof-of-Principle Program;
• Industry-Partnered Collaborative Research Program; and
• New Investigator's Program.

Also included in CIHR's current Open Suite of Programs are a set of direct training programs, which are not included in the re-design:

• Frederick Banting and Charles Best Canada Graduate Scholarships - Master's Award Program;
• Frederick Banting and Charles Best Canada Graduate Scholarships - Doctoral Awards Program;
• Doctoral Foreign Study Award Program;
• Vanier Canada Graduate Scholarships Program;
• CIHR Fellowships Program; and
• Banting Postdoctoral Fellowships Program.

In 2010-11, CIHR supported approximately 3,000 unique Nominated Principal Investigators through its Open Suite of Programs. As shown in Figure 2 below, the average total amount of in-year funding held by all unique Nominated Principal Investigators was approximately $162K per year (all grants), and the average value for an individual grant was approximately $123K per year.

Amount of in-year Funding from the Open Suite of Programs held by Nominated Principal Investigators in 2010-11

Figure 2 long description

Figure 2: The amount of in-year funding (grants) awarded through the Open Suite of Programs held by unique Nominated Principal Investigators in 2010-11. Note there is only one Nominated Principal Investigator per grant awarded, and that in-year funding does not include fellowships. CIHR defines a Nominated Principal investigator as a funded Nominated Principal Applicant. The definition of a Nominated Principal Applicant can be found in CIHR's Grants and Awards Guide.

CIHR is also accountable for its contributions to Tri-Council Programs, which have not historically been included in the 30% top-down/70% bottom-up funding distribution and include programs such as:

• Canada Research Chairs Program;
• Canada Excellence Research Chairs Program;
• Centres of Excellence for Commercialization and Research; and
• The suite of Networks of Centres of Excellence Programs.

The focus of this document will be on the proposed changes to the Open Suite of Programs – the bottom-up strategy. Note, however, that CIHR's direct training programs are not changing, and will continue to be a part of CIHR's open research funding strategy to support a sustainable pipeline of talented new health researchers into the health research enterprise.

3. Governing Council's Vision for a New Open Suite of Programs

CIHR was created in 2000 to deliver on a broad mandate that encompasses the full spectrum of health research. The 2011 CIHR International Review Panel recognized both the breadth and depth of this mandate, and the advantages and challenges that this brings to the Canadian research enterprise and its researchers. In this context, CIHR requires programs and a peer review system that are capable of identifying and supporting research excellence across the entirety of its mandate. This includes having an Open Suite of Programs capable of developing and supporting a well-trained base of investigators with the skills and expertise essential to designing and conducting innovative and diverse research and knowledge translation activities aimed at improving health. It also includes an expert peer review system that is well-managed, fair, and transparent in the selection and support of the most innovative and cutting-edge research proposals.

Health research plays an important role, not only in improving the health outcomes of Canadians and people around the world, but also in contributing to the overall societal and economic prosperity of Canada [2]. The framework for Canada's health research enterprise in support of this goal is complex. The major sources of funding for health research in Canada include a mix of federal funders, provincial funders, industry, academia, not-for-profit organizations and foreign collaborators/investments [3]. Although each funds health research in the context of their own mandates, there exist complex collaborative relationships between these funders that support shared interests and a common pool of researchers.

As a major federal funder of health research in the country, CIHR must take appropriate action to ensure the long-term sustainability of its contributions to the health research enterprise. It is therefore important that CIHR's new Open Suite of Programs is designed to be flexible and nimble enough to support all types of health research across all domains. It must also be designed in such a way that it facilitates engagement and coordination with other health research funders in the complex funding landscape. The current Open Suite of Programs has been highly successful in generating outcomes and impacts that have improved health and the healthcare system, however, we believe that CIHR can, and must, do more to sustainably advance CIHR's mandate and maintain Canada's competitiveness in today's knowledge economy. With this in mind, Governing Council made reforming the Open Suite of Programs and peer review system a priority in CIHR's second five-year strategic plan, Health Research Roadmap: Creating innovative research for better health and health care.

Funding agencies around the world are being challenged to keep pace with a rapidly advancing research frontier. The current Open Suite of Programs presents both real and perceived barriers (including cultural, structural, and mechanistic barriers) to certain types of research and researchers. As health research is increasingly becoming both a multidisciplinary enterprise and a team pursuit, the existing Open Suite of Programs is not well suited to capture and support the innovative and ground-breaking research that is emerging at the borders of the diverse fields relevant to CIHR.

Around the world, two main funding approaches have been found to be successful in enabling innovation and supporting research excellence:

• Programmatic funding supports a broad program of research over a number of years. This approach recognizes that previous success is one of the best predictors of future success and is considered the best way to support some types of research and researchers. It is capable of achieving high-quality, high-impact results and provides researchers with the flexibility to pursue innovative research avenues.
• Project-based funding supports a defined piece of research with a beginning, middle, and end point. This model is focused on identifying the best possible ideas and is well positioned to support essential incremental research projects, innovative and original research projects, as well as pilot and/or potentially high-risk research projects.

These funding approaches embody the two main philosophies for conducting research that investigators typically apply to their own work (i.e., conducting a program of research; conducting a research project). CIHR believes that both approaches are required to successfully achieve CIHR's mandate, drive innovation, and ensure the long-term sustainably of CIHR's contribution to the Canadian health research enterprise. No single approach has been proven to be more successful than the other.

The research enterprise is not static and sustainability requires renewal. CIHR is committed to ensuring the development and integration of new talent into the Open Suite of Programs through a dedicated scheme of direct training support and mechanisms to enable new or emerging investigators to access either project-based or program-based funding.

Governing Council has asked Science Council to take stock of the lessons learned, look at what others have tried, and push the organization and researcher and stakeholder communities to "be bold" as they design a new Open Suite of Programs and peer review processes.

4. Addressing Current Challenges through a New Program Design and Peer Review Processes

CIHR recognizes the importance of investigator-initiated research, and its role in sustaining the Canadian health research enterprise. In striving to support its vision for enabling a coordinated, solutions-focused, multidisciplinary health research enterprise, CIHR has rapidly expanded its activities to provide increasing support for health research.

Consultations with CIHR's health researcher and stakeholder communities have shown that there are challenges with mechanics behind this vision, specifically with CIHR's competition and peer review processes, including its slate of committees. These challenges are consistent with observations by the International Review Panel and by CIHR's own internal assessments. Some of the most frequently cited challenges are summarized in the Figure 3 below:

List of common challenges as identified by various CIHR stakeholders

Figure 3 long description

The time has come to re-design the existing system. For CIHR's competition and peer review processes, this means building a system that is flexible enough to both adapt to the changing landscape of health research (which recognizes that there are different approaches to conducting health research), and that can accommodate future demands for funding from all areas of health research. It also means embracing the opportunity to support investigator-initiated research that acknowledges/enables/integrates the perspectives that citizens, communities, knowledge usersⁱ, and other non-academics may bring to the research process.

Peer review at CIHR has two purposes: (1) to help determine what CIHR should fund, and (2) to improve science. Applicants and peer reviewers will always strive to provide/request more information to support decision-making, which consequently contributes to increased applicant and reviewer burden. CIHR is challenged with figuring out what is the minimum amount of information needed to allow peer reviewers to deliver sound judgments and provide quality feedback. With the increasing use of electronically-enabled, interactive communications, there is an opportunity to bring the right expertise together in the review of individual applications, and to mitigate against the desire to submit/request extraneous information by using a phased review approach.

As we move forward with the design of the new Open Suite of Programs, CIHR is looking to incorporate design elements that would address some of the strains in the system. This section provides a summary of how the current thinking aims to address each of the challenges (Figure 3) in support of a sustainable, adaptable, transparent, and accessible health research funding system.

Program Accessibility and Complexity

The Open Operating Grants Program, one of the Medical Research Council's legacy programs, remains CIHR's largest funding mechanism. This funding mechanism was successful in meeting the needs of its intended research communities (Pillar 1 - Biomedical), and contributed to the successful results that underpin Canada's reputation for research excellence. Most importantly, these results have benefitted the health of Canadians.

As health research evolves, the program has tried to meet the needs of a much broader mix and a greater number of researchers within CIHR's mandate. While changes have been made to improve the Open Operating Grant Program's administration and performance, the core criteria and application requirements of the program have not significantly changed. This has created difficulties for researchers in Pillars 2 (Clinical), 3 (Health System and Services) and 4 (Social, Cultural, Environmental, and Population Health) and rapidly evolving and multidisciplinary areas in all pillars, including Pillar 1 (Biomedical), to access grants, and has resulted in significant gaps in the delivery of CIHR's mandate.

To address these gaps, CIHR created a number of small, purpose-built funding mechanisms that (as CIHR's 2011 International Review Panel notes) have resulted in an increasingly complex mixture of programming over time [4]. Researchers are challenged with trying to fit their research projects into existing (and sometimes inconsistent) program criteria, and are sometimes required to apply to multiple programs to support a single program of research.

A more rational and focused approach is needed to fulfill CIHR's full mandate and sustainably fund excellence in all areas of health research. A review of international research granting agencies has shown that comparable organizations have experienced the same challenges as CIHR, and have taken action to improve their funding schemes.

Although opinions are divided as to whether either funding model is more effective than the other, there are two main funding approaches that have been found to be successful in supporting research excellence:

• Programmatic funding supports a broad program of research over a number of years, usually at a fixed rate, but sometimes varying in relation to the type of research and the costs involved. Typically awarded to established investigators, it is considered the best way to support some types of research and researchers, and one of the best models to achieve high-quality, high-impact results. Several recognized funding agencies, such as the Wellcome Trust and the Howard Hughes Medical Institute, have successfully implemented programmatic funding schemes with positive results [5, 6].
• Project-based funding supports a defined piece of research with a beginning, middle, and end point. Typically awarded to applicants with the best ideas, this model is well positioned to support incremental research projects, innovative and original research projects, as well as early stage and/or potentially high-risk research projects. Project-based funding has been successfully implemented by the National Institutes of Health (e.g., NIH Research Project Grant Program – R01), and the Gates Foundation (e.g., Grand Challenges in Global Health competition) [7, 8].

It should be noted that through the years, although the Open Operating Grants Program had been set up as a project-based mechanism, researchers have been establishing programs of research using this mechanism by coordinating a series of individual Project grant proposals and renewals.

The programmatic research and project-based funding mechanisms are not intended to be viewed as part of a continuum. They are two distinct funding streams that support two different approaches to funding health research: through programs of research that build towards long-term health research goals; and, through projects that answer specific health research questions and have a defined end point. CIHR believes that both types of funding schemes are required to meet its needs, and will provide applicants with the flexibility to choose the most appropriate type of mechanism for the type of research being conducted and their personal research style, irrespective of research domain and/or pillar. The intent is to merge as many of the existing, small, purpose-built funding mechanisms as possible, including the Open Operating Grants Program, into these two Schemes.

Applicant Burden/"Churn"

CIHR is committed to building a sustainable foundation of Canadian health researchers. Since CIHR's inception, there has been an increase in support for health research aimed at building Canada's health research capacity. However, there has been an increasing trend for CIHR-funded researchers to submit multiple applications to open grant competitions to support a single program of research. With so much time spent writing and applying for multiple grants, researchers have complained that there is little time left to conduct research. Writing grant applications can be extremely costly to researchers and institutions. A study conducted in Australia last year estimated the total cost of applications to be over $17,000 (Australian dollars) per application submitted [9]. Although CIHR has not conducted a detailed study, it is estimated that the costs in Canada are similar.

An increase in applicant churn over the past five years led CIHR's 2011 International Review Panel to recommend that "CIHR [should consider] awarding larger grants with longer terms for the leading investigators nationally" [4]. Stakeholders from some of Canada's leading health research institutions have also recommended that CIHR establish Foundation/Programmatic Research grants that are based on track record, and give the holder leeway to pursue a research path based on their best judgment.

The longer duration and larger value of grants awarded through the Foundation/Programmatic Research Scheme is intended to reduce applicant burden and give greater flexibility to investigators. Successful applicants would not need to write multiple research proposals to competitively fund their research programs, nor would applicants need to apply for funding as frequently.

To further reduce applicant burden, applications will need to remain focused on the most relevant information needed to support decision-making. A multi-phased competition process with short Stage 1 applications are being considered to reduce the amount of work required to complete an application. This would free additional time for applicants to conduct research or engage in other activities.

Application Processes and/or Attributes Do Not Capture the Correct Information

Over the years, CIHR has expanded its Open Suite of Programs to provide additional support for integrated knowledge translationⁱⁱ, training, new investigators, and emerging areas of health research. This proliferation of targeted funding mechanisms has resulted in CIHR adding new information to existing application forms and/or processes. However, as new information was added to capture specific researchers and their programs/projects of research, information that was not relevant was not removed.

In managing this proliferation of funding mechanisms, the peer review criteria and the information captured in the applications were never comprehensively and systematically matched. This resulted in applicants producing increasingly lengthy applications to meet the application requirements. There is now an opportunity to refine the review criteria and structure applications to capture the information needed to conduct an effective review of all types of health research.

In designing the new Open Suite of Programs, CIHR is considering a multi-phased application that would capture different types of information at different stages of the review. This would include only information that is relevant to the proposal received and only information that is relevant for a peer reviewer to make an informed judgment at a given stage of the review.

Insufficient Support for New/Early Career Investigators

CIHR recognizes the role that new/early career investigatorsⁱⁱⁱ play in creating a sustainable foundation for the Canadian health research enterprise. Specific funding mechanisms targeted to new/early career investigators have made it easier to obtain first-time funding. In general, new/early career investigators have provided CIHR with positive feedback regarding increased access to funding. However, many have expressed concern regarding success rates related to the renewal of funding.

Both project-based and programmatic funding schemes are being designed to include features to support new/early career investigators. CIHR's Project Scheme is considering focusing the first stage of review on the quality of the idea with limited information about the track record of the applicant. This would remove some biases/barriers (real or perceived) for new/early career investigators and will be important for CIHR to monitor. For the Foundation/Programmatic Research Scheme, CIHR is considering a specific stream to support new/early career investigators to ensure that these researchers have an opportunity to build promising programs of research.

Researcher and Knowledge User Collaborations Are Not Fully Valued

Knowledge translation is a prominent and innovative feature of the CIHR mandate. It has the potential to significantly increase and accelerate the benefits flowing to Canadians from their investments in health research. It also holds the potential to establish Canada as an innovative and authoritative contributor to health-related knowledge translation.

Over the years, CIHR has launched innovative funding mechanisms that signaled its commitment to knowledge translation and began to build capacity in this area.

Integrating knowledge translation has brought researchers and knowledge users^iv together to shape the research process, and has been found to produce research findings that are more likely to be relevant to, and used by, knowledge users. However, CIHR's Open Operating Grants Program was not designed to actively incentivize partnerships and collaborations with knowledge users/decision makers. Without reinforcing appropriate links between discovery-based research and its application, CIHR will not be able to achieve its mandate to contribute to "improved health for Canadians, more effective health services and products and a strengthened Canadian health care system".

As part of the new design of the Open Suite of Programs, both the Foundation/Programmatic Research and Project funding schemes would encourage and expect collaboration with relevant partners (where appropriate). This would mainstream integrated knowledge translation^v into CIHR programs. In doing so, CIHR would be able to ensure that the principles of integrated knowledge translation funding mechanisms (such as the Partnerships for Health System Improvement program) are embedded in the new program design for the Open Suite of Programs, and that CIHR continues to support researchers and knowledge users collaborating on investigator-driven health research.

Lack of Expertise Availability

The availability of expertise is a critical component of CIHR's peer review process. In 2010-11, CIHR recruited approximately 2,000 experts from across a variety of health research disciplines to review more than 6,500 grant applications for competitions in the Open Suite of Programs. As the nature and diversity of health research evolves, there is a growing need for CIHR to recruit peer reviewers from a broader base of expertise to ensure all aspects and future impacts of health research are considered. Some researchers have expressed concerns that CIHR's current pool of experts may not possess the disciplinary expertise to review all types of applications. Researchers from multidisciplinary, emerging, and established fields have expressed difficulty in identifying the most appropriate CIHR peer review committees to review their research. As well, it is becoming increasingly difficult for CIHR to fit certain applications into discipline-based review committee mandates.

Currently, CIHR populates standing peer review committees with experts based on the subject area and applications that have historically been received; additional members are added as required to accommodate unexpected applications. The creation of a College of Reviewers will serve as a framework for organizing and managing groups of reviewers, and aims to provide a broader base and access to the appropriate expertise to review applications from all areas of health research. The College model supports the proactive recruitment of a variety of expertise, and CIHR is considering incentives for those who choose to join the College. Meaningful, non-monetary reviewer incentives and recognition approaches are currently being considered to attract and retain the breadth and depth of expertise required to populate the College.

In some cases, the correct expertise to review an individual application may actually reside with individual members from multiple CIHR peer review committees. Application-focused review is guided by the principle of assigning the right reviewers to the right application. This model avoids "force fitting" applications into standing committee structures by individually aligning and assigning reviewers based on a list of common descriptors in a reviewer's curriculum vitae and the application package. Application-focused review would also allow for the use of experts with varying degrees of specialty to: (1) provide an evaluation of not only the feasibility of a research proposal, but also of the vision, research strategies, and broader impact; and/or (2) provide a lay perspective, where appropriate. By assigning the right mix of expertise to provide a robust review of the proposed program of research/research project, CIHR would be able to better ensure that peer review decisions are informed by the right set of expertise and are of high quality. In addition, it would help ensure that applicants from all areas of health research are on equal footing when it comes to the quality of peer review they receive.

Recent advances in technology and social media provide an opportunity to use technology to support enhancements to the peer review process, including the facilitation of application-focused review. CIHR is looking to bring reviewers together in a virtual space supported by mechanisms that facilitate Internet-assisted discussions ("virtual peer review") similar to what the National Institute of Health's Centre for Scientific Review is currently implementing [10]. Reviews, for the most part, would be individual. However, reviewers would be given an opportunity to discuss, justify, and exchange perspectives to inform the substance of their final individual reviews of applications. Using this technology, CIHR could gain broader access to the required expertise (including international experts) and bring together multiple perspectives to inform peer review decisions.

CIHR has heard concerns that reducing the use of face-to-face committee meetings could negatively impact the consistency and quality of reviews and make it more difficult for individual reviewers to calibrate themselves to the overall committee. However, there is evidence that individually-conducted reviews are effective at identifying the top applications, which are most likely to be funded.

A recent report from CIHR's Evaluation Unit on peer review in CIHR's Open Operating Grants Program showed that individual review is consistently able to select top applications approximately 75% of the time when compared with committee-centered (face-to-face) reviews. Similarly, a study conducted by Obrecht, Tibelius, and D'Aloisio also found that individual review rankings are good predictors of placement at the summary stage [11] and Vener et. al. have shown that individually conducted reviews can reliably identify both competitive and non-competitive applications [12].

It is CIHR's intention to make more judicious use of face-to-face committee meetings as a mechanism to integrate the results of remote reviews and determine the final recommendation for funding. The primary focus of face-to-face committee reviews would be on the applications that require further discussion, particularly regarding rankings near the funding cut-off (i.e., the grey zone).

Inconsistency of Reviews

The quality and consistency of peer review decisions are critical components of CIHR's selection process. Cole, Cole, and Simon found a 25% level of disagreement over which proposals should be "funded" when they were assessed by two different and independent peer review committees [13].

Results from a survey conducted for CIHR's 2011 International Review on Stakeholder Satisfaction in Peer Review also show that, while CIHR's peer reviewers find the process to be fair and efficient, applicants and institutional stakeholders believe there is room for improvement.

Stakeholder Satisfaction - Peer Review
(Percent of respondents who provided an opinion)

Figure 4 long description

Figure 4: Survey conducted by Ipsos Reid (2010) showing CIHR stakeholder satisfaction in Peer Review. ("Satisfied" category includes very satisfied, somewhat satisfied and neutral; "Dissatisfied" category includes somewhat dissatisfied and very dissatisfied).

As shown in Figure 4 above, 79% of peer reviewers are satisfied with the efficiency of the peer review process, while 70% of peer reviewers are satisfied with the overall fairness of the process. However, nearly 48% of "institutional stakeholders" are dissatisfied with the consistency of CIHR's peer review judgments, while 44% and 58% of "applicants and grantees" are dissatisfied with both the quality and consistency of CIHR's peer review judgments, respectively. As Mayo and colleagues point out:

Despite science's pre-occupation with accurate measurement, there is no precise method measuring the quality of proposals – "good enough" for funding is typically left to the subjective opinion of a very few number of reviewers. [14]

The quality, fairness, and consistency of peer review judgments are integral to maintaining a transparent and accountable peer review system. As mentioned above, CIHR will establish a College of Reviewers. As part of the College's membership, reviewers will participate in a training program to provide them with the knowledge and resources necessary to conduct consistent and reliable reviews. Furthermore, CIHR aims to improve the reliability and consistency of reviews by ensuring that evaluation criteria are consistently applied by all reviewers. A cross-disciplinary study on the reliability of peer review by Cicchetti showed that formal training for peer reviewers would enable them to use specific evaluation criteria more consistently, and would contribute to standard, reliable, and valid recommendations [15]. Additionally, Obrecht, Tibelius, and D'Aloisio note that:

"It is reasonable to predict that a review in which criteria are clearly defined, benchmarked and individually rated will always be fairer than a review in which they are not. A structured review ensures that all applicants are assessed against the same criteria" [11]

To this end, CIHR is considering the use of structured review criteria to provide clear instructions to reviewers and address this challenge.

Conservative Nature of Peer Review

CIHR's peer review process plays a critical role in selecting the diverse array of health research CIHR supports to achieve its mandate.

Based on the nature and the diversity of applications CIHR receives, CIHR has witnessed an increase in new, emerging areas of health research over the years, including work across multiple research disciplines. Feedback from researcher and stakeholder communities suggests that current committee structures, which implement equal funding cut-offs for established and relatively unchanging mandates, favour established approaches. In an environment where only a small proportion of applications are funded, there is less incentive and comfort to accept riskier, unproven areas of research. These two factors lead to a conservative peer review system.

As part of the design considered for the new Open Suite of Programs, CIHR will be building in features for project-based and programmatic funding schemes to address this challenge. The Project Scheme attempts to remove track-record bias from Stage 1 of the review process. This has been successfully implemented by the National Institute of Health and the Gates Foundation [7, 8]. One of the benefits of the Foundation/Programmatic Research Scheme is sufficient, stable, and long-term funding. This would allow researchers to pursue high-risk and innovative ideas without the pressure of imminent grant renewal.

High Peer Reviewer Workload

As mentioned previously, CIHR relies on its peer reviewers to identify exemplary projects and individuals that merit funding. Support from this community of experts is critical. Without it, CIHR would not have the necessary financial and human resources to fund the same amount of quality health research.

The increase in number of applications received, including re-submissions, has increased the number of reviewers CIHR needs to deliver on its mandate. The majority of reviewers are expected to travel to Ottawa to review more than 6,500 increasingly long applications within a short period of time, resulting in reviewer fatigue [16].

In designing the new Open Suite of Programs, CIHR aims to implement design elements that will reduce the overall time a reviewer spends reviewing, discussing, and providing feedback on an application. To achieve this, CIHR is considering a number of design elements:

1. A multi-phased competition process that involves a two-stage screening process prior to face-to-face review. Multi-phased competition processes have been successfully implemented in research granting agencies in Australia and the United Kingdom, as well as in large research organizations such as the European Research Council [5, 17].
2. Structured review criteria would provide peer reviewers with clearly defined review criteria and relevant application information in order to evaluate success. This would help reduce the amount of time reviewers spend analyzing an application package, and providing free form feedback to applicants [11, 15].
3. Conducting screening reviews and conversations in a virtual space (internet-assisted discussions) would help reduce reviewer burden by alleviating the need for reviewers to travel to Ottawa, freeing time to conduct research or engage in other activities. This model would also facilitate international expert review by supporting cost-effective access to international research leaders.

Each design element highlighted has the potential to address concerns raised by applicants and peer reviewers (summarized in Figure 5 below).

Figure 5 long description

Figure 5: This figure maps examples of how the proposed design elements address multiple challenges in the current competition and peer review systems.

5. A New Open Suite of Programs

CIHR's new Open Suite of Programs is designed to contribute to a sustainable foundation of excellence for the Canadian health research enterprise by supporting world-class researchers in the conduct of research and its translation across the full spectrum of health research.

To this end, CIHR is considering a suite of two separate, yet complementary grants funding schemes:

1. A Foundation/Programmatic Research Scheme, which will provide long-term support both to established investigators with demonstrated track records of success, and new/early career investigators with excellent training and early-career productivity, to pursue innovative, high impact programs of health research (and enable integrated knowledge translation). Flexible, long-term funding will also provide top researchers with the opportunity to pursue novel and/or emergent avenues of health research with a less frequent requirement for grant renewals.
2. A Project Scheme, which will provide support for defined projects with a beginning, a middle, and a definite end point that capture the most original, innovative, and/or impactful ideas across the spectrum of health and health system research and knowledge translation. This scheme is open to both established and promising new/early career researchers.

Both the Foundation/Programmatic Research and Project Schemes will undoubtedly support the training and mentoring of promising trainees, much as the Open Grants Program does today. However, applications to the Foundation/Programmatic Research Scheme will be required to include a comprehensive mentoring and training plan that will be evaluated in peer review.

CIHR currently funds some 14,000 trainees and researchers, which include a mix of nominated principal investigators, principal investigators, and co-applicants. Of the approximately 9,000 researchers funded through the current Open Suite of Programs, approximately 3,000 are unique nominated principal investigators. In designing the new Schemes, Science Council emphasized the importance of continuing to support a similar number of principal investigators.

CIHR is currently in the process of modeling a variety of financial scenarios to support the implementation of the new schemes. Given a finite budget, there will be trade-offs between the number of grants awarded in each scheme and the average grant size and duration^vi. With this in mind, CIHR is considering allocating 45% of its open grants funding budget to the Foundation/Programmatic Research Scheme and 55% of its open grants funding budget to the Project Scheme. Specific details about the proposed relative size and duration of grants are outlined for each Scheme under the sub-heading "Grant Value and Duration."

Foundation/Programmatic Research Scheme

The Foundation/Programmatic Scheme is designed to provide long-term funding to researchers who are making substantial contributions to the Canadian health research enterprise (or in the case of new/early career investigators, have the clear potential to do so). Based on current modeling and an assessment of current open research funding profiles, CIHR anticipates that the Foundation/Programmatic Research Scheme will support a mix of competitive small, medium, and large programs of research (commensurate with the varying costs of research in different domains). The Foundation/Programmatic Research Scheme will also include a separate stream for new/early career investigators.

This Scheme is to:

• Support world-class researchers in various stages of their careers to conduct programs of research and translation of findings that contribute to improved health for Canadians, more effective health services and/or products, and a strengthened Canadian healthcare system.
• Support a cadre of talented new/early career investigators^vii to build programs of research and knowledge translation that contribute to improved health for Canadians, more effective health services and/or products, and a strengthened Canadian healthcare system.
• Provide flexible, long-term funding that allows for:
• Evolution of research-related activities, such as discovery, demonstration, validation, application, and dissemination;
• Collaboration with other researchers, engagement of knowledge users, and international linkages;
• Exploration of new or high-risk areas of inquiry; and
• Attraction, training, and mentoring of students, postdoctoral fellows, and other new researchers.

Foundation/Programmatic Research Scheme Design

Eligible Applicants

Eligible applicants would include independent researchers (new or established) with a demonstrable track record of excellence and/or impact in their field of study. While not required for programs in all areas, CIHR would encourage researchers to partner and collaborate with knowledge users^viii and other partners^ix, as appropriate, to support the successful execution of a program of research. The level of expected partner engagement and contribution would depend on the value added by the expected contribution, whether in-kind or financial, with particular emphasis on the need for partner collaboration to successfully pursue the research and knowledge translation program proposed. In this model, principal Foundation/Programmatic Research grant holders (i.e., Nominated Principal Investigators and other lead Principal Investigators peer reviewed in the competition) would not be able to apply for a new Project grant.

CIHR recognizes that research is increasingly being conducted by groups of researchers, and has heard concerns over the eligibility of applications with multiple leads. CIHR is supportive of such research activities and is currently determining how best to assess these types of applications within the context of the proposed competition process for the Foundation/Programmatic Research Scheme.

Examples of possible programs of research that could be funded through the Foundation/Programmatic Research Scheme are listed below (Figure 6). This is meant to be illustrative of the types of programs that could be funded across the pillars of research and is by no means a comprehensive list.

Examples of programs of research that may be funded through CIHR's new Foundation/Programmatic Research Scheme

Figure 6 long description

Grant Value and Duration

The value and duration of a Foundation/Programmatic Research grant would usually be greater than the typical value and duration of individual grants currently funded at CIHR. Foundation/Programmatic Research grant budgets would be commensurate with scientific need and, for any successful applicant, would not usually be less than the sum of grants currently held by that investigator in the Open Suite of Programs. Based on the current modeling, it is estimated that most Foundation/Programmatic Research grants would be for 7 years for established investigators and 5 years for new/early career investigators with a targeted average of approximately $300K/year. Discussions are underway to determine what proportion of funding should be directed towards the new/early career investigator stream.

A mid-term review process to evaluate progress is being considered. Budgetary adjustments, if needed, may be considered as part of the review. Foundation/Programmatic Research grants are intended to support the direct costs of research and do not include a salary support component.

Institutional Commitment

Institutions play a critical role in supporting health research and share with CIHR a common interest in supporting research success. Recently, CIHR has heard from researcher and stakeholder communities that there is a need for institutions to provide more support for the conduct of research. Concerns over teaching loads and protected time for research vary within and across academic institutions, as well as across different pillars of research.

To ensure that researchers are able to successfully execute their programs of research, CIHR's new Foundation/Programmatic Research Scheme would require applicants to secure a formal commitment from their institution to provide significant support. This support may include providing the necessary time release, infrastructure, resources, knowledge translation support (e.g., technology transfer), training, salary, and career development support to investigators over the duration of the Foundation/Programmatic Research grant awarded. CIHR recognizes that there are growing demands on the resources available to Institutions, and that there may be significant costs associated with providing these types of support. CIHR will need help from each institution to identify what constitutes a sufficient amount of support for the Foundation/Programmatic Research and Project schemes, and how this support can be integrated into CIHR's funding scheme design.

To mitigate some of the strain on institutions, CIHR is also working to harmonize the requirements for its new Open Suite of Programs and Peer Review Enhancements with those of other granting programs/agencies, such as the Canada Research Chairs program and Canadian Foundation for Innovation. As the design of the new Open Suite of Programs evolves, CIHR will need help from institutions to determine what future impacts the new Open Suite of Programs will have on research institutions, and what can be done to ensure the successful implementation of the new design.

Competition Process: Foundation/Programmatic Research Scheme

It is important to note that this scheme is still in the design phase. The competition process and design elements described in this section are intended to stimulate a discussion about how to best capture scientific excellence and innovation, and how to best address the challenges of applicant and peer reviewer burden. The design elements are described at a high level.

CIHR is considering a multi-phase competition process for the Foundation/Programmatic Research Scheme. The process is summarized in Figure 7 below. Details about the competition timing for the Foundation/Programmatic Research Scheme are still being discussed. Current thinking suggests that there would be one Foundation/Programmatic Research competition per year (in the Fall), and that grant applications would undergo a three-stage assessment.

Competition process for the Foundation/Programmatic Research Scheme

Figure 7 long desciption

Stage 1 would assess the caliber of an applicant's track record based on an applicant's curriculum vitae and an outline of their achievements and research contributions. All foundation/programmatic research applications will be reviewed in the context of an applicant's career stage. This application would be designed to be reviewed in a relatively short period of time (i.e., approximately 30-60 minutes or less).

CIHR also recognizes that in some cases, a program of research is co-led by two or three researchers of equal standing. How to assess the caliber of the applicant's track record when the "applicant" includes more than one leader has not yet been determined, but as this represents a growing trend in research (as it does in other professional fields) it will be essential to tackle this challenge to the review process.

The Stage 1 application would be matched to reviewers with the appropriate expertise (including knowledge users where appropriate), based on common descriptors identified by both the reviewer and applicant, and evaluated using structured peer review criteria. Applications would be sent to approximately 5-8 reviewers, which is an optimal range consistent with the available literature on peer review [9, 14, 18]. Each reviewer would receive between 15-20 applications. At this stage, each reviewer would be asked to conduct their individual reviews remotely, supported by technology to capture the structured feedback. Once a reviewer has completed the reviews of individual applications, he/she would rank the "set" of applications and submit the results electronically. Work is underway to determine an approach to appropriately integrate the individual ranking results.

Applicants would be advised of their results and those who are successful would be invited to proceed to the second stage of the process. Applicant burden and reviewer burden will be actively managed by controlling the number of applicants invited to participate in Stage 2.

Applications at Stage 2 would contain information that would allow reviewers to assess the intent of the proposed program of research, as well as the quality of the environment supporting the proposed program of research. This application would be longer than in Stage 1, is anticipated to be shorter than current CIHR applications, and would be designed to be reviewed in approximately 1-3 hours.

As in Stage 1, the Stage 2 applications would be matched to approximately 5-8 reviewers with the appropriate expertise based on common descriptors identified, and screened using structured peer review criteria. It is expected that each reviewer will receive no more than 10 applications. At this Stage, reviewers would again be asked to conduct their reviews remotely, supported by technology to capture the structured feedback. This stage would be supported by on-line discussions between reviewers. In some cases, reviewers may seek clarification from applicants on aspects of their proposals in order to complete a robust review. Discussions are ongoing to determine the requirements and design for such a mechanism. However, it is anticipated that a coordinator or moderator would be identified to resolve issues with applications and to coordinate queries to applicants, as required.

Through the use of an appropriate range of peer reviewers and structured review criteria, CIHR aims to improve the reliability of peer review recommendations for all types of health research. Some concerns have been raised about integrating different points of view into the peer review ranking process without the reviewers being in the same room together. CIHR is currently assessing various ways to integrate different points of view in a virtual space.

Once the individual reviews are complete and have been informed through electronic discussions with other reviewers, reviewers would rank a set of applications and submit the results electronically.

After Stage 2, recommended applications would undergo a Stage 3 review. This review would be a face-to-face meeting of an inter-disciplinary committee which would integrate the results of the remote review process and determine the final recommendations. The primary focus of this face-to-face review is on the applications that did not reach consensus at the remote review stage, and that require further discussion (i.e., the grey zone). Again, peer reviewer burden will be actively managed by controlling the advance to Stage 3. Based on current modeling, CIHR anticipates that anywhere from 30% to 50% of all Stage 3 applications will be successful.

CIHR has heard concerns regarding the transition to, and the renewal of, Foundation/Programmatic Research grants. At this time, CIHR is considering mechanisms that would provide transitional support to researchers. For those transitioning to the Foundation/Programmatic Research Scheme, CIHR is considering a process to roll-up existing Project grants into the new Foundation/Programmatic Research funding mechanism. For those who are unsuccessful with grant renewal, transitional support may be provided. Details on both transition models are still being discussed.

Project Scheme

The Project Scheme is designed to provide support for original, innovative, and impactful ideas brought forward by researchers and/or knowledge users.

This Scheme is to:

• Support projects initiated by researchers and knowledge users for a specific purpose and period of time, where the results have the potential for original or innovative advances in health knowledge or knowledge translation that contribute to improved health for Canadians, more effective health services and products, and/or a strengthened Canadian healthcare system.

This Scheme aims to improve upon the existing Open Operating Grants Program by consolidating the suite of existing open programs, including their objectives and principles, under a scheme that emphasizes support for the "best ideas."

Project Scheme Design

Eligible Applicants

Eligible applicants may include independent, new, or established researcher(s) and/or knowledge user(s) who have not been awarded a new Foundation/Programmatic Research grant. Applicants could be individuals or teams of collaborators. While not required for all applications in all areas, CIHR would encourage knowledge users^x to collaborate with researchers as part of the project. For some types of projects, partner^xi collaboration or knowledge user involvement may be necessary based on lack of feasibility of execution of the project in their absence. The level of expected partner engagement and contribution would depend on the value added by the expected contribution, whether in-kind or financial, with particular emphasis on the need for partner collaboration to successfully pursue the research and knowledge translation project proposed.

Examples of possible projects that could be funded through the Project Scheme are listed below (Figure 8). Again, this is meant to be illustrative of the types of projects that could be funded across all areas of research and is by no means a comprehensive list.

Examples of research projects that may be funded through CIHR's new Project Scheme

Figure 8 long desciption

Grant Value and Duration

The value of Project grants would typically be less than that of a Foundation/Programmatic Research grant. This is because Foundation/Programmatic Research grants are expected to fund complex programs of research that include a number of interrelated projects over a longer period of time; whereas Project grants are focused on a specific project with a defined start, middle, and end. Project grant budgets would be commensurate with scientific need. Based on the current modeling, it is estimated that most Project grants would be for 3 to 5 years with a target average of approximately $125K/year.

Given the size and scope of projects CIHR has historically funded, CIHR believes that the majority of Project Scheme applications would be adequately supported, including a large number of Randomized Controlled Trials and team-based research projects. CIHR is also developing strategic initiatives, such as the Strategy on Patient-Oriented Research, as avenues to support large-scale projects.

Institutional Commitment

Applicants would need to secure support from their host institutions similar to what is required today in the Open Operating Grants Program. This may include providing access to expertise and resources, as appropriate for the project.

Competition Process: Project Scheme

It is important to note that this scheme is still in the design phase. The competition process and design elements described in this section are intended to stimulate a discussion about how to best capture scientific excellence and innovation, and how to best address the challenges of applicant and peer reviewer burden. The design elements are described at a high level.

CIHR is considering a multi-phase competition process for the Project Scheme. The process is summarized in Figure 9 below. Details about the competition timing for the Project Scheme are still being discussed. Current thinking suggests that there would be two Project competitions per year (one in the Spring and one in the Fall), and that grant applications would undergo a three-stage assessment.

Competition process for the Project Scheme

Figure 9 long description

Similar to submitting a short letter of intent, Stage 1 would assess the quality of an idea and the potential for generating innovative and original research results, methodologies and/or tools. Applicants would be invited to complete a short project proposal (2-3 pages) describing the research idea. The application would be designed to be reviewed in a short period of time (less than 30 minutes). CIHR has heard concerns from peer reviewers that with a short application in Stage 1, application pressure may become unmanageable. Some have suggested that CIHR limit the number of applications received per investigator per competition to mitigate this risk. CIHR is currently looking into potential options to ensure application pressure remains manageable.

As the track record of the applicant would not be assessed at this stage, specific information about the researcher or research team would not be peer-reviewed at this stage. In early discussions about this Scheme, concerns were raised about an applicant's ability to properly contextualize the importance of a research idea as it relates to his/her experience, expertise, and understanding of what it will take to successfully realize the idea. CIHR is currently looking at which components of an applicant's track record are important and/or relevant in assessing the quality of an idea.

The Stage 1 application would be matched to reviewers with the appropriate expertise (which may include knowledge users) based on common descriptors identified by both the reviewer and applicant, and evaluated using structured review criteria. Reviews would be conducted remotely using structured review criteria. Applications would be sent to approximately 5-8 reviewers which is an optimal range consistent with the available literature on peer review [9, 14, 18]. Each reviewer would receive between 25-30 applications, and would be asked to conduct the reviews remotely, supported by technology to capture the structured feedback. Once a reviewer has completed the reviews of individual applications, he/she would rank the "set" of applications and submit the results electronically. Work is underway to determine an approach to appropriately integrate the individual ranking results.

Applicants would be advised of their results and those who are successful would be invited to proceed to the second stage of the process. Applicant burden and reviewer burden will be actively managed by controlling the number of applicants invited to participate in Stage 2.

The Stage 2 application would contain information that would allow CIHR to assess the merit and feasibility of the approach for conducting the proposed research project. The application would be longer than in Stage 1, and would be designed to be reviewed in approximately 2-3 hours.

As in Stage 1, the Stage 2 applications would be matched to approximately 5-8 reviewers with the appropriate expertise based on common descriptors identified, and evaluated using structured peer review criteria. It is expected that no more than 15 applications will be provided to a single reviewer. At this Stage, reviewers would again be asked to conduct their reviews remotely, supported by technology to capture the structured feedback. Feedback would be shared with all other reviewers who are reviewing a particular application. This stage would be supported by on-line discussions with other reviewers. In some cases, reviewers may seek clarification from applicants on aspects of their proposals in order to complete a robust review. Discussions are ongoing to determine the requirements and design for such a mechanism; however it is anticipated that a coordinator or moderator would be identified to resolve issues with applications and to coordinate queries to applicants, as required.

Once the individual reviews are complete and have been informed through electronic discussions with other reviewers, reviewers would rank their set of applications and submit the results electronically. Recommended applications would be submitted to the final assessment stage. At this time, the specific evaluation criteria for each stage of the review process are under development.

After stage 2, applications would undergo a Stage 3 review. This review would be a face-to-face meeting with an inter-disciplinary committee which would integrate the results of the remote reviews and determine the final recommendations. The primary focus of this face-to-face review is on the applications that did not reach consensus at the remote review stage, and that require further discussion (i.e., the grey zone).

Again, peer reviewer burden will be actively managed by controlling the advance to Stage 3. At this time, CIHR anticipates that anywhere from 30% to 50% of all Stage 3 applications will be successful.

Concerns have been raised regarding continuity in the Project Scheme. CIHR recognizes that research can be incremental. As such, the Project Scheme may be a vehicle for funding incremental research, with the results of one defined project being used to inform future projects. Other concerns that have been raised about this Scheme include whether it will really attract and support innovative/novel proposals. CIHR is now looking into what additional elements are required to ensure high-risk/high return projects are supported.

6. Transition

This design discussion document will be released to researcher and stakeholder communities early in 2012. The goal is to gather feedback and make some decisions about the design of the Open Suite of Programs in the spring of 2012.

CIHR is committed to ensuring that the transition to a new Open Suite of Programs occurs with minimal disruption. Although planning such a transition strategy is difficult until the design is finalized, work is underway to model various transition scenarios related to the implementation of the Foundation/Programmatic Research / Project Schemes, the phasing out of existing program competitions, and the management of ongoing grantees. Integrated in the transition planning is the development of a robust monitoring and evaluation system to ensure continuous quality improvement for the new system, to which CIHR is also committed.

Current thinking suggests a gradual phase-in strategy will be used to implement the new design, and that changes will be introduced in small, progressive steps. CIHR does not intend to cease its existing grants competitions until 2013, at the earliest. Applicants and reviewers would be provided with a minimum of one year to prepare, from the time of the announcement of changes to the first competition launch. This means that the first funded researchers under the new schemes would be announced (at the earliest) at some point in 2014-15. Between now and the launch of the new Open Programs, CIHR is considering piloting some elements of the new Open Programs' design with the aim to validate approaches and to familiarize the community with the new processes in a phased manner.

CIHR recognizes the proposed changes may also have an impact on other health research funding groups/agencies, and welcomes the opportunity to engage with funding partners to further discuss the implications of the changes and the next steps needed to ensure minimal disruption to the health research funding landscape.

7. Conclusion

In developing this new design for the Open Suite of Programs, CIHR took into consideration the literature on research funding and peer review; examined national and international research funding programs; and engaged in thoughtful reflection on what new design elements would best work for health researcher and stakeholder communities and for CIHR. We have listened closely to our stakeholders, and based on our own analyses and the existing evidence, have identified a number of changes that could transform the current Open Suite of Programs to better deliver on the full spectrum of CIHR's mandate.

The design of the new Open Suite of Programs continues to evolve. CIHR has been conscious throughout the design of the new Open Suite of Programs to select design elements that would help us meet our mandate and address challenges in our current Open Suite of Programs.

Work is now underway to model and assess the potential impact of these reforms on the existing cadre of funded researchers, peer reviewers, partners, institutions, systems, processes, and staff. In order for this reform to be successful, CIHR needs your input to help identify the strengths and weaknesses of the new design, and help identify areas for improvement before implementing the new design.

As you reflect on this document, general questions you should consider in your feedback include:

• What do you feel are the strengths of the new design for the Open Suite of Programs?
• What are the gaps in this design that CIHR should address to ensure a successful implementation?
• What challenges do you anticipate as a researcher/peer reviewer in adopting these changes?

CIHR is looking forward to having an on-going, active, and productive discussion about the new Open Suite of Programs. You are encouraged to share your perspectives and feedback to help CIHR refine this new design and transform the way health research is funded. As part of the discussion, please submit your comments and questions to our Web-enabled discussion forum, or via e-mail at support-soutien@cihr-irsc.gc.ca.

Date modified:

2015-06-16