CIHR Operating Support Program - Evaluation Summary

About the CIHR Operating Support Program

Approximately three-quarters of the budget at the Canadian Institutes of Health Research (CIHR) is used to support investigator-initiated research (IIR), which are projects created by individual researchers and their teams. The focus in this evaluation is on CIHR’s Operating Support Program (OSP), a sub-program of CIHR’s broader Investigator Initiated Research (IIR) Program. During the period under review, the OSP represents CIHR investments in the Open Operating Grant Program (OOGP), and its successor programs, the Foundation and Project Grant Programs (FGP and PGP, respectively). The OSP, which has an annual budget of over $500M, aims to contribute to a sustainable Canadian health research enterprise by supporting world-class researchers in the conduct of research and its translation across the full spectrum of health. CIHR’s IIR funding is currently provided primarily through the Project Grant Program.

Results: What We Found

• The OSP addressed a continued need for investigator-initiated health research and funding IIR remains an effective means to support health research and build health research capacity.
• Broadly, the OSP (OOGP, FGP & PGP collectively) aligned with the CIHR Act and strategic directions in Roadmap II (the Strategic Plan in place during the evaluation period).
• OOGP funding facilitated the creation, dissemination, and use of health-related knowledge (mainly within academia) and contributed to building health research capacity.
• Bibliometric analyses showed that OOGP-funded researchers and FGP and PGP applicants were more productive and impactful than health researchers in Canada and other OECD countries.
• OOGP-funded research results have demonstrated limited translation of knowledge beyond academia.
• Less than half of OOGP grants involve and impact stakeholders beyond researchers as reported by NPIs though end of grant reports, and less than 15% of grants resulted in the translation of knowledge beyond academia into longer-term health impacts, or socio-economic impacts.
• Given the programmatic changes made at CIHR, there is a need to better define and align the objectives of the PGP in relation to the CIHR Act, specifically regarding building Canadian health research capacity. 
• CIHR needs to improve the monitoring and assessment of outcomes and impacts of its investigator-initiated research funding.
• There are concerns about the availability and reliability of the data from the current end of grant report, which is limiting the ability to assess whether programs are effectively achieving their objectives.
• There is also a lack of output/outcome data collected to assess progress on objectives beyond the end of grant report, which is administered 18 months post grant expiry.
• Although CIHR is making advances in data governance, challenges with data ownership and management further affect the ability to monitor and assess program performance and inform programmatic changes.
• CIHR needs to ensure funding decisions are made equitably.  The evaluation showed that there are differences in funding and outcome characteristics by pillar, gender, career stage, and language of application across individual OSP programs that need to be considered in the design and delivery of programs going forward.

Recommendations and Management Response

1. CIHR should revise the PGP objectives to ensure they are clearly defined, fully aligned with, and support key aspects of the CIHR Act related to building Canadian health research capacity.

   Response: Management agrees and wishes to ensure understanding that the objective of building Canadian health research capacity is a joint responsibility beyond the PGP and is also embedded in the Training and Career Support Framework and action plan. This Framework will define and outline how CIHR will support the achievement of relevant objectives in the CIHR Act. CIHR will develop an objective outlining the role of capacity building within the PGP (informed by the CIHR Framework for Research Excellence and Training and Career Support) and will align peer review and PGP competition processes to support the capacity building objective.

2. CIHR needs to ensure that investigator-initiated research funding is distributed as equitably as possible while minimizing the potential for peer review bias. The design and implementation of investigator-initiated grants must account for differences within the health community observed by the evaluation (e.g., pillar, sex, career stage and language) and well as in the research more broadly.

   Response: Management agrees, and numerous improvements have been made on this front since the completion of the evaluation (e.g., equalization measures related to career stage, gender and language, iterative review of Indigenous applications). CIHR will: review current equalization principles within PGP to identify additional identity dimensions, refine peer review recruitment processes to ensure diversity of membership, analyze the results of the Review Quality Assurance program, and finalize a new bias module.

3. CIHR needs to improve the monitoring and assessment of activities and investments in investigator-initiated research.

   • CIHR needs to enhance the way performance data is collected related to capacity building (e.g., indirect support of trainees), knowledge translation beyond academia (i.e., informing decision making), collaborations, health impacts, and broad socio-economic impacts to better understand the full impact of grant funding.
   • CIHR needs to revise the current end of grant reporting template and process in order to improve the availability, accuracy, and reliability of the data collected.
   • CIHR should consider additional ways to collect data beyond end of grant reports via interim reporting as well as longer term follow-up to assess impact.

   Response: Management agrees. CIHR has committed to the acquisition and design of a new Tri-agency Grants Management System (TGMS) and will continue to contribute to its Business Architecture through the active involvement in the TGMS Business Working Group to validate existing data collection and outline the new data components required to enhance reporting on investments. CIHR will conduct a review of current formats and models of grant reports to ensure the timing and frequency of reporting along with the types and formats align with needs to report on progress and impacts of funding.

About the Evaluation

CIHR's Evaluation Unit conducted the evaluation covering the period from 2011-2012 to 2017-18 to meet requirements of the Policy on Results and the Financial Administration Act, to inform the 2021-2031 Strategic Plan, and to provide CIHR senior management with valid, insightful and actionable findings regarding:

• the performance of the former Open Operating Grants Program OOGP
• the relevance and design and delivery of the successor programs – the Foundation and Project Grant Programs

Associated Links

• Full Evaluation Report
• Management Action Plan (MAP)

Date modified:

2023-11-22