Evaluation of the Collaborative Health Research Projects Program
Final Report

March 2021

At the Canadian Institutes of Health Research (CIHR), we know that research has the power to change lives. As Canada's health research investment agency, we collaborate with partners and researchers to support the discoveries and innovations that improve our health and strengthen our health care system.

Canadian Institutes of Health Research
160 Elgin Street, 9th Floor
Address Locator 4809A
Ottawa, Ontario K1A 0W9

This publication was produced by the Canadian Institutes of Health Research. The views expressed herein do not necessarily reflect those of the Canadian Institutes of Health Research.

Acknowledgements

Special thanks to all participants in this evaluation through end of grant reports, surveys, and interviews, NSERC’s Research Partnerships, NSERC/SSHRC’s Evaluation Division, CIHR’s Research Programs Portfolio – Initiative Management, CIHR’s Financial Planning Unit, CIHR’s Funding Policy and Analytics Unit, and members of the CHRP Working Group as well as the Evaluation Advisory Committee.

Angela (Angel) Mackenzie, Sheldon Polowin, Kimberly-Anne Ford, Sabrina Jassemi (student), Ellie Fletcher, Shevaun Corey, Stephen Kester, Michael Goodyer, and Ian Raskin.

For more information and to obtain copies, please contact: evaluation@cihr-irsc.gc.ca.

Table of Contents

List of Tables

List of Figures

List of Acronyms

Acronym Meaning
AD Assistant Director
AI Artificial Intelligence
AIHS Alberta Innovates Health Solutions
CFI Canada Foundation for Innovation
CGS Canada Graduate Scholarships
CHRP Collaborative Health Research Projects
CIHR Canadian Institutes of Health Research
CRCC Canadian Research Coordinating Committee
CRD Collaborative Research and Development
CREATE Collaborative Research and Training Experience Program
CQDM Consortium de Recherche Biopharmaceutique
ECR Early Career Researchers
EDI Equity, Diversity and Inclusion
FAA Financial Administration Act
GOC Government of Canada
HP-I Health Professional – Investigator
HQP Highly Qualified Personnel
I2I Ideas-to-Innovation
IDR Interdisciplinary Research
IRAP Industrial Research Assistance Program
ISED Innovation, Science and Economic Development Canada
KTU Knowledge/Technology User
LOI Letter of Intent
MOU Memorandum of Understanding
MSF Michael Smith Foundation
NFRF New Frontiers in Research Fund
NPI Nominated Principal Investigator
NRC National Research Council of Canada
NSE Natural Sciences and Engineering
NSERC Natural Sciences and Engineering Research Council of Canada
PRC Peer Review Committee
R&D Research and Development
S&T Science and Technology
SME Small and Medium Enterprises
SPOR Strategy for Patient-Oriented Research
SSHRC Social Sciences and Humanities Research Council
TBS Treasury Board Secretariat
TRL Technology Readiness Level
UD University Delegates

Executive Summary

Program Overview

The Natural Sciences and Engineering Research Council (NSERC) launched the Collaborative Health Research Projects (CHRP) program in 1999, while the Canadian Institutes of Health Research (CIHR) joined the program in 2004. The CHRP program supports interdisciplinary collaborative research, involving any field of the natural sciences or engineering, and any field of the health sciences, to facilitate the transfer and translation of knowledge; generate health and economic benefits for Canadians; create more effective health services and products, and/or strengthen the health care system. CIHR and NSERC share the administrative costs of the program. NSERC administered the program from its inception until the end of 2011, while CIHR has administered the program since 2012. From 2009-10 until 2017-18, NSERC and CIHR invested $78.5M and $82.2M, respectively, in the CHRP program. During this period, the program received 1,063 full applications, awarded 309 grants, and resulted in an average success rate of 11% from applications at the Letter of Intent (LOI) stage, 29% at the full application stage. In April 2018, the Social Sciences and Humanities Research Council (SSHRC) became a partner in the CHRP program’s “Special Call” funding opportunity.

Evaluation Objective, Scope and Methodology

The objective of the evaluation was to provide an independent and objective assessment of the CHRP program’s relevance and performance over the period from 2009-10 to 2017-18. This is the second evaluation of the program; the first evaluation was completed in 2014. Building on the first evaluation, this evaluation used multiple lines of evidence including analyses of documents, end of award reports and other administrative and financial data, surveys, and key informant interviews. The evaluation meets the requirements of the Treasury Board Secretariat (TBS) of Canada under the Policy on Results and the Financial Administration Act.

Key Findings

Relevance

Based on policies and priorities of the Government of Canada and the research community, there is an ongoing need to fund interdisciplinary research (IDR) that fosters collaboration between health and natural sciences and engineering (NSE) researchers, and that facilitates translation and commercialization of research to improve the Canadian health system and related services. As currently designed, the CHRP program does not appear to be the most effective funding mechanism to achieve these needs. Broadly, the CHRP program is well aligned with the mandates of the Tri-agencies and key priorities of the federal government. The program is distinct from and largely complements other funding programs. Unlike other funding mechanisms, the CHRP program has a broad scope, funding projects across the continuum from basic/exploratory research to market-ready technology. The CHRP program funds IDR that integrates health sciences and natural sciences and/or engineering, facilitates collaborations between researchers as well as knowledge technology users, and emphasizes the need for knowledge translation.

However, with respect to knowledge translation, other funding mechanisms (although not directly comparable), including CIHR’s open funding programs, have demonstrated equal or greater success in facilitating the translation, application, and/or commercialization of scalable new technology. There is no consensus on whether the program should focus on funding research at a specific point on the continuum from basic/exploratory research to market-ready technology or specific Technology Readiness Level (TRL). There is clear uptake of this program by researchers, with an average success rate of 11% from LOI stage applications. However, approximately one quarter have received multiple CHRP grants, and while the number of LOIs increased from 2009 to 2012, the number of LOIs has been decreasing since its peak in 2012.

Performance

The CHRP program has continued to effectively facilitate collaborations at the intersection of health and NSE. The integration of health and NSE expertise has been necessary to complete the CHRP-funded research. Beyond the requirements of the program, the collaborations were effective at advancing the projects, and have led to research that would not otherwise have been conducted. The CHRP program has effectively contributed to building interdisciplinary capacity by providing interdisciplinary research and training opportunities for both researchers and trainees. The CHRP program has effectively enabled students to develop the skills and knowledge required to find employment and other revenue-generating opportunities related to their fields of expertise. Trainees reported high levels of satisfaction with the training they had received, noting that they gained exposure to new areas of research and had improved their research, analytical, technical, and professional skills.

There is some evidence that CHRP-supported research has resulted in innovations, efficiencies, technologies, and/or health systems and services. The majority of grant recipients reported that they developed or improved a product/service or process/treatment, or contributed to policies, guidelines, or regulations. Some grants have resulted in patents. However, despite these outcomes, the evaluation found that the CHRP program’s knowledge technology users (KTUs) facilitated the translation, application and/or commercialization of scalable new technology to a moderate extent. Surveyed Recipients reported that the scale-up and use of research results by KTUs was more likely to occur in the future. KTU involvement varied among CHRP projects. Despite formalizing the KTU involvement in 2012 (along with a requirement to include a KTU in all stages of the research process where applicable), evaluation findings indicate that the expected increase in KTU engagement and use of research results (as noted in the previous evaluation) was not observed.

There is limited evidence that CHRP-funded research has resulted in economic and health benefits for Canadians. For example, less than 25% of grants resulted in economic benefits (e.g., spin-off companies or product licenses), and less than 15% of grants reported health benefits (e.g., professional practice or policies/guidelines). Surveyed Recipients reported that these impacts are more likely to occur in the future and research shows that longer term outcomes and impacts of health research can take upwards of 17 years. There is also evidence that IDR research impact takes longer to come to fruition. Based on available evidence, the majority of research projects funded by the CHRP program are not technology ready. An assessment against the Technology Readiness scale shows that 69% of projects started at the lower end and 64% of projects reported an increase in technology readiness, with an average increase of two levels. While the CHRP program funds projects across the continuum from basic/exploratory research to market-ready technology, commercialization is more likely to occur the closer the project is to market-ready technology or the later stages of the technology readiness scale. Therefore, the objectives of the program to support research across the levels of technology readiness, and also to produce commercializable outputs, may be contradictory.

Taken together, the findings suggest that some design and delivery elements of the program may be limiting the achievement of intended outcomes; in particular, those related to the KTU requirement and the expected translation and commercialization of research results. The three-year funding period of the program, particularly given that the program offers support for research along the full continuum of technology readiness including early-stage innovation, presents challenges related to translation of project outputs and the achievement of longer-term outcomes.

There is a need for the Tri-Agencies to improve the availability and consistency of data collection and management, and to ensure effective ongoing performance measurement. The agencies currently have differing practices and end of grant reports, which presented challenges to the tracking and identification of stakeholders, as well as the assessment of outcomes and previous funding history of CHRP-funded researchers. Performance measurement and data limitations restrict the ability to determine whether the translation of knowledge to KTUs and stakeholders is effectively occurring.

Recommendations

The evaluation makes two recommendations aimed at improving the performance of the CHRP program.

Recommendation 1
  1. CIHR and NSERC should review the CHRP program objectives and identify the best ways to achieve these objectives, either through redesign of the program or delivery via other funding opportunities.
Recommendation 2
  1. Performance measurement and data availability related to the CHRP program should be strengthened.
    1. CIHR needs to improve the performance measurement of the CHRP program and enhance the way that data is collected related to collaborations and partnerships as well as longer term outcomes (i.e., innovations and health care efficiencies) to better monitor the impact of CHRP program funding.
    2. NSERC and CIHR need to establish a means by which to improve the consistency of data collection, data management and data sharing processes related to the CHRP program.

Program Profile

Program Description

The Natural Sciences and Engineering Research Council (NSERC) launched the Collaborative Health Research Projects (CHRP) program in 1999 with federal funds targeted towards the establishment of the Canadian Institutes of Health Research (CIHR), Canada’s national health research funding agency. CIHR succeeded the 40 year-old Medical Research Council of Canada in 2000.

Training highly qualified personnel (HQP) has been a key goal of the program since 2001.Footnote 1 In 2002-03, it began supporting collaborative research between health and NSE, and translating knowledge to end-users. The Program intersects the mandates of both agencies and supports a broad spectrum of research activities, ranging from basic to applied research. The program vigorously supported early-stage research, particularly in its first decade of operation. Between 2003 and 2011, the Program aimed to achieve the following objectives:

  1. Translate research results to end-users and stakeholders;
  2. Encourage the NSERC and CIHR communities to collaborate and integrate their expertise in novel research activities;
  3. Advance IDR leading to knowledge and technologies useful for improving the health of Canadians; and
  4. Train students, research assistants, fellows and other HQP in collaborative and IDR relevant to health.

CIHR and NSERC share the costs of administering the CHRP program. Between 2009-10 and 2017-18, NSERC and CIHR invested close to $78.5M and $82.2M, respectively in the CHRP program (see Table 2: Total Annual Investments in the CHRP Program by CIHR and NSERC (in Millions) 2009-10 to 2017-18). There have been substantial changes in the research funding landscape over the past decade. One of the most notable developments has been the growing demand for innovative research that leads to the creation of new products, services and processes. NSERC administered the CHRP program until the end of 2011, while CIHR has managed it since then. In April 2018, the Social Sciences and Humanities Research Council (SSHRC) became a partner in the CHRP program’s “Special Call” funding opportunity. More than $24M was available for the call, including nearly $6M for projects that investigate the ethical, legal and societal impacts of using Artificial Intelligence (AI) in the health sector.Footnote 2

In 2012, changes in the program guidelines and selection criteria increased the requirement for end-user partner involvement in projects, throughout all phases of the research process as appropriate. Although the program had always encouraged the participation of non-academic KTUs from all sectors of the economy, it was not formally required until 2012. According to the 2014 evaluation of the CHRP program, a lack of partner engagement was reported as a factor limiting knowledge transfer and partner use of research results. Thus, the inclusion of a partner as a criterion for application was introduced as a requirement, as this was expected to increase the achievement of outcomes.

The objectives of the CHRP program, changed slightly between 2003 and 2012, and are currently as follows:

The CHRP program has supported the research priorities of several CIHR Institutes (such as the Institute of Cancer Research, the Institute of Genetics, and the Institute of Musculoskeletal Health and Arthritis). The program also funded various CIHR strategic investment envelopes and “special calls”. For example, between 2012 and 2016, it provided more than $1.9M to develop molecular diagnostics and point of care devices, in support of the Personalized Medicine Signature Initiative [ PDF (323 KB) - external link ].

The program’s logic model (Figure 1: CHRP Program Logic Model), which identifies the linkages between the activities of the CHRP program and its ultimate outcomes, can be seen in Appendix B.

CHRP Application Process

The CHRP program funds researchers with defined projects, ranging from fundamental knowledge creation to research on knowledge application relevant to industry or public policy, for up to three years. To qualify for CHRP program funding, applicants and co-applicants must hold eligible appointments at a Canadian postsecondary institution that take effect by April 1 following the year of the application. Nominated Principal Investigators (NPIs) must be eligible under NSERC and CIHR eligibility guidelines. The application requires at least two Principal Applicants (including the NPI), one from the natural sciences or engineering community and one from the health sciences community.

The application process for the CHRP program includes two phases. In the first phase, applicants submit a letter of intent (LOI) and research proposal to the program administrator. Administrators screen the LOIs to determine if they meet the program’s objectives, and arrange for the appropriate expertise on the peer review committee. In the second phase, the applicants whose projects best fit the program’s objectives (based on the LOI criteria) submit a full application, which among other things must include a detailed project budget.

Between 2005-06 and 2010-11, the number of applicants to the CHRP program grew by more than 50%, rising from 209 to 342 (applications at the LOI stage). While application pressure grew over this period, the number of applications at the LOI stage peaked in 2011-12 at 516, and then showed a decline to 212 in 2017-18. During the evaluation period from 2009-10 until 2017-18, the CHRP program received 1,063 full applications and awarded 309 grants, resulting in an average success rate of 11% from the LOI application stage (29% at the full application stage).  By comparison, the average success rate at the full application stage for CHRP is 29%. Based on recent program evaluation data, the average success rate for CIHR’s Operating Support Program (OOGP) was 17% (between 2006-2018), and the average success rate for NSERC Discovery Grants was 64% (between 2013-2017). See Table 3: CHRP Program Application and Success Rates 2009-2017.

Description of the Evaluation

Evaluation Purpose and Scope

This evaluation covers the CHRP program’s activities and achievements during the 2009-10 to 2017-18 period.Footnote 3 The purpose of this evaluation is to provide independent, objective and actionable evidence to the management of CIHR, NSERC and SSHRC, regarding the:

The evaluation of the CHRP program is included in both CIHR’s and NSERC’s 2018-19 Evaluation Plans, in order to meet accountability requirements outlined in the Treasury Board’s Policy on Results, and subsection 42.1 of the Financial Administration Act (FAA). The evaluation is intended to guide CHRP Program decision-making as well as planning for IDR. A key scoping decision in the current evaluation was to gain greater insight into the role and involvement of knowledge users and partners in CHRP-funded grants, expanding on the limited data gathered in the previous evaluation. The current evaluation did not explicitly consider the impact of SSHRC’s investments on the CHRP program’s performance, given that the relevant funding opportunity occurred after the period under review (2009-10 to 2017-18). Thus, it is considered to be beyond the scope of this evaluation and too early to assess the outcomes of this investment

Previous Evaluation

In 2014, NSERC’s Evaluation Division conducted an evaluation [ PDF (552 KB) - external link ] of the CHRP program, focused on the relevance and performance of the program, covering the time period from 1999-2000 to 2008-09. Findings from the first evaluation indicated that the program helped NSERC and CIHR meet their mandates by supporting interdisciplinary and collaborative research and knowledge translation leading to health and economic benefits for Canadians. Findings also confirmed the limited availability of other funding sources and the continued need for the program, which suggested that the program fills a niche in the continuum of research funding programs in Canada.

The evaluation also found that the CHRP program was effective in meeting its outcomes pertaining to collaborations between NSE and health researchers, multidisciplinary research (i.e., interdisciplinary research; IDR), as well as training of HQP. However, while the program made a considerable contribution towards addressing health-related issues through multidisciplinary research it was noted that the relatively early-stage nature of the research and a lack of partner involvement appeared to be the main factors that limited knowledge transfer and partner use of research results. The previous evaluation conducted surveys of partners and HQP; however, the sample sizes for these groups were too small to report on, and therefore data for these individuals were not included.

Overall, it was found that the CHRP program was delivered in an efficient manner in that the administrative costs to deliver the program were comparable to those of the NSERC Research Partnership Programs Directorate. Although, it should be noted that there were limitations to the analysis of cost efficiency due to data availability and differing practices in calculating program costs between NSERC and CIHR.

The evaluation made two recommendations:

  1. Consider continued funding to collaborative health research involving health and NSE researchers through the CHRP program and further clarify and communicate the position of the Program in the continuum of funding opportunities provided by NSERC and CIHR; and,
  2. Make improvements to program design and ongoing performance measurement through:
    1. Assessing whether it is feasible to provide more substantial feedback on LOIs to applicants;
    2. Reviewing the program’s performance measurement strategy (including the logic model) to ensure that it effectively monitors the extent to which the CHRP program supports its new objectives, the impacts of the new partner requirements on the research community and program impacts on HQP;
    3. Recording information on whether researchers can be identified primarily as NSE or health researchers; and,
    4. Establishing a protocol for sharing Applicant, Partner and HQP data between NSERC and CIHR.

Evaluation Methodology

Evaluation Approach

The evaluation employed both quantitative and qualitative approaches to data collection and analyses. Reflecting best practices in evaluation as well as TBS guidelines, multiple lines of evidence were used to triangulate evaluation findings. Lines of inquiry included a document and data review (including end of grant report data); an environmental scan; a Funding History Analysis (based on administrative data from NSERC and CIHR); and surveys of Recipients (Nominated Principal Investigators and Co-applicants) and Applicants (Nominated Principal Investigators and Co-applicants), Trainees and Partners (including both KTUs and other partners, due to the fact that researchers may have had more than one partner, and also that a KTU was not formally required until 2012). There were also key informant interviews conducted with CHRP program staff (CIHR, NSERC, SSHRC), CIHR, NSERC and SSHRC Senior Management, Recipients and Applicants, KTUs/Partners, Peer Review Committee (PRC) Chairs, University Delegates (UDs), Trainees, and Assistant Directors (ADs) of CIHR Institutes. Please note that for the purposes of this report, recipients are defined as those who received a CHRP grant; whereas, applicants are defined as those who applied for but did not receive a CHRP grant. As outlined below, some applicants received other funding for the research they included in their CHRP applications.

The current evaluation built on the previous evaluation of the CHRP program (2014), where appropriate and feasible. Although no equity, diversity and inclusion (EDI) concerns related to the award of CHRP program grants were identified, it should be noted that demographic information was collected related to EDI variables, including gender, language, Indigenous status, minority status, and disability status. Equity analyses were undertaken based on discipline (NSE vs. health) and gender, where possible. Furthermore, the evaluation reflects any feedback provided on potential EDI-related barriers associated with the CHRP program eligibility guidelines. Additional details about the methodology are provided in Appendix C: Methodology – Additional Details.

Evaluation Questions

The evaluation addresses the following specific questions.

Relevance
  1. To what extent is there an ongoing need for the CHRP program?
    • 1.1. What are the distinctive aspects of the CHRP program that facilitate interdisciplinary research (IDR) at the intersection of the participating funding agencies’ mandates?
    • 1.2. Does the program align well with the mandates of participating funding agencies and key priorities of the federal government?
    • 1.3. Does it duplicate or complement other federal programs?
    • 1.4. Should the CHRP program focus on specific technology readiness levels (TRL) (e.g., TRL 1-3)?
Performance
  1. How effectively has the CHRP program facilitated collaborations between CIHR and NSERC researchers?
    • 2.1. To what extent has the integration of natural science and engineering (NSE) and health expertise been necessary to complete the CHRP-funded research projects?
  2. How effectively has the CHRP program facilitated capacity building?
    • 3.1. To what extent has innovative, interdisciplinary capacity been built among CHRP-funded researchers within the health care or science and technology sectors?
    • 3.2. To what extent does the program enable students to develop the skills and knowledge required to find employment and/or other revenue-generating opportunities related to their fields of expertise?
  3. What are the innovations, technologies and/or health systems and services resulting from CHRP-supported research?
    • 4.1. Has CHRP-supported research led to innovations and/or efficiencies in the health care field?
    • 4.2. How effectively have its knowledge technology users (KTUs) facilitated the translation, application and/or commercialization of scalable new technology?
    • 4.3. To what extent has the program generated economic, health (e.g. diagnoses, treatments) and social benefits for Canadians?
    • 4.4. How technology-ready are research projects funded by the CHRP program?
  4. Is the CHRP Program delivered in a cost-efficient manner?

Limitations of this Evaluation

The evaluation leveraged a variety of data sources. The value of this evidence-based strategy lies in the efficiency of utilizing currently available data and synthesizing it through a single evaluative lens. However, as with all evaluations, this evaluation encountered some limitations (discussed in more detail in Appendix C: Evaluation Limitations and Mitigation Strategies). The main limitations associated with this evaluation are:

Evaluation Findings

Relevance

Key Findings

There is an ongoing need for the funding of interdisciplinary research involving NSE and health sciences

Based on policies and priorities of the Government of Canada (GOC) and the research community, there is a continued need to support IDR, the translation of research results in the areas of Science and Technology (S&T), and commercialization of research that improves the Canadian health system and related services. However, it is not clear that the CHRP program, as currently designed, is the most effective funding mechanism to achieve these needs. The CHRP program aims to fund collaborative IDR that integrates health and natural science and/or engineering expertise in order to develop innovative new health technologies, processes and policies. Few, if any other funding programs in Canada, support such research, irrespective of the maturity level of a project’s underlying technology or process. Furthermore, a recent study published in the Nature research journal, Palgrave Communications (Okamura, 2019), reported that IDR is associated with a 20% higher research impact than single discipline research in health and natural sciences.

The CHRP program aligns well with the key priorities of the federal government

The evaluation shows that the program is aligned well with the key priorities of the federal government. The environmental scan, conducted as part of the evaluation, found that the CHRP program aligns with recent federal budget priorities. For example, several key initiatives for Budget 2017 focused on expanding IDR initiatives that integrate health sciences and NSE expertise. Budget 2018 prioritized idea generation and translation for commercialization within the global economy. Additionally, the program also aligns with Innovation, Science and Economic Development (ISED) Canada’s key priorities which focus on transforming ideas into new products and services; and is consistent with the both the Science and Innovation Strategy (2014), which emphasized fostering partnerships and making Canadian science more collaborative, and Canada's Science Vision. Specific objectives include making science more collaborative through increased support for research through the granting councils, supporting universities, colleges and polytechnics, and helping businesses, academia and government to work together.

The CHRP program aligns with the Fundamental Science Review [ PDF (7.82 MB) - external link ]’s call to increase federal support and strengthen systems within granting councils to better adjudicate IDR proposals and to support programs that encourage IDR. The recommendations made by the Fundamental Science Review surrounding strategies to encourage multidisciplinary research included the formation of the Canadian Research Coordinating Committee (CRCC). The CRCC was created to improve the coordination efforts of the Tri-Agencies and a key priority is interdisciplinary, international, high-risk/high-reward, rapid-response research.

Almost all interviewees across respondent groups felt that the CHRP program aligns well with federal government priorities. All three program staff interviewed considered the CHRP program congruent with federal priorities, particularly in relation to support for collaborative IDR, and the recommendations of the Fundamental Science Review. All four PRC Chairs interviewed considered federal support of the CHRP program appropriate despite its “high-risk/high-reward” nature.

The CHRP program aligns well with the mandates of the Tri-Agencies

The evaluation shows that the CHRP program is also aligned well with the mandates, priorities, and funding mechanisms of the Tri-Agencies (as outlined in their respective Acts: Canadian Institutes of Health Research Act, 2000; the Natural Sciences and Engineering Research Council Act 1985; and the Social Sciences and Humanities Research Council Act, 1985), as well as the Canada Foundation for Innovation (CFI).

All three senior Tri-Agency managers (CIHR, NSERC, and SSHRC) interviewed confirmed the program’s alignment with their organizations’ mandates and objectives and all agreed that the CHRP program aligns with federal government priorities (e.g., healthcare is always a priority). One of the ADs of a CIHR Institute indicated that the alignment was consistently high. However, the other Institute AD noted that the degree of alignment between its priorities and those of the CHRP program fluctuated over time: in certain years, depending on the focus of the CHRP program “special calls”, there was greater congruence between the CHRP funding opportunities and Institute priorities.

The CHRP program has features that are distinct from other funding programs

Although objectives related to IDR and/or partnerships to foster knowledge translation are not unique to CHRP, the program does have elements that make it distinct from other IDR support (i.e., currently available funding opportunities and programs) that has been or is provided by the Tri-Agencies and other funders. The environmental scan identified that the integration of the following three elements distinguishes the CHRP program from other funding opportunities or programs: IDR that integrates health science, natural sciences and/or engineering; the facilitation of research collaborations; and an emphasis on knowledge translation. This finding is consistent with those of the previous evaluation. Another distinctive aspect of the CHRP program is that it funds projects along the full continuum from basic/exploratory research to market-ready technology.

A specific definition of IDR is not provided in CHRP program documentation; therefore, a definition was adopted for the purposes of this evaluation. The definition used was that provided by the National Science Foundation (NSF). IDR is defined by the NSF as a mode of research conducted by teams or individuals that integrate information, data, techniques, tools, perspectives, concepts and/or theories from two or more disciplines, or bodies of specialized knowledge, to advance fundamental understanding, or to solve problems whose solutions extend beyond the scope of a single discipline or area of research practice. This definition is applied specifically to the context of collaborative health and NSE research.

Program stakeholders recognized the importance of the CHRP program. Survey results showed that the most common reason Recipients and Applicants (NPIs) applied to the CHRP program was because it was a means to enable them to conduct IDR (Recipients: 78% out of 103; Applicants: 72% out of 76). Recipients and Applicants also felt that the CHRP program’s requirement for collaboration between health and NSE researchers facilitated research that otherwise would not occur (Recipients: M = 4.12 out of 5, SD = 1.11; Applicants: M = 3.96 out of 5, SD = 1.13). Both Recipients and Applicants strongly agreed that the CHRP program funding fulfilled an important need among researchers (Recipients: M = 4.68 out of 5, SD = 0.73; Applicants: M = 4.50 out of 5, SD = 0.76).

Most surveyed Recipients (NPIs: 79% out of 101; Co-applicants: 70% out of 103Footnote 4) reported that they would not have continued with the project without CHRP program funding, suggesting that CHRP is the only mechanism that may have enabled the research to proceed. More than half of Applicants (NPIs: 73% out of 63; Co-applicants: 57% out of 186) did not continue with the project.

Almost all key informants interviewed identified features of the CHRP program that are distinct from other funding opportunities and/or programs. All Recipients recognized that the CHRP program facilitates the creation of IDR teams that address issues whose resolution requires NSE and health expertise. In particular, they noted that it supports projects at the intersection of CIHR’s and NSERC’s mandates, including those that are rejected and/or improperly assessed by other funding mechanisms. Almost all (of 14) interviewed Applicants (NPIs) considered the CHRP program’s most distinctive attribute to be its propensity to integrate scientific and engineering expertise for the purposes of creating healthcare applications and solutions. All four PRC Chairs also agreed that the CHRP program’s distinctiveness reflects its key objectives of facilitating health and NSE research collaborations, knowledge translation, and HQP training.

Two out of three Partners interviewed affirmed the program’s ability to foster links among commercial partners, academic researchers, laboratories, health practitioners, and other stakeholders. Approximately two thirds of Partners surveyed (62% out of 45) felt that it would not have been possible to achieve the objectives of their organization by engaging in the CHRP program through any other avenue. While interviewed Trainees were aware of other funding programs (e.g., Foundation Grants, Canada Graduate Scholarships), they viewed the CHRP program as the only one that supports the type of research they were engaged in.

Four out of the five UDs interviewed strongly supported the CHRP program, primarily because of its interdisciplinary and collaborative nature, and emphasis on knowledge translation. In general, UDs supported retaining the program, although in some cases with minor modifications. A few UDs warned that the program risks forfeiting its unique position in Canada’s research funding landscape, due to the recent emergence of programs that replicate some of its features.

The CHRP Program complements other federal programs

Although there is some overlap between the CHRP program and other federal funding programs(e.g., eHealth Innovations and Operating Grant: Bioinformatics and Computational Biology; 2015, New Frontiers in Research Fund [NFRF]), there are fundamental differences between its scope and mandate and those of other programs, notably its focus on collaborative research between health and NSE across a broad spectrum, ranging from basic to applied research. Overall, evidence from the evaluation suggests that the CHRP program complements without fully duplicating other programs, and no other major funding program in Canada that supports collaborative IDR that integrates health science and NSE expertise in the same way.

The environmental scan included an assessment of overlap, duplication, complementarity, and/or synergies between the CHRP program and other funding opportunities/programs. The following are the key findings from the environmental scan relating to potential overlap and differences between CHRP and other research funding programs. Although CHRP’s focus on encouraging collaborative IDR may appear to overlap with National Research Council of Canada’s (NRC) Industrial Research Assistance Program (IRAP), IRAP focuses primarily on supporting small and medium enterprises (SME) to innovate and penetrate new markets, whereas CHRP targets recipients based on the nature of their collaborations (i.e., health and NSE, combined with a KTU) rather than based on the characteristics of the industry partner and their respective organization. NSERC’s Ideas-to-Innovation (I2I) program supports university-based technologies that have potential applicability in business settings, focusing on research and development (R&D) in the early stages of technology validation and market connection, with a focus on supporting NSE research; whereas the CHRP program supports projects regardless of their position along the TRL continuum, at the intersection of health and NSE. The NFRF is currently only open to Early Career Researchers (ECR), unlike CHRP which does not specify or describe career stage of researchers in its requirements.

The environmental scan also identified a number of additional funding programs at the federal and provincial levels that broadly complement the CHRP program in terms of collaborative or interdisciplinary research and/or knowledge translation:

Survey findings also indicate that the CHRP program complements other programs without fully duplicating them. On a 5-point scale from Strongly Disagree to Strongly Agree, both Recipients and Applicants (NPIs) agreed, on average, that the CHRP program was the only source of funding for collaborative research between health science and NSE (Recipients: M = 4.11 out of 5, SD = 1.04; Applicants: M = 3.73 out of 5, SD = 1.24); however, Co-applicants did not agree as strongly on this (Recipients: M = 3.5 out of 5, SD = 1.1; Applicants: M = 3.46 out of 5, SD = 1.06). None of the interviewees from any of the respondent groups could identify a federal funding mechanism that duplicates the CHRP program’s scope and mandate, and, as a result, consider the program to be unique.

While there is no duplication indicated, one-half of interviewees across the majority of respondent groups could identify other funding mechanisms that complement the CHRP program. Some Recipients and Applicants were aware of programs that complement the program, and referenced both federal and non-federal programs and entities such as Genome Canada, NSERC’s Strategic and Discovery Grants, Ontario Research Foundation, NET TECH, AGE-WELL, MITACS programs and the funding offered by charitable foundations (e.g., The Heart and Stroke Foundation). While such programs fund research in similar areas to that funded by CHRP, they have a narrow focus and fund fewer types of research activities on the continuum of basic to applied research or related to IDR than the CHRP program. For example, two Applicants noted that the focus of NSERC’s I2I program resembles the CHRP program, although the NSERC’s I2I program requires that the technology be more mature (i.e., technology cannot be basic or discovery research) than what is funded through CHRP.Footnote 5

Two out of three program staff interviewed agreed that the CHRP program complements other federal programs. They noted the existence of “one-off” or more narrowly defined programs that address some of the same strategic priorities (e.g., eHealth) as the CHRP program. Program staff stated that its distinctiveness lies in its ability to bring IDR researchers together to improve the efficiency and capabilities of Canada’s healthcare system. They also stated that, unlike other funders, NSERC does not cover the development costs of medical devices, due to its “restrictive” (i.e., NSE-centric) mandate. Senior managers contended that the CHRP program fills the void between NSERC and CIHR programs. Senior managers identified that the NFRF might partially overlap with CHRP; however, they expect that the CHRP program will remain unique as long as it continues to fund projects that are technologically oriented. Analyses of administrative data show some support for the benefits of the CHRP program for NSE researchers (compared to health researchers) as 59% (n = 158) of CHRP Recipients have received other funding from NSERC, whereas only 23% (n = 62) of CHRP Recipients have received additional funding from CIHR.  

One of the interviewed Partners could not identify any program that duplicates or complements the CHRP program. However, this Partner stated that CIHR used to offer a “Proof of Concept” grant, referring to CIHR’s sunset Proof of Principle (POP) program that was less interdisciplinary than the CHRP program. The respondent noted that pharmaceutical companies – which frequently collaborate with researchers, governments and international organizations – often make substantial investments in healthcare technology, but they tend to be involved in projects that are further “downstream” than those that the CHRP program funds.

The PRC Chairs were unaware of any federal funding opportunities/programs that duplicate the CHRP program, although they stated that some of NSERC’s and CIHR’s “open programs” complement it. They cited NSERC’s I2I program and Collaborative Research and Development (CRD) grants as examples as well as elements of the NFRF. One Chair referenced the Strasbourg-based Human Frontier Science Program, which supports IDR projects in the physical and environmental sciences. The discovery-oriented initiative requires participation by investigators from at least two countries. Finally, another Chair remarked that although several programs fund medical applications, they typically exclude the developmental costs associated with it. According to the Trainees interviewed, other research funding programs offer training opportunities (Foundation Grants, Canada Graduate Scholarships [CGS]); however, only the CHRP program funds training which is by design interdisciplinary and collaborative in nature.

The UDs indicated that the NFRF and Discovery Programs partially overlap with the CHRP program. However, the ADs could not identify other initiatives that duplicate the CHRP program, because of its ability to bring together researchers from the health, natural sciences and engineering domains.

No consensus on whether the program should focus on specific technology readiness levels

As described above, the CHRP program funds a broad spectrum of research activities, ranging from early concept basic research to later stages of applied research. One objective of the program is for CHRP-funded research to result in knowledge translation, including commercializable outputs. Therefore, it is useful to understand at which stage along the continuum the technologies and innovations produced through the CHRP-funded projects start and finish. Although there is currently no specific requirement related to TRL at any stage of the project outlined in the funding opportunity, the technology readiness level (TRL) scale was introduced in this evaluation to measure the relative stages of commercialization of CHRP-funded research projects. The TRL scale, originally developed by NASA, is used by organizations around the world to rank the developmental state of a technology as it evolves from concept to commercialization (see Table 4: Technology Readiness Level (TRL) Scale Framework).  The 9-step scale covers the research process from the basic research stage to point of being fully commercialized: from Technology Readiness Level (TRL) 1: basic principles observed and reported; to TRL 9: actual process and/or project proven successful.

Interviewed stakeholders had varying views regarding whether the CHRP program should target specific levels of technology readiness:

Performance

Key Findings

The CHRP program continues to facilitate new and existing collaborations between health and NSE researchers

The CHRP program aims to facilitate two types of collaborations: collaboration between NSE and health researchers to integrate their expertise; and collaboration between researchers and KTUs/partners in order to engage the latter in the research process and ultimately enable their use of the research results. In terms of the objective of facilitating collaborations between NSE and health researchers, this is supported through the requirement of applicants to include one researcher from the NSE community and one from the health sciences community in the principal applicant roles (NPI and Co-applicant). Additional co-applicants are also allowed.

Both the previous (2014) and current evaluations found that the CHRP program contributed to establishing new relationships between health and NSE researchers and maintaining existing relationships between researchers following the completion of their projects. The CHRP program previously required the formation of new relationships between co-applicants, although this requirement was dropped in 2012. Surveyed CHRP Recipients (n = 99) reported an approximately equal number of new (NPIs: M = 2.1, SD = 1.4, Range: 1-8; Co-applicants: M = 2.9, SD = 2.5, Range: 1-17) versus existing relationships (NPIs: M = 2.0, SD = 1.3, Range: 1-6; Co-applicants: M = 2.4, SD = 1.4, Range: 1-7) per application with their co-applicants, which is comparable to (although slightly higher than) the number of new relationships identified in the previous evaluation (M = 1.5). Surveyed Recipients also reported that 89% of grants (out of 75) involved new co-applicant relationships (compared to 81% in the previous evaluation), and 93% of grants involved existing relationships.

An analysis of NSERC end of grant data for the program showed that CHRP grant applications had an average of 1.2 collaborators each (SD = 2.0), although surveyed Recipients reported higher numbers of co-applicants per grant application, on average (NPIs: M = 3.0, SD = 1.8; Co-applicants: M = 4.0, SD = 2.7). Reported numbers of co-applicants are roughly comparable to the findings from the previous evaluation, wherein surveyed Recipients (NPIs and Co-applicants) identified an average of 2.6 co-applicants per grant. In terms of involvement on the grants, NPIs more frequently initiated the project(s) themselves (Recipients: 93% out of 97; Applicants: 86% out of 84).

All three senior managers interviewed indicated that the requirement for CHRP-funded projects to include a multidisciplinary team has led to an increased number of collaborations and/or enhanced existing ones. All three program staff suggested that the CHRP program, while also increasing health-NSE collaborations, is likely more effective at sustaining existing collaborations than fostering new ones, since the program has many repeat grantees. Administrative data also show that 23% of CHRP funded Recipients have held multiple CHRP grants. Many of the Recipients interviewed had participated in multiple CHRP-funded projects and several revealed that at least one of their other projects involved a new collaboration. Almost all Recipients reported collaborating with their NSERC/CIHR counterparts on some aspect of the funded project prior to receiving a CHRP grant, and believed that the CHRP project enhanced their collaboration.

Collaborations may be driven more so by NSERC researchers

An analysis of administrative data from NSERC and CIHR suggests that the collaborations between health and NSE may be driven more often by NSERC researchers than by CIHR researchers on CHRP grants. More specifically, an analysis of the funding history of Recipient NPIs identified that over half (59%, n = 158) of CHRP-funded researchers were also funded as NPIs through other NSERC grants, compared with less than a quarter (23%, n = 62) who were funded as NPIs through other CIHR grants. Currently, there are no other grants at NSERC that fund research aimed at improving health outcomes. However, consistent with the findings of the previous evaluation, the inconsistencies in data management between CIHR and NSERC made it challenging to fully assess various aspects of the Recipients’ relationships, agency affiliations, and funding history. Therefore, these results should be interpreted with some caution. Improvements in administrative program-related data collection and management are needed and additional analyses are recommended by the program to fully understand its uptake from both the NSE and health research community.

Researchers were highly satisfied with collaborations

Across the various stages of research, surveyed Recipient NPIs (n = 96) reported that their co-applicants were highly involved in most stages, particularly the development of the research idea/question and the research proposal (82%), and analysis/interpretation of results (81%). Most co-applicants were also involved in the other stages, such as training/supervising of HQP (77%) and data collection/project implementation (73%). Involvement from co-applicants was lowest for end of grant knowledge translation activities (51%).

Survey findings indicated that, consistent with the previous evaluation, Recipients (both NPIs and Co-applicants) were very satisfied with all aspects of their collaboration with co-applicants. The aspects of collaboration assessed included communication, decision-making, involvement of co-applicants from diverse disciplines, productivity in terms of knowledge translation (e.g., papers, patents, products, services and processes),  and overall contribution to the project (see Table 5: Recipients’ (NPIs') satisfaction with collaborations and Table 6: Recipients’ (Co-applicants’) satisfaction with collaborations). NPIs reported experiencing few challenges with their co-applicants, other than administrative burden, which was identified by two thirds of NPIs (64% out of 50).

CHRP-funded collaborations were effective in advancing projects

Recipients felt the collaborations were successful in advancing projects to a great extent (NPIs: M = 4.29 out of 5, SD = 0.78, n = 92; Co-applicants: M = 4.04 out of 5, SD = 0.97, n = 77). Applicants to the program who were not successful in securing a CHRP grant, but who had continued with their projects, felt that their collaborations had been slightly less effective in contributing to project outcomes (M = 3.55 out of 5, SD = 1.44, n = 11; note low n for this group).

Most key informants interviewed (across all respondent groups) felt that the CHRP program enhances collaboration between health and NSE experts through the formation of IDR teams that require both health and NSE expertise to bring the results of the research to fruition. All CHRP Recipients interviewed said that being part of an IDR team increased their knowledge of other fields, particularly those within the NSE and health realms. They also felt that the engagement of interdisciplinary teams with expertise in diverse fields has led to the development of new technologies and processes that have benefitted Canada’s health care sector. The following are some examples of the types of projects reported by interviewees:

Several Recipients interviewed felt that collaboration with researchers from other disciplines (and KTUs) ensured that their projects remained focused on practical applications. Moreover, a few Recipients reported that their collaborations enabled them to recruit participants for clinical trials, thereby achieving greater progress on their projects. Although some Recipients reported that they could have achieved similar results without these collaborations, they indicated that it would have taken longer. Most UDs (out of five) interviewed also believed that the productivity and innovativeness of experts increased as a result of their interdisciplinary collaborations, which facilitated technological innovation and knowledge translation.

Most surveyed Recipients reported continuing to collaborate with their co-applicants on the same project (NPI’s: 82% out of 90; Co-applicants: 79%, out of 77). A slightly lower proportion of Applicants (Co-applicants) reported this ongoing collaboration with other co-applicants (69%, out of 55). Nearly all NSERC end of grant survey respondents indicated they were planning future collaborations (99% out of 79). Most recipients interviewed (out of eight) claimed that they maintained, or expected to maintain, a working relationship with their CHRP collaborators at the conclusion of the project. They reported applying for additional research grants with their partners and/or advancing their CHRP-funded research through subsequent collaborations.

CHRP-funded collaborations resulted in research that otherwise would have been delayed or not conducted

Most Recipients (NPIs: 79% out of 101; Co-applicants: 70% out of 103) reported that they would not have continued with the project without CHRP program funding. More than half of Applicants (NPIs: 73% out of 63; Co-applicants: 57% out of 186) did not continue with the project (although this was true less frequently for Co-applicants than NPIs), and those that did largely modified the project and/or secured other funding for the project. Most Recipients surveyed (NPIs: 87% out of 101; Co-applicants: 81% out of 104) felt that the monetary value of the CHRP program grant was sufficient to meet the objectives of their projects.

The latter finding was also reflected in interviews with Applicants, who reported that projects were either delayed or not pursued without CHRP funding. Over half of Recipients interviewed (five out of eight) believed that their project would not have achieved its results without collaboration with other NSERC/CIHR researchers (and KTUs). Both ADs see a strong and continued need for the CHRP program because it encourages grantees to interact with researchers and developers from other communities, including those who have not worked in health-related research. They maintained that the technological innovations and “real-world knowledge translation” associated with the CHRP program likely would not occur in its absence.

Interdisciplinary training opportunities have been provided to other researchers and trainees

Another key objective of the program is to provide training opportunities in collaborative and interdisciplinary research relevant to health, and to prepare trainees for employment opportunities. The CHRP program has enabled other researchers (those beyond the NPI and co-applicants) as well as HQPs (e.g., graduate and post-graduate student trainees) to gain new skills and valuable work experience. Trainees carried out IDR and interacted with subject matter experts from diverse fields, including those from other sectors of the economy (e.g., private companies, NGOs, government).

Surveyed recipients (NPIs) reported that the types of HQP included most frequently in their CHRP grants, both on average and in total, were other researchers (M = 3.6, SD = 2.8, n = 54, total = 197) and undergraduate students (M = 3.5, SD = 3.1, n = 57, total = 200). This included both Canadian and international HQP. However, it should be noted that graduate students were involved in more CHRP grants than were other researchers beyond co-applicants (PhD students: M = 2.7, SD = 1.8, n = 69, total = 188; Master’s students: M = 2.6, SD = 1.6, n = 69, total = 176). The number of trainees of each type were similar, on average, to those involved in CHRP projects identified in the previous evaluation.

There were some observable differences between Recipients based on their research granting agency affiliation, according to survey results. Recipient NPIs surveyed identified as CIHR-affiliated researchers had a higher average number of researchers involved in their grants than Recipients who identified as NSERC-affiliated researchers (CIHR: M = 4.2, SD = 3.56, n = 25; versus NSERC: M = 2.85, SD = 1.66, n = 27); whereas, Recipient NPIs who identified as NSERC-affiliated researchers had higher average numbers of research assistants (NSERC: M = 2.78, SD = 4.21, n = 18; versus CIHR: M = 1.45, SD = 0.6, n = 20) and research technicians (NSERC: M = 3.85, SD = 10.89, n = 20; versus CIHR: M = 1.45, SD = 0.76, n = 20), per grant.

In terms of interdisciplinary collaboration among Trainees, survey results indicated that approximately half of NSE and health Trainees interacted frequently or very frequently with trainees in the other discipline (NSE Trainees: 51% out of 107; health Trainees: 53% out of 51). NSERC-affiliated Recipient NPIs’ grants involved more Canadian (NSERC: 345 HQP total, M = 10.8, SD = 11.2, n = 51 grants; CIHR: 202 HQP total, M = 9.6, SD = 6.0, n = 38 grants) and international HQP (NSERC: 112 total, M = 3.9, SD = 2.9, n = 35 grants; CIHR: 49 total, M = 3.0, SD = 3.2, n = 17) than did CIHR Recipient NPIs’ grants.

Interviewed Recipients said they benefited from working across disciplines on IDR teams, which enabled them to gain a broader understanding of their projects’ goals, challenges, and potential utility by moving beyond their knowledge areas and comfort zones. Program staff and senior managers that were interviewed indicated that the interdisciplinary aspects of the CHRP program have led to the development of additional research capacity, and UDs reported that the program enabled their institutions to build capacity by supporting IDR and its knowledge translation.

However, program staff suggested that the amount of capacity increased by a single CHRP grant may be marginal given that 23% of researchers have received multiple CHRP grants (administrative data analyses) and the program may be supporting the same researchers and research teams over time. Of those interview respondents who felt able to assess whether the program allowed HQP to acquire new skills and knowledge, almost all indicated that students and recipients benefited. Program staff reiterated that training HQP was a requirement of the CHRP program and noted that some NPIs used a large proportion of their grant to pay the salaries of HQP.

CHRP-funded research projects have improved trainee skill development

Almost all surveyed Trainees felt that participating in a CHRP-funded project enabled them to expand and improve their research, analytical, technical, and soft skills. They saw the greatest improvement in their research and technical skills as a result of their involvement in CHRP (M = 4.53 out of 5, SD = 0.81, n = 156, with 89% identifying noticeable or significant improvement in this area).  Two-thirds of surveyed Trainees also felt they improved in professional skills (M = 4.16 out of 5, SD = 0.68; 68% saw noticeable or significant improvement) and interdisciplinary research skills for sectors outside academia (M = 3.88 out of 5, SD = 1.17; 67% saw noticeable or significant improvement). A slightly greater proportion of NSE Trainees surveyed reported that they had improved in most types of skills, particularly interdisciplinary skills, compared to Health Trainees (see Table 8: Trainee Skill Development by Research Discipline). The stage of research that trainees were most frequently involved in was the analysis of research results, according to the Recipients (NPIs) and Trainees that were surveyed (NPIs: 100% out of 91; Trainees: 94% out of 159).

Most interviewed Trainees said that their training exceeded their expectations due to its diversity and comprehensiveness, and many were impressed with the quality of their training experience. Interviewed Trainees also identified that exposure to new areas, such as data mining and analysis, machine learning, imaging, modeling, tissue engineering, biology, nanomedicine, and computer science expanded their perspective, knowledge, and expertise. They also had an opportunity to use their newly acquired skills in various practical situations, which they felt complemented, rather than duplicated, their academic training. The following serve as examples:

The previous evaluation found that trainees improved in a wide range of technical and transferable skills, based on interviews with Recipients and HQP. However, a direct comparison of trainee skill development cannot be made as trainee survey results were not included in the previous evaluation.

Trainees were involved in research collaborations, some of which were interdisciplinary

The frequency of interactions between Trainees and their colleagues on the project team varied substantially. Project teams met as often as bi-weekly and as infrequently as once annually, according to interview respondents. Survey data found that most Trainees met at least weekly with their project NPI (88% out of 155) and other HQP at their university (77% out of 159). According to survey data, approximately one third of Trainees in both health and NSE disciplines interacted frequently or very frequently with researchers in their respective discipline who were outside of their project team (NSE Trainees: 37% out of 103; health Trainees: 39% out of 51). Most Trainees interacted frequently or very frequently with other HQP within their own disciplines (NSE Trainees: 81% out of 107; Health Trainees: 71% out of 51).

As one might expect, the survey results found that almost all Trainees interacted most frequently with researchers at their own universities (99% out of 156). External collaborations were reportedly less frequently: less than half of surveyed Trainees interacted with researchers at other universities in Canada (42%) or organizations outside Canada (36%). Still, some interviewed Trainees reported that their collaborations with co-applicants, partners, and other HQP were international, as well as national, in scope. Representatives of health care organizations (e.g., hospitals, medical clinics), engineering and scientific firms, as well as private sector and civic organizations, figured prominently in these reported interactions.

In terms of involvement in research projects, Trainees provided research findings (e.g., documents and on-site presentations) to colleagues, academics, clients, industry associations, governments, and the public-at-large, and some Trainees contributed articles to peer-reviewed journals. In some instances, a few individuals supervised other trainees, produced and scaled-up technology, and helped in launching new businesses.

CHRP-funded research projects have had a positive impact on trainee employment

Surveyed Trainees felt that their participation in the CHRP project was moderately to extremely useful in launching their professional career (M = 3.78 out of 5, SD = 0.92, with 66% out of 99) rating this participation as very or extremely useful). Approximately one quarter of Trainees secured full-time employment in an area related to their field of study (23% out of 151), and some were offered jobs and/or consulting opportunities (19.3%) as a direct result of their involvement in the project. Surveyed NPIs indicated that HQP involved in the funded projects were most frequently either hired by industry/private sector (total of 139) or still in academic training (total of 131). Two-thirds of surveyed Trainees (65% out of 152) said they were currently still in academic training. NSERC-affiliated NPIs reported that more of their trainees had been hired by the partner involved in the CHRP project (NSERC: 21, CIHR: 3) and by industry/private sector than CIHR-affiliated NPIs (NSERC: 100, CIHR: 39, respectively).

Approximately two thirds of surveyed Trainees reported that they were working full time (62% out of 152), and were working mainly in the academic sector (70%), two-thirds of whom were in research assistant or postdoctoral fellow positions. Most Trainees that were interviewed reported that they had secured positions in industry as a direct result of the CHRP grant and felt that the skills, knowledge and experience they acquired would increase their value to future employers, business associates and the Canadian public.

Most interviewed Recipients thought that CHRP-funded projects had a favourable impact on PhD students and postdoctoral trainees. They felt that the experience gave trainees a broader understanding of their work, and a competitive edge in terms of finding future employment. Consistent with survey results, interviewed Recipients reported that some students earned post-graduate degrees based on their work in CHRP-funded projects, and several students obtained employment in academia, industry, and/or the health sector at the end of their involvement in CHRP-funded projects.

KTU involvement in CHRP-funded projects varied

Another key objective of the CHRP program is the translation of research results to KTUs and other stakeholders. As outlined in the Program Description, prior to 2012 the program encouraged the involvement of partners and KTUs and then in 2012 this became a requirement. More specifically, the program requires the translation of the research results to KTUs and related stakeholders outside the academic or training environment. As per the funding opportunity, the proposed research projects must have a strong focus on knowledge translation and lead to health and economic benefits for Canadians, more effective health services and products, and/or a strengthened health care system. KTU organizations should be meaningfully engaged throughout the research process, as appropriate, to inform research planning and design.

More than three quarters of Recipients and Applicants included a KTU in their application (or intended to include, had they continued with the project - NPIs: 81% out of 98 included; Applicants who continued with the project: 88% out of 16 included; Applicants who did not continue with the project: 77% out of 44 intended to include). Recall that the period covered by the evaluation was from 2009-10 to 2017-18; therefore, it is not surprising that the proportion was not 100% given that the requirement did not come into effect until 2012. Nonetheless, the proportion of grants that included KTUs or partners was much higher than what was reported in the previous evaluation, in which 40% of 116 surveyed Recipients stated that they included partners in their funded projects (only 16% at the application stage). A smaller proportion of Co-applicants (Recipients and Applicants) reported including a KTU in their application (or intended to include, had they continued with the project - Recipients: 48% out of 95 included; Applicants: 55% out of 66 included, 64% out of 95 intended to include). While it is not entirely clear why NPIs and Co-applicants differ in their reported inclusion of KTUs on CHRP grant applications, this may reflect a difference in the timeline of grant funding between these two populations. That is, surveyed Co-applicants may have more frequently been recipients of grants which were received prior to the 2012 requirement to include a KTU, whereas surveyed NPIs may more frequently represent CHRP grant recipients during the period after this requirement was introduced.

An even smaller proportion of both Recipients and Applicants included other partners beyond KTUs on their applications. These other partners may have either been an additional (i.e., secondary) partner beyond the primary KTU, or were identified as such by researchers who applied prior to the 2012 KTU requirement and were thus unfamiliar with the term KTU. NPIs identified that a key benefit of including KTUs and other partners was increased opportunities for the translation of research results, while Partners felt that the main value added by the collaboration with the researchers was academic expertise (research/methods) provided by the researchers.

According to Recipients surveyed, KTUs and other partners were involved less frequently in the research process than co-applicants, and were most often involved in providing materials/facilities, as reported by Partners themselves (58% out of 40), NPIs (KTUs: 58% out of 79; partners: 53% out of 15), and Co-applicants (KTUs: 26% out of 46; partners: 15% out of 13), with the latter reporting a lower proportion of KTUs and other partners involved in this phase. The other phases KTUs were most frequently involved in, according to NPIs, were the development of the research idea/question (51%) and end of grant knowledge translation activities (44%) including knowledge exchange (44%; see Figure 4: Proportion of grants (%) with KTU involvement in research stages). Partners reported similar levels of involvement of their organizations in each of these areas (38-43%). Lower levels of involvement from KTUs in both these areas was reported by Co-applicants (15-24%). Overall, KTU involvement was similar or slightly lower than in the previous evaluation, despite the current program requirement of KTU involvement across stages. Overall, Recipients (NPIs and Co-applicants) indicated that they were satisfied with the collaborations with KTUs and other partners (see  Table 5: Recipients’ (NPIs') satisfaction with collaborations and Table 6: Recipients’ (Co-applicants’) satisfaction with collaborations), although satisfaction ratings were slightly lower than for the collaboration with research co-applicants. Recipients most frequently reported being satisfied with the communication with the KTUs (89% of NPIs and 74% of Co-applicants were satisfied or very satisfied); whereas Recipients reported the least satisfaction with overall productivity in terms of knowledge production in the context of collaboration with the KTUs (69% of NPIs and 60% of Co-applicants were satisfied or very satisfied). Partners (including KTUs) reported that they were also satisfied with the collaboration with the research team (see Table 7: Partners’ satisfaction with collaboration). Partners also reported greatest satisfaction with communication with the research team (79% satisfied or very satisfied) and least satisfaction with overall productivity in terms of knowledge translation (64% satisfied or very satisfied).

Collaborations with KTUs and other partners were perceived as being moderately to greatly successful in terms of advancing the research project, as rated by surveyed Recipients, both NPIs (KTUs: M = 3.53 out of 5, SD = 0.95, n = 72; other partners: M = 3.79, SD = 0.89, n = 14) and Co-applicants (KTUs: M = 3.48 out of 5, SD = 1.03, n = 33; other partners: M = 3.45 out of 5, SD = 1.29, n = 14). These collaborations were rated as slightly less successful than the collaborations with co-applicants. Similarly, Partners (including KTUs) rated their collaboration with the research team as being moderately successful (M = 3.3 out of 5, SD = 1.02, n = 38). Just over half (56% out of 88) of Recipients (NPIs) surveyed indicated that networking and collaborative relationships with KTUs had been established or improved as a result of the CHRP project.

The challenges most frequently associated with the collaboration with KTUs identified by surveyed Recipients (NPIs and Co-applicants), were additional administrative burden (NPIs: 60% out of 47; Co-applicants: 38% out of 21) and a lack of engagement by the KTUs (NPIs: 26%; Co-applicants: 43%). Partners themselves (including KTUs) also identified administrative burden as the greatest challenge in working with the research team (46% out of 22). Some Recipients suggested that the administrative requirement of having a KTU included on the application and involved throughout the research process was a hindrance, and suggested the program should either reduce or facilitate the requirement for this partnership. This was a finding replicated from the previous evaluation, wherein researchers had expressed similar concerns shortly after the requirement to include a KTU/partner was introduced, and had recommended that this requirement be relaxed.

The extent to which collaborations were, in fact, genuine partnerships was questioned by PRC Chairs, who indicated that a KTU could be included in the grant application merely to satisfy the program’s eligibility requirements. The existence of superficial KTUs in CHRP projects was confirmed by one interviewee who admitted to having no real involvement in a CHRP project, despite the fact that the grant application outlined his roles and responsibilities as a KTU. This limited engagement on the part of the KTUs was further evidenced by the difficulty recruiting KTUs and partners as key informants or survey respondents, and several that did complete the survey indicated a similar lack of involvement in the project despite being named as the KTU or partner on the application. One of the Chairs commented that PRCs require a more effective mechanism to validate KTUs, and to ensure that they fulfill the commitments made in the grant application. Thus, the addition of the KTU requirement for CHRP projects did not appear to mitigate the challenges related to KTU/partner involvement in the research process identified in the previous evaluation.

With the varied level of involvement of KTUs in CHRP projects, there was no evidence of an increase in KTU involvement relative to the previous evaluation, even though the requirement to include a KTU on the project application and throughout the research process was introduced in 2012 with the intention to increase partner engagement. According to key informant interviews, some KTUs consistently focused on developing application(s) and addressing end-user requirements, but the ability of KTUs to develop and deploy new technology also varied by research project.

Use of research results by KTUs has been moderate, and is expected to occur more commonly in the future

Although most Recipients that were surveyed identified that an increase in opportunities for the translation of research results was a key benefit of including KTUs and other partners in the research process (NPIs: 82% out of 67 for KTUs, 86% out of 14 for other partners; Co-applicants: 91% out of 31 for KTUs), Recipients felt the results of CHRP-funded research was only being used by KTUs to a moderate extent (NPIs: M = 2.73 out of 5, SD = 1.03, n = 71; Co-applicants: M = 2.47 out of 5, SD = 1.25, n = 36). Recipients indicated that their collaboration with the KTUs only contributed to the project’s outcomes to a moderate extent (NPIs: M = 3.43 out of 5, SD = 1.07, n = 70; Co-applicants: M = 3.27 out of 5, SD = 1.05, n = 30). Similarly, surveyed Partners themselves (including KTUs) indicated that their collaboration with the research team only contributed to the project outcomes to a moderate extent (M = 2.75 out of 5, SD = 0.94, n = 36) and, as previously mentioned, both Recipients and Partners reported lower levels of satisfaction with overall productivity in terms of knowledge production than with all other elements of the research collaboration (69% of NPIs, 60% of Co-applicants, and 64% of Partners were satisfied or very satisfied; see Table 7: Partners’ satisfaction with collaboration). However, it should be noted that the majority of Recipients expected that results will be used by KTUs/partners in the future. Specifically, Recipients (NPIs) expected that the majority (87% out of 70) of KTUs would use the research results in the future.

Approximately half of Recipient NPIs surveyed planned to continue their collaboration with KTUs and other partners on the same project (KTUs: 52% out of 71; partners: 50% out of 14). Close to two thirds of Recipient Co-applicants planned to continue collaboration with KTUs on the same project, although collaborations were planned for only about one quarter of their other partners (KTUs: 63% out of 32; partners: 27% out of 11). Similarly, half of Applicant Co-applicants who continued with the project were also continuing their collaboration with KTUs on the same project (NPIs: 50% out of 24). While Applicant NPIs were less frequently continuing this collaboration with KTUs (18%, out of 11), they more frequently reported being involved with the KTUs in a formal collaboration on a new project (36%) or an informal collaboration (46%) than did Recipient NPIs (14% and 30%, respectively; although the n was low for Applicant NPIs). Just over half of Partners reported continuing the collaboration with the research team on the same project (55% out of 38).

Given that the evaluation period covers 2009 to 2018, during which time the requirement to include a KTU in the application and throughout the CHRP-funded research process was introduced (in 2012), a certain amount of variability in KTU involvement over that period might be expected. However, despite the KTU requirement, KTU and partner involvement in the CHRP program has remained moderate at best, indicating that the requirement to include a KTU in the CHRP projects has not had the intended effect. Given this variability in reported KTU involvement, it is therefore not surprising that KTU use of research results has been moderate at best and that it is more commonly expected that these research results will be used in the future. Thus, the evaluation evidence suggests that the objective of knowledge translation to KTUs involved in the CHRP project is not being fully met.

CHRP-funded projects have advanced knowledge and produced some innovations and technologies

Another objective of the CHRP program is to advance IDR leading to knowledge and technologies with the potential to benefit Canada by improving the Canadian healthcare system and/or services and, where appropriate, lead to economic opportunities in Canada. Although there are no defined benchmarks or expected thresholds for outputs specified in CHRP’s program theory, the evaluation found evidence that CHRP-funded projects have resulted in advances in knowledge as well as some innovations, technologies, and efficiencies in the health care field.

CHRP-funded research contributed to advancing knowledge, mainly within academia, as well as training of HQP, and has resulted in some innovations, technologies, and efficiencies in the health care field. Among Recipient NPIs surveyed (n = 90), the most frequently identified project outcomes were knowledge production, both within academia (90% ) and beyond academia (62%), and useful opportunities for HQP (72%). See Figure 2: Proportion (%) of grants resulting in outcomes. Moreover, the most frequent outputs resulting from these projects were peer reviewed journal articles and presentations at international conferences (Figure 3: Proportion (%) of grants producing selected outputs). According to end of grant reports for NSERC (2010-12) and CIHR (2012-13), there were, on average, 6 published journal articles and a total of 203 invited presentations and 325 “other” presentations per grant.

With respect to innovations, surveyed Recipients (NPIs) were asked whether they had developed or improved a variety of health and economic outcomes. NPIs reported that 74% of grants (out of 90) produced at least one of the following: developing (54%) or improving (21%) a product/service; developing (32%) or improving (9%) a process/treatment; or contributing to policies, guidelines, or regulations (6%). The previous evaluation found that 53% of grants had reportedly developed a new or improved process, technology, or product, indicating a possible increase in these outputs during the current evaluation period. Interestingly, a greater proportion of CIHR-affiliated Recipient NPIs surveyed identified that their projects developed a new product or service (68% out of 37) compared to NSERC-affiliated Recipient NPIs (45% out of 62).

Surveyed Recipient NPIs (n = 89) reported that their projects’ objectives were met to a great or very great extent (79%). Two-thirds of NPIs (67% out of 79) reported their projects’ achievements fully met (49%) or exceeded (18%) their expectations (NSERC end of grant data, 2010-2012). Further, NPIs reported that the impact the project had on their research influenced its direction to more industrial relevant topics (64% out of 74), and opened up new opportunities for research beyond the original objectives (93% out of 81). According to the survey data, approximately three-quarters of Recipients and Applicants (NPIs and Co-applicants) stated that there was a demonstrated market need for the product, service, or process that was the focus of their CHRP application (NPIs: 78% out of 101 and 86% out of 71, respectively; Co-applicants: 74% out of 107, 67% out of 202,respectively).

Scale-up and commercialization of CHRP-funded research has been moderate

According to CIHR end of grant data, some CHRP grants had produced patents (28% advanced; 20% newly developed, out of 43). Survey data indicated that 41% of grants produced filed and/or patented results (12% produced filed patents, 36% produced granted patents; Figure 3: Proportion (%) of grants producing selected outputs). According to the previous evaluation, in 2014 one-quarter of CHRP grants had resulted in filed patents (23%) and granted patents (25%). Based on NSERC end of grant data, CHRP projects resulted in a total of 121 filed patents and 13 issued patents, in Canada, US, and other countries. A 2015 evaluation of CIHR commercialization programs using end of grant data reported that the CIHR Proof of Principle Grant program produced more patents (70%), and several other programs produced a comparable number of patents, including CIHR’s open funding program (Open Operating Grant Program: 48%), and the Regenerative Medicine and Nanomedicine Initiative (46%). A 2018 evaluation of NSERC’s I2I program found that that 90% of researchers that received Phase I or Phase II program funding have filed for or secured patent protection as a result of their project (based on administrative data). In NSERC’s 2017 evaluation of the Centres of Excellence for Commercialization in Research (CERC) program, companies supported by the Centres were surveyed: 26% reported a patent had been filed and 11% reported a patent had been issued as a result of this support. Note that these NSERC programs are not focused on health research, and are thus not directly comparable to CHRP. However, taken together these findings indicate that other CIHR and NSERC programs, whether focused on commercialization outcomes or not, have had equal or greater success in producing commercializable outputs like patents.

While surveyed Recipient NPIs rated their projects as demonstrating the potential for scale-up to a great extent (M = 3.71 out of 5), they also reported that projects had only been scaled up to a moderate extent relative to their potential (M = 2.88 out of 5). Results on the scale up and potential scale up on projects from surveyed Co-applicants differed slightly compared to NPIs, and were slightly lower. More specifically, Co-applicants rated their projects as demonstrating the potential for scale-up to a moderate extent (M = 3.16 out of 5, SD = 1.29, n = 64) and as having been scaled up relative to their potential to a slight extent (M = 2.29 out of 5, SD = 1.16, n = 59). NPIs also reported that one-quarter (23% out of 43) of grants had the potential to produce a patent in the future (CIHR end of grant data, 2012-13).

While some interviewed Recipients reported that their projects’ KTUs translated, applied, and commercialized the results of the research project very effectively (including spin-off companies, patents, and licenses), three out of eight of the Recipients reported that their KTUs applied and/or commercialized the results ineffectively. According to the interviews, this occurred because the KTUs thought that applying the research results would be prohibitively expensive, the focus changed and collaborating with the research team was no longer feasible; or the KTU went out of business. Some interviewed Recipients suggested that involving a KTU in a CHRP-funded project with a high TRL was most beneficial, since there is a greater likelihood of successfully commercializing the product or service in development.

Some Partners felt that CHRP results led to innovative solutions, but most felt they did not contribute to increased productivity or competitiveness

Almost half of surveyed Partners believed the use of research results have led to innovativeness (45% out of 31) or maintained and/or improved upon the culture of innovation (49% out of 33) within their organizations. This includes increased collaboration, knowledge or technology transfer, advances in theory, methods or analysis, or product design. However, almost all surveyed Partners indicated the research results did not impact their organization’s productivity (91% out of 34), and two-thirds (67% out of 33) reported that it had no impact on their organization’s competitiveness.

With regard to evidence from the key informant interviews, a number of respondent groups were either unaware of, or lacked the information to speak to health-related innovations. In particular, the three members of program staff were unaware of any innovations or health-related efficiencies resulting from CHRP-funded research. The three members of senior management could not speak authoritatively about specific healthcare-related innovations or efficiencies resulting from CHRP-supported research, and the two ADs were relatively uninformed about their particular accomplishments.

Of the respondent groups in a position to respond to this question, one-half of those interviewed indicated that CHRP-supported research led, or will lead, to innovations that improve health care in significant ways. Recipients cited specific accomplishments relating to their research including advancements in treatments, processes, and materials that have created health benefits. For instance, one project contributed to the identification of molecules for psoriasis treatment and another contributed to new surface materials to reduce adverse effects of surgical tools.

There is some evidence that CHRP-funded projects have generated economic and health benefits for Canadians

One of the objectives of the CHRP program is to advance interdisciplinary research leading to knowledge and technologies with the potential to benefit Canada by improving the Canadian healthcare system and/or services, and, where appropriate, lead to economic opportunities in Canada. There is some evidence of broader economic and health benefits for Canadians resulting from CHRP-funded research; however, these are longer term outcomes that would be expected to take a number of years to be realized, and evidence of their impact is limited. It is possible that more of these impacts may result in the future and thus may not have been captured by this evaluation.

In terms of economic benefits, CIHR end of grant data (2012-13) indicated that one quarter of CHRP grants (23% out of 43) resulted in spin-off companies, and close to one half (44%) had the potential to produce a spin-off company in the future. However, survey data indicated that fewer grants resulted in spin-off companies (12% out of 83). Product licenses resulted less frequently from grants: 16% (out of 43) according to CIHR end of grant data, and 10% (out of 83) according to survey data. Very few grants reportedly resulted in direct cost savings, based on CIHR end of grant data (9% out of 43). However, a few of the recipients interviewed reported economic benefits, such as the creation of new companies and patents. 

In terms of health benefits, outcomes from CHRP-funded research are variable. Although most Recipients (NPIs: 88% out of 43) indicated that they produced a tool, technique, instrument, or procedure, according to CIHR end of grant data which may lead to health benefits, other health benefits, such as professional practices, policies or guidelines, were reported very infrequently (less than 15% of NPIs, ranging from 0-14%). Few comparable health benefits were reported in the survey results, with policies, guidelines/standards, and programs produced by 2% of grants or less (0-2% out of 83). A greater proportion of Recipients (NPIs) expected economic and health benefits to result from their CHRP projects in the future (21-54% out of 43), based on CIHR end of grant data. Interviewed Recipients reported that their CHRP-funded project generated academic and/or social benefits; greater interest in their research, and/or health benefits (e.g., improved diagnoses or treatments).

A couple of UDs interviewed reported about health benefits arising out of CHRP-funded research, including:

It should be acknowledged that longer term outcomes and impacts of health research can take upwards of 17 years, depending on intended outcomes (Balas & Boren, 2000; Grant, Green, & Mason, 2003; Wratschko, 2009; Morris, Wooding, & Grant, 2011), and IDR research impact can take longer (Van Noorden, 2015).Footnote 6

It is important to note that the validation of project outcomes was challenging due to the scope and approach in the current evaluations, as well as data limitations. For example, the lack of full impact study, availability and completeness of end of grant data, and the lack of consistency between CIHR and NSERC’s reports, restricted the ability to assess outputs and outcomes resulting from the projects.

Results may be limited by the three-year grant duration

Although interviewees indicated that CHRP-funded research has produced some processes and technologies which benefit the Canadian health sector, some interviewed UDs and interviewed or surveyed Recipients felt that the CHRP program’s three-year timeline for funding was limiting, as it is unrealistic to expect a collaboration to produce tangible results in such a short period of time. A theme that came out of interview responses was that the collaborative process can delay a project’s progress, at least initially, since researchers may find it challenging to adapt to their colleagues’ different terminologies, perspectives, work ethic, and approaches to issues. These findings are supported by research that suggests that IDR takes longer to have an impact. For example, a 2015 analysis by Van Noorden of Web of Science research papers found that after the first three years, papers associated with multiple disciplines have fewer citations than the norm; however, they gain more over a period of 13 years than single discipline papers.

Most funded projects were at the lower levels of technology readiness, with almost two-thirds reporting an average increase of 2 levels

As described earlier in the Relevance section findings, the TRL scale was utilized as a means of measuring the relative stage of technologies and innovations produced via CHRP projects.  Although there was no explicit TRL-related requirement or expectation outlined in CHRP program objectives, neither in relation to a particular start or end stage of the projects nor an expected relative increase in TRL, this is a commonly used scale for quantifying the development stages of technology, from the early concept through to commercialization stage. As such, the application of the TRL scale is used as one metric to measure program outcomes; however, no direct conclusions should be drawn about the program performance based on TRL stage alone.

While the CHRP program funds projects across the continuum from basic/exploratory research to market-ready technology, commercialization occur is more likely to occur the closer the project is to market-ready technology or the later stages of the technology readiness scale. Therefore, the objectives of the program to support research across the levels of technology readiness, and also to produce commercializable outputs, may be contradictory.

Based on available evidence, the majority of research projects funded by the CHRP program are not technology ready. The majority of funded projects started at the lower end of the TRL continuum.Footnote 7 According to surveyed Recipients (NPIs), TRLs at the start and the end of the funded projects ranged from 1 to 9, with only a few (4% out of 90) that could not be rated on the TRL scale. Recipients most frequently identified their projects at the start as being at TRL 3 (34%), with more than two-thirds (69%) of projects identified as starting at the earliest stages of TRL 1-3. Almost all Recipients (NPIs: 91% out of 86; Co-applicants: 72% out of 76) and Partners (75% out of 36) expected an increase in TRL.

Although projects started at the lower end of the TRL continuum, over half of those projects demonstrated increases in technology readiness. Two-thirds of surveyed NPIs (64% out of 90) reported an increase in TRL, with an average increase of 2.2 levels, while more than half of Co-applicants (59% out of 78) saw an increase in technology readiness (of at least one TRL) for their product/service. Interestingly, a similar proportion of Applicants (NPIs) that continued with the project saw an increase in TRL (58% out of 12). At the end of their projects, final TRLs for Recipients remained in the TRL 1 to TRL 9 range, with projects most frequently identified as being at TRL 4 (NPIs: 23% out of 91; Co-applicants: 16.5% out of 79). Recipient NPIs reported that at the end of their projects, more than half (52%) of projects were at a TRL between 4 and 6; whereas only about one third (37%) of Co-applicants’ projects were at these levels at the end of their projects, reflecting a more even distribution across the TRL levels. The factors most frequently cited by Recipients (NPIs and Co-applicants) as influencing technology readiness were the nature of the collaboration, funding, and existing or available technology, materials or facilities. Responses among Recipients interviewed varied, with most Recipients reporting figures in the TRL 1-2 range at the projects’ inception. These interviewees also said that the TRLs increased by the end of the project.

It is difficult to interpret the value of an increase in TRL for the following reasons. First, there is no accepted standard for expected increase in the TRL of a technology or innovation within the time frame specific to the CHRP granting period. Second, some interpretations of the TRL scale have conceptualized the scale in such a way that certain levels may be combined to represent phases of technology readiness (e.g., TRL 1-3 is “pre-concept refinement” and TRL 4 is “concept refinement”), thereby rendering increases at certain stages to be unequal to those at others (e.g., Olechowski, Eppinger, & Joglekar, 2015). Third, as the TRL metric was not used in the previous evaluation, increases in TRL cannot be compared within the CHRP program over time.

There was a perception among key informants that PRC Chairs favour funding projects within a higher range (TRL 5-9), with those in the lower range (TRL 1-3) perceived to be too risky and more geared towards other funding programs that would better meet their needs. However, the funding profile would suggest that projects at the lower end of the TRL continuum are more frequently funded. According to the four PRC Chairs interviewed, the technological readiness of projects funded through the CHRP program includes all points along the TRL continuum – extending from basic research (TRL 1) to imminent commercialization (TRL 9).

It is not clear whether the CHRP program is being delivered in a cost-efficient manner

The ratio of program administrative costs to total program expenditures and the proportion of a program’s budget that is expended both speak to how efficiently a program is being run. However, the cost efficiency analysis in the present evaluation compares only direct administrative costsFootnote 8 of the CHRP Program (from both CIHR and NSERC) against total program investments for the fiscal years 2015-16 through 2017-18.Footnote 9

This focus only on direct costs was due to issues validating CIHR’s salary data prior to 2015, and the lack of indirect costs reported by both CIHR and NSERC.Footnote 10 Further, CIHR’s annual direct costs, which were relatively consistent between 2009-10 and 2014-15 ($46-60K per year), nearly doubled for the years 2014-15 to 2017-18. The reason for this increase in direct costs is not clear.

The evaluation found the ratio of direct administrative costs to total program expenditures to be very low, remaining between 0.60% and 0.82% since 2009-10 (see Table 1: CHRP Program Expenditures for CIHR and NSERC 2015-16 to 2017-18). Given that the current evaluation focuses only on direct administrative costs it is not surprising that the ratio is low; however, even using a different method of calculation, the previous evaluation also found direct costs of the CHRP program to be very low (ranging from 1.5-2.3%).

Table 1. CHRP Program Expenditures for CIHR and NSERC 2015-16 to 2017-18
2015-16 2016-17 2017-18
CIHR Direct Administrative Costs $91,829 $93,520 $85,397
NSERC Direct Administrative Costs $68,035 $70,946 $61,141
NSERC Total program expenditures $9,797,852 $9,724,073 $9,957,266
CHIR Total program expenditures $10,997,544 $10,224,0683 $10,163,417
Total Direct Adminstrative Costs $159,864 $164,466 $146,538
Total program expenditures $20,795,396 $19,948,141 $20,120,683
Direct Costs as % of Total Program Expenditures 0.77% 0.82% 0.73%

Note: Figures represent expenditures within each fiscal year.

Source: CIHR Financial Planning and Advisory Services; NSERC Research Partnerships

A different approach was taken for calculating costs in the previous evaluation, which both direct and indirect costs and found that the CHRP Program was being delivered efficiently (with an average operating ratio of 5.3 cents for each dollar of grant funds awarded).Footnote 11 The availability of financial data for the program has been an ongoing challenge because the estimate of administrative costs for the CHRP program was readily available for only five of the nine years under review from both councils (fiscal years 2004–05 to 2008–09). Due to these data limitations, it is not possible to draw conclusions about the cost efficiency of the CHRP program.

Stakeholder perspectives on the administrative efficiency of the CHRP program varied across interviewees. All three senior managers believed that the Tri-Agencies operate efficiently, and assumed that this would also be the case with the CHRP program. Program staff from two of the granting agencies were unsure about the program’s efficiency because they lacked relevant information. However, program staff from the remaining agency claimed that the CHRP program has high operational costs. They explained that collaborative, interdisciplinary, multi-agency programs tend to have large, diverse peer review committees, which are costly to operate. All four PRC Chairs felt that the peer review process has operated very efficiently.

Conclusions and Recommendations

Conclusions

Relevance

There is an ongoing need for the funding of collaborative research involving NSE and health sciences

The evaluation concludes that there is an ongoing need to fund interdisciplinary research (IDR) that fosters collaboration between health and natural sciences and engineering (NSE) researchers, and that facilitates the translation and commercialization of research to improve the Canadian health system and its services. However, it is not clear that the CHRP program, as currently designed, is the most effective funding mechanism to achieve these needs. Broadly, the program is well aligned with key Federal Government priorities (Budget 2018, Budget 2019, Canada’s Vision for Science, Fundamental Science Review) as well as the mandates of the Tri-Agencies. The evaluation found that the CHRP program is distinct from and complements other Federal programs. Although objectives related to IDR and/or partnerships to foster knowledge translation are not unique to CHRP, unlike other programs, this program has a broad scope, funding projects across the continuum from basic/exploratory research to market-ready technology. In addition, it funds interdisciplinary research that integrates health sciences and natural sciences and/or engineering, facilitates collaborations between researchers as well as knowledge technology users, and emphasizes the need for knowledge translation. However, with respect to knowledge translation, other funding mechanisms, including CIHR’s open funding programs, have demonstrated equal or greater success in facilitating the translation, application, and/or commercialization of scalable new technology.

There is no consensus on whether the program should focus on funding research at a specific point on the continuum from basic/exploratory research to market-ready technology or specific Technology Readiness Level. However, almost all interviewees acknowledged the importance of funding projects in the low to mid-range (applied research and development to product/prototype testing) and a few felt that those at more advanced levels should get more funding.

There is clear uptake of this program by researchers, with an average success rate of 11% (from LOI stage applications), as well as the majority of surveyed Recipients reporting that they would not have continued without CHRP funding. However, approximately one quarter have received multiple CHRP grants, and while the number of LOIs increased from 2009 to 2012, the number of LOIs has been decreasing since it peaked in 2012.

Performance

CHRP-funded research has continued to facilitate new and existing collaborations between health and NSE researchers

Overall, the evaluation found that the CHRP program has met some of its objectives. The program continues to be effective in facilitating collaborations between CIHR and NSERC researchers, including both new and existing relationships between co-applicants. The integration of health and NSE expertise has been necessary to complete the CHRP-funded research.  Beyond the requirements of the program to involve NSE and health researchers on all grants, Recipients were highly satisfied with the research collaborations, and felt that these collaborations had been effective at advancing the projects and had led to research that would not otherwise have been conducted.

The CHRP program has effectively facilitated capacity building

The evaluation found that the CHRP program has been effective in contributing to building interdisciplinary capacity, providing interdisciplinary research and training opportunities for both researchers and trainees. The CHRP program has effectively enabled students to develop the skills and knowledge required to find employment and other revenue-generating opportunities related to their fields of expertise. Trainees reported high levels of satisfaction with the training they had received, noting that they gained exposure to new areas of research and had improved their research, analytical, technical, and professional skills. In addition, trainees were directly involved in research collaborations, some of which were interdisciplinary. They reported that the CHRP program had been useful in launching their professional career, with approximately one quarter of Trainees securing full-time employment in an area related to their field of study, and almost 20% being offered jobs and/or consulting opportunities as a direct result of their involvement in the CHRP project.

There is some evidence of innovations, technologies, and health systems/services resulting from the CHRP program; although use of research results and impacts are expected to occur more frequently in the future

The evaluation found some evidence that CHRP-supported research has resulted in innovations, technologies, and/or health systems and services. The majority of grants reported that they developed or improved a product/service or process/treatment, or contributed to policies, guidelines, or regulations. Some grants have resulted in patents. There is also some evidence that the CHRP program has resulted in innovations and efficiencies in the health care field. Interviewed Recipients reported that their CHRP-funded project generated academic and/or social benefits; greater interest in their research, and/or health benefits (e.g., improved diagnoses or treatments).

While the CHRP program has resulted in some innovations, the evaluation found that its knowledge technology users (KTUs) facilitated the translation, application and/or commercialization of scalable new technology to a moderate extent. Surveyed Recipients reported that the scale-up and use of research results by KTUs was more likely to occur in the future. KTU involvement varied among CHRP projects. Despite formalizing the KTU involvement in the in 2012 (along with a requirement to include a KTU in all stages of the research process where applicable), evaluation findings indicate that the expected increase in KTU engagement and use of research results (as noted in the previous evaluation) was not observed. In addition, Recipients felt that KTU involvement in CHRP funded research advanced the project from a moderate to a great extent.

There is limited evidence that CHRP-funded research has resulted in economic and health benefits for Canadians. For example, less than 25% of grants resulted in economic benefits (e.g., spin-off companies or product licenses), and less than 15% of grants reported health benefits (e.g., professional practice or policies/guidelines). Evaluation evidence shows that these impacts are more likely to occur in the future. In addition, although some Partners/KTUs identified that CHRP project results led to innovative solutions, most felt results did not contribute to increased productivity or competitiveness. The previous evaluation identified that achievement of longer-term outcomes typically does not occur for many years beyond project completion. Longer term outcomes and impacts of health research can take upwards of 17 years (depending on intended outcomes), and there is evidence that IDR research impact takes longer to come to fruition.

Most CHRP projects were at the lower levels of technology readiness; however, increases were observed

Based on available evidence, the majority of research projects funded by the CHRP program are not technology ready. An assessment against the TRL scale shows that 69% of projects started at the lower end of the nine-stage technology readiness continuum and 64% of surveyed Recipients reported an increase in technology readiness, with an average increase of two levels. While the CHRP program funds projects across the continuum from basic/exploratory research to market-ready technology, commercialization occur is more likely to occur the closer the project is to market-ready technology or the later stages of the technology readiness scale. Therefore, the objectives of the program to support research across the levels of technology readiness, and also to produce commercializable outputs, may be contradictory.

Outcomes may be limited by program elements such as the KTU requirement and expectations for translation and commercialization

Taken together, the findings suggest that some design and delivery elements of the program may be limiting the achievement of intended outcomes; in particular, those related to the KTU requirement and the expected translation and commercialization of research results. The three-year funding period of the program, particularly given that the program offers support for research along the full continuum of technology readiness including early stage innovation, presents challenges related to translation of project outputs and the achievement of longer-term outcomes.

Limitations in performance measurement data negatively affect the assessment of program outcomes

Consistent with findings from the previous evaluation, there is a need for the Tri-Agencies to make improvements to the availability and consistency of data collection and management, and to ensure effective ongoing performance measurement. The agencies currently have differing practices and different end of grant reports, which presented challenges to the tracking and identification of study stakeholders, as well as the assessment of outcomes and previous funding history of CHRP-funded researchers. Performance measurement and data limitations restricted the ability to determine whether the translation of knowledge to KTUs and stakeholders is effectively occurring, thereby making it difficult to assess the extent to which this program objective has been met.

Recommendations

The evaluation makes two recommendations to improve the performance of the program.

Recommendation 1
  1. CIHR and NSERC should review the CHRP program objectives and identify the best ways to achieve these objectives, either through redesign of the program or delivery via other funding opportunities.
Recommendation 2
  1. Performance measurement and data availability related to the CHRP program should be strengthened.
    1. CIHR needs to improve the performance measurement of the CHRP program and enhance the way that data is collected related to collaborations and partnerships as well as longer term outcomes (i.e., innovations and health care efficiencies) to better monitor the impact of CHRP funding.
    2. NSERC and CIHR need to establish a means by which to improve the consistency of data collection, data management and data sharing processes related to the CHRP program.

Appendix A: Tables

Table 2. Total Annual Investments in the CHRP Program by CIHR and NSERC (in Millions) 2009-10 to 2017-18
Fiscal Year 2009-10 2010-11 2011-12 2012-13 2013-14 2014-15 2015-16 2016-17 2017-18 Total
CIHR $7.05 $6.96 $8.74 $6.50 $10.39 $11.17 $11.00 $10.22 $10.16 $82.19
NSERC $5.90 $6.85 $6.84 $7.85 $11.69 $9.57 $6.77 $12.97 $10.05 $78.49
Total $12.95 $13.81 $15.58 $14.35 $22.08 $20.74 $17.77 $23.19 $20.21 $160.68

Note: Figures represent expenditures within each fiscal year.

Source: CIHR Financial Planning and Advisory Services; NSERC Research Partnerships
Table 3. CHRP Program Application and Success Rates 2009-2018
Year Letters of Intent Received Full Applications Received Grants Awarded Letters of Intent Success Rates Full Applications Success Rates
2009-10 326 111 31 9.5% 27.9%
2010-11 342 118 34 9.9% 28.8%
2011-12 516 149 37 7.2% 24.8%
2012-13 376 127 40 10.6% 31.5%
2013-14 276 123 36 13.0% 29.3%
2014-15 255 111 35 13.7% 31.5%
2015-16 243 116 34 14.0% 29.3%
2016-17 306 114 32 10.5% 28.1%
2017-18 212 94 30 14.2% 31.9%
Total 2852 1063 309 10.8% 29.1%

Source: CIHR Funding Analytics, NSERC Research Partnerships

Table 4. Technology Readiness Level (TRL) Scale Framework
Technology Readiness Level Description Details
TRL 1 Basic principles observed and reported This is the lowest level of technology readiness. The process of translating scientific research into applied research and development (R&D) begins. Examples might include paper studies of a technology's basic properties.
TRL 2 Technology concept and/or application formulated Invention begins. After observing the basic principles, inventors create practical applications. Applications are speculative, and there may be no proof or detailed analysis to support the assumptions.
TRL 3 Analytical and experimental critical function and/or characteristic proof of concept Active R&D begins. This includes analytical studies and laboratory studies that physically validate the analytical predictions of separate elements of the technology.
TRL 4 Product and/or process validation in laboratory environment Testing of basic technological products and processes, to see if they work.
TRL 5 Product and/or process validation in relevant environment The reliability of product and/or process innovation increases significantly. Integration of basic products and/or processes occurs, to allow for testing in a simulated environment.
TRL 6 Product and/or process prototype demonstration in a relevant environment Prototype testing takes place in a relevant environment. This represents a major step up in a technology's demonstrated readiness. Examples include testing a prototype in a simulated operational environment.
TRL 7 Product and/or process prototype demonstration in an operational environment The prototype is near or at planned operational system and requires demonstration of an actual prototype in an operational environment (e.g., in a vehicle).
TRL 8 Actual product and/or process completed and qualified through test and demonstration Demonstrate that the innovation works in its final form under expected conditions. In most cases, this TRL represents the end of true system development.
TRL 9 Actual product and/or process proven successful Actual application of the product and/or process innovation in its final form or function.

Note: More resources for the TRL Scale can be found via the Government of Canada and the International Organization for Standardization (ISO).

Source: Public Works and Government Services Canada

Table 5. Recipients’ (NPIs’) satisfaction with collaborations with co-applicants, KTUs and other partners
Elements of Collaboration Collaboration with co-applicants (n = 70) Collaboration with KTUs (n = 90) Collaboration with other partners (n = 14)
M (SD) (out of 5) Satisfied or Very Satisfied (%) M (SD) (out of 5) Satisfied or Very Satisfied (%) M (SD) (out of 5) Satisfied or Very Satisfied (%)
Communication 4.48 (0.67) 95.5% 4.15 (0.69) 88.8% 4.21 (1.19) 78.5%
Decision-making 4.43 (0.72) 92.2% 3.94 (0.80) 71.4% 4.14 (0.86) 71.5%
Involvement of co-applicants from diverse disciplines 4.54 (0.67) 95.5% - - 4.14 (0.77) 78.6%
Overall productivity in terms of knowledge translation (e.g., papers, patents, products, services and processes, etc.) 4.33 (0.79) 90.0% 3.87 (0.74) 68.6% 3.71 (1.20) 50.0%
Overall contribution to the project 4.41 (0.79) 91.1% 3.99 (0.83) 78.6% 3.79 (1.19) 57.1%
Table 6. Recipients’ (Co-applicants’) satisfaction with collaborations with co-applicants, KTUs and other partners
Elements of Collaboration Collaboration with co-applicants (n = 76) Collaboration with KTUs (n = 31) Collaboration with other partners (n = 12)
M (SD) (out of 5) Satisfied or Very Satisfied (%) M (SD) (out of 5) Satisfied or Very Satisfied (%) M (SD) (out of 5) Satisfied or Very Satisfied (%)
Communication 4.16 (0.98) 82.9% 3.87 (0.92) 74.2% 3.67 (0.65) 58.3%
Decision-making 4.15 (0.96) 82.4% 3.74 (0.96) 67.8% 3.50 (0.71) 40.0%
Involvement of co-applicants from diverse disciplines 4.32 (0.84) 88.2% - - 3.55 (0.69) 45.5%
Overall productivity in terms of knowledge translation (e.g., papers, patents, products, services and processes, etc.) 4.01 (0.95) 80.0% 3.63 (0.96) 60.0% 3.30 (0.67) 20.0%
Overall contribution to the project 4.17 (0.86) 85.4% 3.67 (1.09) 66.7% 3.70 (0.67) 60.0%
Table 7. Partners’ satisfaction with the research team
Elements of Collaboration M (SD) (out of 5) Satisfied or Very Satisfied
Communication 4.03 (1.05) 78.9%
Decision-making 3.92 (1.11) 71.8%
Overall productivity in terms of knowledge translation (e.g., papers, patents, products, services, and processes, etc.) 3.72 (1.05) 64.1%
Overall contribution to the project 3.85 (1.04) 69.2%
Table 8. Trainee skill development by research discipline
Skill Type NSE Trainees (n = 106) Health Trainees (n = 51)
M (SD) (out of 5) Satisfied or Very Satisfied (%) M (SD) (out of 5) Satisfied or Very Satisfied (%)
Research and technical skills 4.63 (0.71) 92.5% 4.32 (0.98) 80.4%
Interdisciplinary research skills with sectors outside of academia 3.93 (1.12) 70.1% 3.75 (1.28) 58.8%
Professional skills (e.g., communication, teamwork, project management) 4.24 (0.89) 79.5% 3.98 (1.13) 74.5%
Leadership skills 3.75 (1.07) 69.3% 3.34 (1.22) 45.1%
Networking/collaboration skills 3.74 (1.05) 64.1% 3.46 (1.18) 51.0%
Marketing skills 2.38 (1.17) 16.8% 2.20 (1.13) 13.8%
Entrepreneurial/business skills 2.37 (1.19) 17.1% 2.20 (1.41) 17.6%
Knowledge of other sectors outside of academia (e.g., industry, government) 3.41 (1.13) 50.0% 2.77 (1.25) 25.5%
Job-readiness 3.70 (1.19) 60.3% 3.27 (1.32) 47.0%
Knowledge translation 4.09 (0.97) 73.6% 3.80 (1.09) 59.2%

Appendix B: Figures

Figure 1: CHRP Program Logic Model

Figure 1 long description

The logic model identifies the linkages between the activities of the CHRP program and its ultimate outcomes. It delineates the set of activities that make up the program and the sequence of outputs and outcomes that are expected to flow from these activities. The logic model serves as a “roadmap,” connecting activities to the ultimate outcomes, and thus identifies the steps that will demonstrate progress towards expected results. Five levels of performance are delineated in the logic model: activities, outputs, immediate outcomes, intermediate outcomes and ultimate outcomes. It is important to note that the program was not designed to achieve the intermediate outcomes INT.03 and INT.04 until 2012, when the involvement of partners became a funding requirement.

Activities and Outputs

The program activities, which are within the control of NSERC and CIHR staff, include:

  • Information on the program reaches targeted researchers (A.01) – A description of the program is available on the NSERC and CIHR websites.
  • Letters of Intent (LOI) received and reviewed; meritorious candidates invited to submit full proposals (A.02) – interested parties submit a letter of intent that briefly outlines their proposed project. Selected applications that meet CHRP’s criteria are invited to submit full proposals.
  • Applications reviewed by external volunteer reviewers and peer review panel (A.03) – The applications are reviewed by external experts as well as a peer review panel using the program selection criteria.
  • Feedback to applicants (A.04) – Successful and unsuccessful applicants receive feedback from the reviewers on areas of strength and weakness, as well as ways of improving their applications for future submission.
  • Monitoring of CHRP grants (A.05) – Monitoring of the program consists of financial monitoring and measurement of program outcomes through interim and final reports to NSERC/CIHR.

The activities described above enable NSERC and CIHR to produce the program outputs: program material and CHRP awards.

Outcomes

The outcomes for the CHRP program are expected to occur at different points in time. It is important to note that the achievement of program outcomes rely on activities of and decisions made by awardees that are not under the direct control of NSERC and CIHR.

Immediate Outcomes

The immediate outcomes are a direct result of activities relating to the CHRP program and occur during the awardees’ grant term. During this term, awardees conduct interdisciplinary research (IM.01). Through the process of conducting research, relationships are established between NSE and health researchers (IM.02), and plans for transfer of knowledge are in the early stages of implementation (IM.03). At the same time, HQP (e.g., students, research assistants, fellows) receive research training (IM.04).

Intermediate Outcomes

The intermediate outcomes are expected to occur following the conclusion of the CHRP grants. Relationships that develop between NSE and health researchers over the term of the research projects are expected to lead to increased collaboration (IN.01), and this in turn contributes to the sustainability of those relationships (IN.02). The collaboration between NSE and health researchers in conjunction with early plans for knowledge transfer is the key focus of the CHRP program. Collaboration and plans for knowledge transfer will help to ensure research findings that are relevant and useful to industry and/or public policy (IN.03), and increase the use of that knowledge and technology within the research community, government and/or industry (IN.04). The research training received over the term of the grant (IM.04) leads to highly qualified researchers/technicians trained in collaborative and interdisciplinary research (IN.05).

Ultimate Outcome

The ultimate outcome represents the long-term goal the program is attempting to achieve. Changes in policies and practices as a result of the use of CHRP research findings, as well as the increase in numbers of highly qualified researchers/technicians trained in collaborative and interdisciplinary research, will, over time, contribute to increased economic development in health-related areas and increased health benefits for Canadians (UI.01)

Figure 2: Proportion (%) of grants resulting in outcomes

n = 90

Note: Based on NPI Recipient survey responses.

Figure 2 long description
Knowledge Production within Academia 90%
Developed/improved a process, product, service, or treatment 74%
Useful opportunities for HQP 72%
Knowledge Production beyond Academia 62%
Contributed to a policy, guideline or regulation 6%

Figure 3: Proportion (%) of grants producing selected outputs

n = 90

Note: Based on NPI Recipient survey responses.

Figure 3 long description
Peer-reviewed journal articles 86%
Presentations at International conferences 83%
Filed or granted patents 41%
Intellectual property claims (other) 13%
Spinoff companies 12%
Product licenses 10%

Figure 4: Proportion (%) of grants with KTU involvement in research stages

n = 79

Note: Based on NPI Recipient survey responses.

Figure 4 long description
Providing facilities, equipment, services and/or materials 58%
Development of the research idea/question 51%
Knowledge exchange 44%
End of grant KT activities 44%

Appendix C: Methodology – Additional Details

Additional details about the multiple lines of evidence and methodology used in the evaluation are presented in this appendix.

Environmental Scan & Document Review

The environmental scan focuses on the relevance of the CHRP program and provides information to address the following evaluation questions:

The environmental scan involved a review of program documentation from CIHR, NSERC, and SSHRC. Specifically, this included program information related to, but not limited to, the following: objectives, eligibility requirements, involvement of research domains (health, natural sciences and engineering, and social sciences and humanities), funding levels, partnership requirements, knowledge user involvement, knowledge translation plans/products, commercialization and HQP training.

The literature review component of the scan involved the analysis of relevant peer-reviewed (scientific and other academic) journal articles as well as relevant grey literature, such as reports disseminated by the three federal granting agencies (CIHR, NSERC and SSHRC), federal government (e.g. National Research Council, ISED), provincial/territorial governments, public agencies, private firms and, where directly relevant, international organizations. The specific publications and articles reviewed were selected based on targeted searches in literature databases and online search engines carried out by the consultant, and through information obtained from the key informant interviews.

End of Grant Report Data

CHRP grant recipients are expected to complete the end of grant report within 18 months of grant expiry. The end of grant report assesses recipients’ outputs and outcomes during the tenure of the grant in terms of numbers of collaborations, trainees involved in the project, and knowledge products (e.g., journal articles, patents).

Available end of grant data included data from 124 CHRP grant recipients covering five years of the CHRP program (2009-10 to 2013-14). The analysis compiled data from NSERC end of grant reports (2009-2012, n = 81) while CHRP was under NSERC’s administration, and CIHR end of grant reports (2012-2014, n = 43) during CIHR’s administration of the program. While NSERC and CIHR’s end of grant reports had some overlap in structure and content, different questions and measures were used; thus, data from the two cohorts of CHRP recipients which used different end of grant reports were reported separately.

Analysis of the demographics of end of grant data revealed that recipients were predominantly male (NSERC: 77%, CIHR: 85%), English-speaking (NSERC and CIHR: 98%), mid-career stage researchers (NSERC: 43%, CIHR: 51%).

Administrative Data Analysis & Funding History Analysis

A review of CHRP program records and administrative data from both CIHR and NSERC provided information on application and success rates, and program expenditures, as well applicant characteristics of such as affiliated institution, preferred language, gender, and funding history, which helped contextualize the program. This analysis also informed the sampling strategies used in the key informant interviews and the Recipient and Co-applicant surveys.

A separate analysis was conducted of all available funding data for grants received by researchers (i.e., NPIs on CHRP grants) from both NSERC and CIHR. This analysis was undertaken to expand on a previous set of exploratory analyses conducted by Snell and Harbord of CIHR Funding Analytics in 2015, which explored the relationship between CHRP-funded researchers and CIHR’s other funding programs.  Snell and Harbord examined the frequency of funding received though other CIHR programs (Open and Research in Priority Areas [RPA]) by CHRP-funded researchers, and the order in which these other sources of funding were received relative to researchers’ CHRP grant. The analyses for this evaluation expanded upon Snell and Harbord’s work in two ways: 1) by extending the timeline to include researchers funded by CHRP up to and including 2018; and 2) by including funding history from NSERC grants in addition to CIHR grants. Several components included in the previous analyses were not included in the present ones (i.e., age of researcher, uptake of researchers by year, and researcher pairings); while some additional descriptive statistics were included in the present analyses (i.e., mean, standard deviation, range, and total number of grants for each subgroup). There were no a priori hypotheses regarding expected relationship between CHRP-funded researchers and either NSERC or CIHR funding programs; these analyses were exploratory, as was the case in the original study.

In order to examine the full context of NSERC and CIHR funding history before and after CHRP funding, all CHRP grant recipients listed in the CIHR/NSERC databases were considered in the funding history analysis, including those who received CHRP during the years prior to the current evaluation period. The funding history analysis included grants on which researchers were listed as the NPI from both NSERC and CIHR grant funding databases, between 1995-2018. The researchers’ first CHRP grant was used as a reference grant, and any additional CHRP funding received after this was recorded separately from other CIHR and NSERC grants. Excluding CHRP grants (both reference and additional grants), the funding history analysis included a total of 950 CIHR and NSERC grants.

Key Informant Interviews

The interviewees were comprised of those who have a direct stake in the CHRP program and/or have been involved in its delivery, with a total of 49 interviews conducted. The following groups were represented:

The interviews were approximately 45 minutes long, fully confidential and semi-structured.  Respondents received an interview guide prior to the interview, to allow them to consider the questions in advance. Interviews were conducted by a combination of CHRP project team members from the CIHR Evaluation Unit and contractors.

Surveys of Researchers, Co-applicants, Trainees, and Partners

The surveys were developed in order to gain feedback from four participant groups involved with the CHRP program; in particular, to inform evaluation questions related to program relevance (need) and performance (program outcomes, and whether the program is meeting its objectives).

The surveys gathered information from the following populations:

Note that the data is self-report (and may be subject to biases and errors in recall) and overall ns for survey sample groups can vary throughout the report, as no survey items, other than demographics, were mandatory for respondents. Thus, the sample size for each sample can vary from question to question depending on responses. Because of this variance, denominators are presented for each sample, by question, whenever they change or differ.

A total of 757 online surveys were completed from approx. 2,952 who received the surveys (note: Trainee population numbers are not known; therefore, the completion number is used for the population number as an estimate).

The survey probed the views and opinions of respondents from these four populations about key aspects of the CHRP program, such as its value and distinguishing features, relative to other funding agencies’ offerings. The survey also revealed how instrumental CHRP funding was in sustaining the research; the level of satisfaction with the program and collaborations among NPIs, Co-applicants, and Partners that were facilitated by it; as well as the outcomes of the programs in terms of training of HQP, collaborations, and knowledge translation and commercialization.

While NPI and Co-applicant administrative data are maintained by NSERC and CIHR, the agencies do not track data on HQP beyond the number of trainees reported by NPIs on end of grant reports; therefore, there is no way to directly access the HQP involved in CHRP projects and the population of HQP is unknown. In order to target HQP for the Trainee survey, Recipients (NPIs) were asked to forward the Trainee survey on to the HQP involved in their CHRP-funded projects, and also to identify the number of HQP to whom the survey had been forwarded so that the Trainee population could be estimated (however, as many NPIs did not indicate the number of HQP they had forwarded the email to, it was not possible to accurately estimate the Trainee survey population). Partner responses were collected using a combined approach: those listed as Decision Makers or Principal Decision Makers on funded CHRP applications from CIHR administrative data were included; partner information was also validated via the applications themselves, and any additional individuals listed as Partners or KTUs on funded applications were included as well. In cases where ns were too low, results were not reported.

A Note About the Assessment of Performance

Due to an absence of explicit objectives related to expected outputs of the CHRP program, program performance, specifically innovations (products, services, and products), resulting from CHRP-funded research was assessed through several metrics, including:

Evaluation Limitations and Mitigation Strategies

Limitation Mitigation Strategy

Data availability and inconsistency in reporting of data:

  • Different End of grant reports used by NSERC and CIHR, neither with full coverage of outcomes of interest
  • Different systems used by NSERC and CIHR for recording of NPI IDs
  • Inconsistency and lack of identification data for partners/KTUs, particularly challenging due to the 2012 change in definition and requirement of KTU
  • Small sample size and/or poor response rate for surveys (Partners, and applicants (Researchers) who continued with the project) and key informant interviews
  • Identification data on trainees involved in the CHRP program not available, relied on NPIs to distribute survey link

End of grant reports were mined for as much data as possible, surveys were also used to assess outcomes related to the grants.

Data associated with NPIs and co-applicants was matched manually where IDs differed between CIHR and NSERC databases, and this data was spot checked for further validation.

KTU/partner information was extracted and cross referenced from multiple sources, including application and administrative data, to ensure that as many KTUs/partners were included.

When sample sizes were too low (N = 10 or less), findings were not reported or were interpreted cautiously, with the strength of evidence described.

Data from all lines of evidence (surveys, interviews, end of grant reports) were triangulated to make conclusions about the program.

Performance results are based largely on self-reported data (surveys, end of grant reports, and interviews), which is subject to potential biases and recall issues

Multiple data sources were included to triangulate findings related to performance wherever possible.

Given the time frame within which the End of Grant report is administered (~18 months post grant expiry), as well as a focus on grants awarded within the current evaluation period (2009-2018), it is possible that longer term impacts are not fully captured.

In addition to end of grant data analyses, researchers were surveyed about both current outcomes and impacts as well as expected future outcomes and impacts from their CHRP projects.

Lack of an appropriate counterfactual (due to small sample of applicants who continued with the project) and appropriate benchmarks makes it difficult to assess the success of the Program’s performance.

Comparisons were made with previous evaluation findings or comparable CIHR and NSERC programs wherever possible. Outcomes were interpreted cautiously in terms of program success, especially in comparisons between recipients and applicants (researchers).

References

Date modified: