Strategic Training Initiative in Health Research (STIHR) – CIHR Evaluation Report 2008-2013

December 2016

Evaluation Unit
Performance and Accountability Branch

At the Canadian Institutes of Health Research (CIHR), we know that research has the power to change lives. As Canada's health research investment agency, we collaborate with partners and researchers to support the discoveries and innovations that improve our health and strengthen our health care system.

Canadian Institutes of Health Research
160 Elgin Street, 9th Floor
Address Locator 4809A
Ottawa, Ontario K1A 0W9

Acknowledgements

Special thanks to all participants in this evaluation – principal investigators, survey respondents, interview participants, the Research, Knowledge Translation and Ethics (RKTE) – Investigator Initiated Research and Science, Knowledge Translation and Ethics Branches at CIHR, CIHR’s Financial Planning Unit, and CIHR’s Planning, Reporting, Measurement and Data Unit. Additional thanks to the members of the STIHR Evaluation Advisory Committee – Dr. Donald Cole (University of Toronto), Dr. Barbara Riley (University of Waterloo), Dr. Slim Haddad (University of Montreal), Dr. Jasdeep Saggar, Ph.D. (AstraZeneca), and Daniel Hubert (CIHR).

The STIHR Evaluation Team:
Shevaun Corey, Carmen Constantinescu, Jean-Christian Maillet, Alyssa Louw (student), Sarah Viehbeck, Michael Goodyer, and David Peckham.

For more information and to obtain copies, please contact Evaluation@cihr-irsc.gc.ca.

Contents

• Executive Summary
  • Program Overview
  • Evaluation Objectives
  • Evaluation Methodology
  • Findings and Conclusions
    • Key Findings: Program Performance
    • Key Findings: Program Relevance
    • Conclusions and Recommendations
• Background
  • Context and Objectives
  • STIHR Profile
  • Evaluation Scope, Objectives and Issues
    • Evaluation Questions
  • Methodology
  • Limitations of this Evaluation
• Evaluation Findings
  • Evaluation Issue 1: Program Performance
    • Key Findings
    • STIHR Programs had High Reach
    • STIHR Programs Used Diverse Training Approaches
    • High Levels of Trainee and PI Satisfaction with the Training Experience/Program
    • Majority of STIHRs Implemented Transdisciplinary Training Programs
    • High Compliance on Ensuring the Delivery of Training on the Ethical Conduct of Research and Related Ethical Issues, Knowledge Translation (KT) and Professional Skills
    • Positive Research Impacts on Trainees
    • Trainees’ Current Employment Concentrated Within Academia, High Anticipated Future Involvement in Health Research
    • Main Source of Funding was from CIHR; Trainees Secured Additional Funding from a Variety of Sources
    • Unanticipated Outcomes Resulting from the STIHR Program Included Exceeded Expectations and Some Workload Challenges for PIs
    • STIHR Programs Established Partnerships and Collaborations to Support Training Programs, but Sustainability was a Challenge
    • Successful Implementation of STIHR Programs
  • Evaluation Issue 2: Program Relevance
    • Key Findings
    • Continued Need for Training Programs Similar to STIHR
    • There is Continued Desire for Funding for Training Initiatives in Health Research from the Federal Government and CIHR
• Conclusions and Recommendations
• Appendix A – STIHR Investment Details
• Appendix B – STIHR Partners
• References

Executive Summary

Program Overview

The Strategic Training Initiative in Health Research (STIHR) was launched by the Canadian Institutes of Health Research (CIHR) in 2001. The purpose of STIHR was to build capacity within Canada's health research community through the training and development of researchers and the fostering and support of their careers. CIHR’s program was developed to introduce novel, innovative, and interdisciplinary methods of training the next generation of health researchers, thereby building health research capacity in Canada; attracting international research talent; and ensuring research excellence. A series of grants were used to establish training programs in specific areas of health research.

The STIHR model was extremely valuable to the Canadian health research landscape. Following over 10 years of funding in CIHR’s open programs, the decision was taken to no longer launch open STIHR funding but rather to make the STIHR tool available for use in priority-driven initiatives. Components of STIHR were also embedded in other CIHR programs and initiatives.

Evaluation Objectives

To provide CIHR senior management with valid, insightful, and actionable findings about the performance and relevance of STIHR and inform CIHR’s training strategy. The evaluation covers the funding period 2008-2013.

Evaluation Methodology

The evaluation used multiple sources of quantitative and qualitative evidence: analyses of documents and data; end of grant and progress reporting; surveys with principal investigators (PIs) and trainees; key informant interviews with PIs, trainees, STIHR partners and university staff. The evaluation meets the requirements of the Treasury Board Secretariat (TBS) under the 2009 Policy on Evaluation.

Findings and Conclusions

Key Findings: Program Performance

• STIHR programs supported the development of training programs that improved the mentoring and training environment for health researchers. A large number of trainees (3300+) and mentors (approximately 3000) were reached by the program. Training programs were successfully implemented by PIs and STIHR trainees reported high satisfaction with their training experience.
• STIHR programs developed collaborative, team research by bringing researchers together from different disciplines to address major health issues and/or health research challenges. STIHR programs addressed diverse and major health issues or health research challenges and developed and deepened inter-organizational, inter-institutional, or inter-program collaborations.
• STIHR programs contributed to the development of well-rounded health researchers by integrating training on the ethical conduct of research, knowledge translation and professional skills.
• STIHR programs positively contributed to the research outcomes and the career trajectories of STIHR trainees. Participation in the training program resulted in a range of positive impacts on trainees. The majority of STIHR trainees were either currently (or planning to be) employed in the academic sector (including those completing postdoctoral fellowships).
• STIHR programs established partnerships to support the training programs; however, there was limited partnership support for sustainability of STIHR-funded programs. The majority of STIHR programs had multiple partnerships/collaborations. Nearly all programs attracted a variety of types of cash and in-kind funding.
• There was disappointment in the decision by CIHR to no longer launch open STIHR funding opportunities. This was expressed by the PIs, trainees and university representatives. The sustainability (i.e., continuation) of programs in their current form, beyond aspects such as curricula (e.g., courses) and networking (e.g., relationships built), is unlikely. On average, STIHR programs secured approximately 38% of the total funding needed to continue the program after the end of the grants from CIHR.
• STIHR programs were successfully implemented in a cost efficient manner and many benefits for an indirect funding model for trainees were identified. Overall, STIHR programs were successfully implemented with high levels of satisfaction with the design, implementation and funding. The ratio of CIHR program expenditures to partner expenditures was $1.00:$0.10.
• Most unanticipated outcomes resulting from STIHR programs were positive. Trainees felt like they were part of a community as a result of STIHR program involvement. PIs felt there were significant increases in the beneficial interactions and collaborations between academic centers both within and outside of Canada.
• Challenges with the program included the high workload associated with the management of the STIHR programs and the level of effort needed to maintain them, the lack of funding for maintaining/running program, a lack of infrastructure to support all training and the associated administrative burden.

Key Findings: Program Relevance

• There is continued need and interest for an initiative similar to STIHR. PIs were motivated to apply in order to build capacity within the Canadian health research community and trainees were motivated to apply because they were interested in the specific research areas and topics of the STIHR-funded programs along with the stipends.
• STIHR is aligned with CIHR and federal government priorities. The objectives of STIHR were aligned with CIHR’s mandate, objectives within the CIHR Act as well as strategic directions from the current and previous strategic plans.

Conclusions and Recommendations

The evaluation found STIHR to be effective and to have met its objectives. The focus of the recommendations is to inform CIHR’s Strategic Action Plan on Training (T-SAP) and future training initiatives given that CIHR is no longer launching open STIHR funding opportunities. Further supporting details for the recommendations are provided in the main body of the report.

1. As CIHR implements its Strategic Action Plan on Training (T-SAP), future CIHR approaches to indirect training should be informed by the STIHR model and there should be increased use of the STIHR tool within CIHR’s priority-driven funding.

The strategic directions of CIHR’s T-SAP can be informed by the STIHR model. The model demonstrated success in developing and implementing strategic transdisciplinary training programs, targeting specific skill areas using multiple and mixed training approaches, while effectively integrating high quality mentors into the training environment. Almost all STIHR programs ensured training on the ethical conduct of research and related ethical issues, knowledge translation and professional skills, all areas of interest to CIHR’s mandate and the T-SAP.

A design element of the STIHR model included indirectly funding trainees via stipends, in addition to direct funding trainees may be receiving external to STIHR. The STIHR model successfully reached a large number of high caliber trainees from a variety of disciplines.

2. CIHR should continue to provide support for training that adds value beyond stipends and what is delivered through degree programs.

Trainees benefited by being engaged in a STIHR program above and beyond what they would have received through their degree program alone. This includes transdisciplinary training in transferable skills, access to multiple mentors and additional funding and different types of opportunities to support the employability of trainees. PIs perceived several benefits from the leadership of STIHR programs through the development of teaching resources and research productivity.

3. Future CIHR training initiatives must have stronger performance measurement.

There is a need for stronger performance measurement, tracking and communication processes. This would help ensure the efficient tracking and contacting of trainees, generation of data for monitoring and evaluation, measurement of outcomes and impacts and partner engagement.

4. CIHR should identify, plan for and communicate sustainability expectations and/or requirements beyond grant funding.

There was widespread disappointment among stakeholders in the decision to no longer launch open STIHR funding opportunities and many programs are unlikely to be sustainable. The process of sun-setting STIHR and transitioning to T-SAP has highlighted the need for purposeful planning, definition, and clear communication, by CIHR, about program objectives including sustainability plans and requirements.

Background

In 2007-2008, the Evaluation Unit of the Canadian Institutes of Health Research (CIHR) conducted an outcome-oriented evaluation of the first six years of the Strategic Training Initiative in Health Research (STIHR), which included review and analysis of the STIHR competitions funded in 2002 and 2003.^{Footnote 1} A second evaluation of STIHR was initiated in 2013, which focused on programs funded since 2004.

Context and Objectives

Implemented by CIHR in 2001, the goal of STIHR was to build capacity within Canada’s health research community, to increase its competitiveness internationally in attracting new, bright, creative research talent and to ensure innovation and excellence in the next generation of Canadian health research training programs. Specifically, the objectives of STIHR were to:

1. Support the development of training programs that improve the mentoring and training environment for health researchers.
2. Support the development of collaborative, team research by bringing researchers together from different disciplines to address major health issues and/or health research challenges.
3. Support the development of well-rounded health researchers by integrating training on:
   1. the ethical conduct of research and related ethical issues;
   2. knowledge translation; and
   3. professional skills such as communication, teamwork, project management, leadership, grant writing, and peer review.

Several factors had an impact on the current evaluation. Most importantly, CIHR announced in 2012 that it would no longer be launching open STIHR competitions. The value of the STIHR model as a capacity building and training tool was acknowledged, and CIHR announced plans to embed training of young researchers and the concepts of training and mentoring into both its strategic initiatives and its reforms to the open suite of programs. By expanding the focus beyond one stand-alone program, it was expected that CIHR would continue to enable the strengthening and broadening of training and mentoring for up-and-coming Canadian researchers. Additionally, and simultaneous to the evaluation, CIHRs Strategic Plan, Health Research Roadmap, was updated^{Footnote 2} and a Strategic Action Plan on Training (T-SAP) was developed. However, the evaluation proceeded in order to meet TBS requirements under the Policy on Evaluation.

STIHR Profile

There has been an investment of $269M in STIHR since 2001. In the time period for this evaluation, a total of 59 STIHR programs were funded (36 were renewals from earlier launches). Funding for these programs ($325K per year over five years), primarily targeted towards supporting research trainees through stipends (71% for trainee stipends and travel, 29% for program expenditures), was provided by CIHR, the 13 CIHR Institutes and 17 key STIHR partners.^{Footnote 3} Similar to the 2008 evaluation, the majority of the STIHR-funded programs belonged to the biomedical pillar (49%; N = 59), followed by social/cultural/ environmental/population health, and clinical and health systems/services (24%, 14% and 14%, respectively).

Evaluation Scope, Objectives and Issues

The evaluation (including development of evaluation questions) was guided by a working group consisting of CIHR staff and representatives from funded STIHR programs and was designed to meet the Tri-Agencies’ requirements to TBS under the Policy on Evaluation, by addressing the core issues of performance (e.g., achievement of expected outcomes) and relevance (e.g., alignment with government priorities and continued need).^{Footnote 4} In addition, the evaluation was also intended to provide senior management with valid, insightful, and actionable findings about STIHR and its approach to supporting training, and inform the development and implementation of CIHR’s training strategy.

Evaluation Questions

Performance

1. To what extent has STIHR supported the development of training programs that improve the mentoring and training environment for health researchers?
2. To what extent has STIHR supported the development of collaborative, team research by bringing researchers together from different disciplines to address major health issues and/or health research challenges?
3. To what extent has STIHR supported the development of well-rounded health researchers by integrating training on: a) the ethical conduct of research and related ethical issues; b) knowledge translation; c) and professional skills such as communication, teamwork, project management, leadership, grant writing and peer review?
4. What contribution has STIHR had on the research outcomes and the career trajectories of STIHR trainees?
5. What unanticipated outcomes if any, have occurred as a result of STIHR?
6. To what extent have STIHR-funded programs established and maintained partnerships to support the training programs and to ensure their sustainability?
7. Has STIHR been delivered by CIHR in a cost efficient manner?

Relevance

8. Is there a continued need for an initiative similar to STIHR?
9. To what extent does STIHR align with CIHR and federal government priorities?

Methodology

Consistent with best practice in evaluation, a mixed methods design was used; involving both qualitative and quantitative data, with a variety of data sources (e.g., document review; progress reports, N = 59; interviews with trainees, n = 12, PIs, n = 10, partners, n = 3, and university representatives, n = 4; and, surveys with trainees, n = 1643, and PIs, n = 37). ^{Footnote 5} By using multiple data sources and methods, the research design was strengthened. Additional strength comes from the heterogeneity of findings when triangulated across multiple lines of evidence. When possible and appropriate, comparisons were made to findings from the 2008 STIHR Evaluation.

Limitations of this Evaluation

The following limitations should be noted:

• Limited comparisons were possible due to the uniqueness of the program.
• Attributing trainees’ achievements solely to STIHR training was not possible given their simultaneous involvement with other training and additional possible confounding variables (e.g., caliber of trainee).
• There was great variability across STIHRs (e.g., pillar, subject matter, approaches) and uneven/small numbers of trainee survey respondents across some questions^{Footnote 6}; therefore, findings are presented at the initiative level only (i.e., STIHR level analyses are not reported).
• The majority of data collection tools were self-report and subject to potential biases and recall issues. Sample sizes were small for interviewed target groups limiting the generalizability of some data.
• PIs were contacted to provide STIHR trainee contact information (with consent) and in addition to being a very time consuming process, it is possible that incomplete documentation or recall resulted in an incomplete sample.
• The timing of data collection across lines of evidence varied (e.g., progress reports analyzed were submitted in fall 2012, interviews occurred in spring/summer 2014 and surveys were distributed in summer 2015), with some data collected after the termination of the initiative.

Evaluation Findings

Evaluation Issue 1: Program Performance

Key Findings

• STIHR programs:
  • supported the development of training programs that improve the mentoring and training environment for health researchers;
  • developed collaborative, team research by bringing researchers together from different disciplines to address major health issues and/or health research challenges;
  • contributed to the development of well-rounded health researchers;
  • positively contributed to the research outcomes and the career trajectories of STIHR trainees;
  • established partnerships to support the training programs; and
  • were successfully implemented in a cost efficient manner.
• There was disappointment in the decision to no longer launch STIHR funding opportunities.
• Challenges with the program included the high workload associated with the management of STIHR programs.

STIHR Programs had High Reach

A large number of trainees were involved in STIHR programs (approximately 7000 since 2001), either as a funded participant (e.g., admission to a STIHR program/receiving a STIHR award/stipend), or in an unfunded capacity (e.g., participating in a training course or workshop that was part of a program). The majority of trainees were at the doctoral level (34%), female (58%), English speaking (84%), from Canada, with institutions located in all provinces (but no territories), with concentrations in Ontario (33%) and Québec (30%).^{Footnote 7} About 15% of trainees’ institutions were located outside of Canada. Host institutions^{Footnote 8} played an important role for PIs, and the STIHR programs overall, by providing space for research and training in addition to other types of support. According to progress reports, the majority of STIHR programs included in this evaluation were hosted by institutions in Ontario, Quebec and British Columbia, which corresponds to the high concentration of trainees in these provinces as well.

A large number of supervisors and mentors were involved in STIHR programs, approximately 3000 overall, with an average of 51 per program. The majority of trainees surveyed had reported having one supervisor (59%), consistent with findings from the 2008 STIHR Evaluation (62%). Over half of the trainee survey respondents (61%), indicated that they had multiple mentors and were exposed to mentors beyond their direct supervisor (54%); which differed slightly from the 2008 STIHR Evaluation, where most had mentors who were also their supervisors (67%).

One of the challenges reported by PIs was the recruitment and maintenance of mentors, especially during program implementation, which could help explain the discrepancy. It is likely that after the initial implementation phase, PIs were more able to focus on enhancing the transdisciplinary and collaborative nature of their programs, including recruitment of mentors. Or they could have simply been more successful at recruiting mentors once their programs became more established.

STIHR Programs Used Diverse Training Approaches

One of the objectives of STIHR was to support the development of training programs that improve the mentoring and training environment for health researchers. PIs confirmed (through interviews and progress reports) that their programs had a defined curriculum with mentoring and training components uniquely designed for each one. Diverse opportunities for training were provided across programs, using a variety of training approaches and methodologies. The most common approaches included research seminars, workshops or meetings, the use of communication technologies in training, annual research meetings, symposia, or retreats, and core courses (>75%), while the least frequent approach was open participation to non-funded students (26%).^{Footnote 9}

Additionally, there was a growth in the use of communication technologies (78%), specifically interactive web technologies, from earlier-funded STIHRs (55%), in order to enhance teaching modalities and link researchers and trainees in multiple settings. Most importantly, trainees felt, to a large extent, that the training they completed both exposed them to new learning opportunities (M = 4.29^{Footnote 10}, SD = .86), and introduced them to new, exciting, and relevant areas of health research (M = 4.19, SD = .83).

High Levels of Trainee and PI Satisfaction with the Training Experience/Program

Consistent with what was found in the 2008 Evaluation, trainees were satisfied to very satisfied with their overall STIHR experience as well as the supervision and mentoring they received (mean ratings ranged from 4.35 to 4.50). In addition, trainees were satisfied with the monetary value and length of the stipend as well as the resources available to them during training (mean ratings ranged from 4.21 to 4.24). They felt they were provided with networking opportunities and that participation in STIHR programs created a sense of community.

“Just a sense of community. My cohort was small, there were six students in my year and we all still keep in touch even though we are scattered throughout the city and my engagement with the mentors I had, I’m on [a] friendly first name basis with all of them when I see them at an event, and I feel like I’ll have people to go to when I graduate and am looking for work.”

Surveyed PIs reported a variety of outcomes that have resulted from STIHR programs, the most frequent were findings cited by others (57%), followed by the development of new research methods (51%), and tools, techniques, instruments or procedures (49%). Least frequent outcomes related to social, economic, and health benefits such as behavior change, direct cost savings, or patents (5-19%).

PIs benefited from their leadership of a STIHR program, mainly through positive impacts on their career and research, with the highest rated item being the development of new courses and/or teaching materials (M = 4.04, SD = .26). Challenges and less-beneficial elements included increased workload (M = 3.85, SD = .82), maintaining and/or increasing salary (M = 1.91, SD = 1.34), and career advancement (M = 3.09, SD = 1.28). In progress reports and interviews, PIs also described the following impacts they felt the STIHR programs had on themselves, mentors and institutions:

• Programs produced highly trained graduates who had a better fit between their training and future study plans or career paths.
• Universities, specific faculties/ departments, and researchers benefited through the recruitment of new talent, collaboration between mentors (e.g., writing research grants), increased research activities and leveraging resources (e.g., pooling funds, use of mentors, use of specific labs).
• Trainees found a community of learning (across Canada and internationally) that would continue beyond their involvement with the STIHR programs, with benefits for mentors/researchers also.

Overall, PIs who responded to the survey felt their STIHR program had a positive impact on their research productivity in a variety of ways. Those activities rated most highly included participation in conferences (M = 4.15, SD = .77) and peer reviewed journal publications (M = 3.93, SD = .87). In addition to benefits a number of leadership challenges were identified, including:

• Increased workload/time pressures (for program management and maintenance)
• Difficulty accessing and/or maintaining mentors and managing active participation
• Missing or developing expertise for delivering training within the program (beyond mentors)
• Developing a cohesive program across distance and disciplines
• Language/translation considerations given the transdisciplinary nature of some programs
• Unexpected costs associated with the management/implementation of the program

PIs and trainees identified key strengths of the programs that showed support for the achievement of all three STIHR objectives (e.g., training programs that improve the mentoring and training environment for health researchers, development of collaborative and transdisciplinary research, and supporting the development of well-rounded health researchers by integrating ethics, KT and professional skills training). The main strengths described by both PIs and trainees included:

• STIHR funding addressed a gap or need in capacity as well as research area.
• The program design in and of itself, specifically the following elements:
  • funding focused on trainee development as a central focus of the program versus funding focused on research or the researcher’s needs;
  • training programs:
    • centered on trans-disciplinary/collaborative training curricula;
    • incorporating beneficial, effective and high quality mentorship;
    • providing trainees with specific knowledge and skills in areas that would assist them in future research careers (e.g., grant writing, knowledge translation, exposure to alternative methodologies); and,
    • providing significant and varied networking opportunities with a variety of collaborators (e.g., PIs, supervisors, mentors, subject area organizations).
• The stipend was important, although it was a smaller factor in trainees’ overall positive perception of the program beyond the initial draw.

Several PIs noted that training provided by STIHR programs should be continued, specifically in the context of transdisciplinary learning, professional skills (e.g., grant writing, presentations), and ethics in research and knowledge translation (PI Interviews). This sentiment was echoed by trainees, who also identified the need for more transferable skills (e.g., project management, data analysis).

Majority of STIHRs Implemented Transdisciplinary Training Programs

The STIHR objective to support the development of collaborative, team research by bringing researchers together from different disciplines to address major health issues and/or health research challenges was achieved. Overall, STIHR programs employed cross-pillar, cross-discipline, multi-method approaches in training and trainee selection. Additionally, the STIHR programs covered a diversity of topics that addressed major health issues (e.g., cancer; chronic disease prevention) or health research challenges (e.g., health equity; population interventions).

In addition, to recruiting trainees from outside of Canada, STIHR program leaders fostered disciplinary diversity among trainees by focusing their selection on those trained in different health research pillars as well as in multiple disciplines across the social, physical and health sciences. Overall, 93% of STIHR programs implemented transdisciplinary training programs and 81% of trainees’ research and course work spanned traditional disciplinary boundaries.

PIs rated the extent to which the individual STIHR programs provided training through collaborative research within different disciplines very highly (e.g., access to mentors with diverse research disciplines, provided training content from different disciplines, used a multidisciplinary approach to research problems and provided training in two or more pillars; mean ratings ranged from 4.41 to 4.81). In addition, trainees were very satisfied with the opportunities for collaboration with a variety of sources ranging from their supervisors and mentors, to other students, researchers (within their discipline in Canada) and knowledge users (with means ranging from 3.67 to 4.46). Lastly, trainees reported skills improvement in multidisciplinary/ transdisciplinary research and networking (M = 5.25^{Footnote 11}, SD = 1.41and M = 5.22, SD = 1.50), but only moderate improvement in collaborative research with private, government and not-for-profit sectors as well as international research collaborations. (M = 3.76 for both). These results suggest that STIHR programs effectively incorporated and delivered training on transdisciplinary collaborative research with some room for improvement in training focused on multi-sectoral and international collaborative research.

High Compliance on Ensuring the Delivery of Training on the Ethical Conduct of Research and Related Ethical Issues, Knowledge Translation (KT) and Professional Skills

One of the objectives of STIHR was to support the development of well-rounded health researchers by integrating training on: 1) the ethical conduct of research and related ethical issues; 2) knowledge translation; and, 3) professional skills (such as communication, teamwork, project management, leadership, grant writing, and peer review). This objective was met. Almost all STIHRs ensured training in these three areas were compulsory and integrated into the program: 88% of programs ensured training on the ethical conduct of research and related ethical issues, 91% ensured training on KT, and 92% delivered training on various professional skills. The levels of compliance in all three areas are higher in the current evaluation than those figures from the 2008 STIHR Evaluation, reflecting the updated initiative design, which made this objective explicit.^{Footnote 12}

For all three areas, the most common method of training was through workshops and/or modules developed within the STIHR program (71% for ethics, 53% for knowledge translation, and 37% for professional skills). Trainees reported an improvement in skills related to ethics and knowledge translation (M = 4.70^{Footnote 13}, SD = 1.59 and M = 5.22, SD = 1.50) and also rated them both as very useful program components (M = 3.84, SD = .96 and M = 4.01, SD = .89).

As indicated above, a variety of professional skills were covered by STIHR programs (progress reports identified communication skills and grant writing as the most frequent). Perceived skill improvement was assessed for entrepreneurship and business management skills and in contrast to ethics and KT, trainees reported very slight to slight improvement in both (M = 2.66^{Footnote 14}, SD = 1.87 and M = 2.45, SD = 1.77). However, it should be noted that they were offered less frequently than other professional skills in STIHR programs. This suggests that when training is required to be offered in targeted areas that trainees are likely to benefit.

Trainees were asked to rate the perceived usefulness of various professional skills such as communication, teamwork, leadership, project management, peer review and grant writing. Overall, mean ratings were relatively high with communication and teamwork rated as most useful (M = 4.10, SD = .83 and M = 3.99, SD = .91). Although addressed slightly differently in the 2008 STIHR Evaluation, a large proportion of trainees also rated communication/teamwork/ leadership, knowledge translation, ethical conduct and grant writing/peer review skills as very useful.

Positive Research Impacts on Trainees

PIs felt that participation in STIHR training programs had resulted in a number of positive impacts on trainees (progress reports). The most frequently cited impacts were peer-reviewed publications and presentations (55%), and positions attained by trainees, including subsequent postdocs and fellowships (54%). Trainees were also asked directly about their outputs related to publications and presentations while completing the STIHR program. Survey respondents indicated that the most frequent output was articles written or co-written that were either published or accepted for publication in a peer-reviewed journal (n overall = 3181, M = 2.95, SD = 4.01), followed by research papers, books, chapters, and/or technical publications authored or co-authored and either published or accepted for publication (n overall = 1701, M = 1.63, SD = 3.90). The least frequent outputs were research tools (e.g., databases), patents (applications or granted) and other intellectual property claims (overall n’s ranged from 48-104). These findings suggest that engagement in a training program can set trainees up for success in research-related fields.

Overall, trainees who had completed schooling or who were at the postdoctoral or fellowship level were responsible for the highest average number of outputs (with the exception of patents and other intellectual property claims). This finding is likely due to a combination of career/academic status and the length of time spent as a STIHR trainee. The average number of outputs was similar to that found in the 2008 Evaluation (with the exception of more conference presentations in latter). Taken together, these findings lend support to the conclusion that STIHR trainees were actively engaged in scientific production, contributing relatively large numbers of a variety of outputs.

Trainees were asked directly, via the survey, to identify outcomes that have resulted while they were STIHR trainees during their degree programs.^{Footnote 15} The most commonly reported outcomes were: findings cited by others (35%), and new research methods (30%). Least frequent outcomes were related to social, economic, and health benefits such as direct cost savings, vaccines/drugs, as well as new spin off companies (1-2%). All outcomes were similar to those reported by PIs.

Trainees’ Current Employment Concentrated Within Academia, High Anticipated Future Involvement in Health Research

According to progress reports, the majority of trainees were still in involved in research (78%) and still in academia (60%), mostly as students or postdoctoral fellows (81%), presumably having moved into the next level of research training.^{Footnote 16} Others were working in clinical settings (14%), the private sector (8%), or the public sector (6%). These findings were confirmed by surveyed trainees, with the majority indicating they were working full time (66%), conducting health-related research (84%), employed within academic sector (67%; most commonly as students/postdoctoral fellows), employed in Canada (78%), with average incomes ranging from $50,000-$99,999.

The most common position for trainees employed within a university was research faculty (31%) followed by postdoctoral fellow or associate (23%). The most common rank of trainees who held teaching or faculty positions (47%) was Assistant Professor (57%) and the most common tenure status was tenure track (44%). For respondents who indicated they were employed in Private, Government or Not-for-profit Sectors, the most common level was intermediate (38%). The average amount of time it took trainees to find a job was 7.08 months (SD = 14.77), although just over half (55%) found employment right away. Taken together, these findings suggest that the majority of trainees selected for STIHR programs were currently employed in the academic sector as students, postdocs, or research faculty who were working full time, conducting health research in Canada.

Those trainees not currently employed (who were either actively or not actively looking for employment), intended on being employed in the academic sector most frequently, followed closely by government, private and not-for-profit sectors. In addition, the majority of unemployed respondents planned to continue conducting health related research (88%) and work in Canada (81%).^{Footnote 17} These findings were consistent with those found in the 2008 Evaluation; however, there were more trainees, in the current evaluation, who intended to work in sectors other than academia, which is likely a reflection of the current job market. These findings suggest that trainees who were selected for STIHR programs were intending to work in the academic sector, conducting health research in Canada, even if not currently employed.

Overall, PIs felt STIHR programs had a positive impact on employability and career development by providing opportunities to experience desirable employment settings (e.g., multi-disciplinary and multi-setting exposure) and develop contacts and networks within those settings. In addition, most programs ensured professional transferable skills’ training was provided (e.g., grant writing, leadership, communication, etc.). To support trainees’ career success, about one quarter of STIHR programs (26%) reported carrying out activities to help trainees identify their career objectives. In fact, interviewed PIs indicated that in some instances, specific career counseling was part of the program. Just over half the programs (52%) systematically exposed trainees to career options and career preparation activities (e.g., rotations in other settings, having representatives from other settings speak at seminars, reflective guest speakers sharing advice and career trajectories, skill-building workshops, preparing for job talks, etc.). Interviewed PIs felt that as a result of the STIHR program, trainees headed into academic careers were provided with opportunities beyond regular degree programs (e.g., working with multiple supervisors/mentors, networking, and access to additional funding).

These findings were supported by interviewed trainees, whereby most felt components of STIHR programs contributed to making them marketable or would assist them in finding employment (e.g., training, quality of trainers, learning transferable skills, and strengthening their CV). Surveyed trainees felt the training they received was very useful in preparing them for their career (M = 3.96, SD = .95), with 68% choosing ‘very useful’ or ‘extremely useful’ (i.e., 4 and 5). This finding was similar to that found in the 2008 Evaluation, where 50% of trainees indicated that they were very optimistic about their career prospects. These findings suggest that the purposeful integration of activities and resources related to employability and career development were perceived as useful and beneficial to trainees.

Main Source of Funding was from CIHR; Trainees Secured Additional Funding from a Variety of Sources

Although the most common source of financial support during trainees’ degree programs was research stipends and/or assistantships from CIHR (78%), according to progress reports, 48% of trainees secured stipends and/or awards from other sources for a total of approximately $64M across all STIHRs^{Footnote 18} (with an average of $1.12M per STIHR and $19,159 per trainee). On average, among trainees paid stipends through the STIHR, the STIHR grant paid 59% of their stipend and other sources paid 41%. According to progress reports, the three main sources of trainee funding were 1) institutions, 2) supervisors, and 3) competitive awards. Specifically, institutions and supervisors/PIs contributed stipends from their own funding, and, in some cases, matching funding was a requirement for entry into the program. Several STIHR programs had developed co-funded scholarships or fellowships with another organization. Lastly, trainees had been awarded external scholarships from a wide variety of Canadian (e.g., AllerGen NCE, CIHR Postdoctoral Fellowship, Michael Smith Foundation for Health Research, Ontario Graduate Scholarship) and some international sources. Many STIHR programs required that trainees submit applications to external funders as a requirement of receiving STIHR funding.

Trainees were asked to identify the sources of the awards and prizes that they received during their degree programs and the most common source was a university (69%) followed by Canadian federal granting agencies (50%), and Canadian Provincial Bodies (44%). Of those who received funding from a federal granting agency the most common types were Other (e.g., travel awards, awards from CIHR Institutes, project stipends), a CIHR Fellowship (51%), a CIHR Doctoral Research Award (49%), and stipends from CIHR grants (47%). The least common sources included specific the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC) postgraduate and doctoral scholarships. Trainees were also asked to identify other sources of financial support (other than the awards and prizes identified above), and the most common source was a research stipend/assistantship from another source (40%) and a teaching assistantship (40%), followed by a stipend/assistantship from a research grant from a federal granting agency (CIHR, NSERC, SSHRC) (35%).

Trainees interviewed confirmed that the majority (73%) received additional grants before, during or after their involvement with STIHR. Most grants overlapped with the receipt of a STIHR stipend and were provided by multiple sources, with amounts ranging from $750-50,000. These findings were similar to that from the 2008 STIHR Evaluation. Taken together, results from multiple lines of evidence indicate that STIHR trainees received multiple sources of funding and financial support throughout their degree programs. Their success with securing funding speaks to the caliber of those associated with the STIHR, requirements to secure additional funding, and to the complementary nature of the training delivered through the STIHR, which is unique in design across these funding sources.

The majority of survey respondents, in the current evaluation, indicated they did not have any debt (65%) upon program completion. Of those that did have some debt (22%), out of the respondents who indicated the amount they had, the average was $29,127 (SD = 38,215; range $0-$380,000). In the 2008 Evaluation, a higher percentage of trainees indicated they were currently carrying debt (37%).

Unanticipated Outcomes Resulting from the STIHR Program Included Exceeded Expectations and Some Workload Challenges for PIs

All target groups were asked to identify unanticipated outcomes through the progress reports, surveys as well as interviews. For the most part the majority of comments were positive, describing ways in which the STIHR program exceeded expectations (for trainees this was in establishing networks, acquiring transferable knowledge and skills, and the sense of community; for PIs this was the successful incorporation of technology and resulting skill development and reduction in teaching burden, the increased collaboration between academic and non-academic centers outside of Canada, and the increased visibility and prestige of their academic organizations).

Challenges identified through multiple lines of evidence included the high workload associated with the management of the STIHR programs, and the level of effort needed to maintain them (e.g., ensuring the recruitment and participation of mentors). Although the increased workload was a challenge it was often considered a “labour of love”, that leading STIHR programs involved passion, commitment and drive in order to provide excellence in health research training. PIs indicated there was a lack of funding for maintaining/running programs (including unexpected costs for travel, etc. and funding needed to sustain STIHR programs) and a lack of infrastructure to support all training. Additionally, there was an increased administrative burden, including: staff turnover; challenges working across departments and faculties, and high costs leading to program constraints. These findings suggest the need to provide sufficient support to program leaders to enable the successful management and maintenance of programs.

STIHR Programs Established Partnerships and Collaborations to Support Training Programs, but Sustainability was a Challenge

Recall, there are two levels of partners: the initiative level (partnerships secured by CIHR) and the program level (partnerships secured by individual STIHRs). At the initiative level, CIHR had partnership arrangements with a number of organizations (Appendix B), who contributed a total of $26,746,160 (10% of total funding), since 2001.

Only a small sample of initiative level partners were interviewed; however, two out of three identified challenges in assessing the impact of STIHRs (due to a lack of baseline data/clear metrics), and communication (lack of consistent engagement, updates/progress reports). Similar challenges were identified in the 2008 STIHR Evaluation (e.g., lack of communication, clarifying roles of partners, and the need for more evaluation and reporting). Although it is acknowledged that sample sizes were low in the current evaluation, this persistent feedback around evaluation and reporting suggests a need to improve performance measurement approaches for the programs.

Similar to the 2008 Evaluation, the majority of individual STIHR-funded programs (79%) had additional partners and/or collaborators (both within and outside of Canada), over and above initiative level partners, with most indicating that that they had multiple partnerships (progress reports). Program level partners played a variety of roles, the most frequent was the co-funding of awards (61%), followed by providing access to resources (30%). Similar to the 2008 Evaluation, Canadian Universities and/or research centres were the most frequently identified partners/collaborators by PIs across programs (81%). Program level partners differ from the initiative level partners, which are mostly made up of provincial organizations and health charities. PIs interacted with partners/collaborators most often by providing updates (48%), followed by regular progress reports (42%) and participating in joint activities (39%).

Overall, PIs were very satisfied with the relationships with STIHR program level partners (M = 4.22, SD = 1.31), which was slightly higher than the 2008 Evaluation. This finding lends support to some improvements in PI satisfaction with partnerships as the initiative continued. Numerous partnership benefits were described by PIs, the most common was improved training opportunities for students (61%) and increased opportunities for knowledge transfer (55%). Although challenges were infrequent across STIHR programs, added administrative burden was the most common (21%). Given that communication and regular progress reporting was a challenge identified by some initiative level partners, the findings above suggest that success was achieved at the program level.

According to progress reports, almost all STIHR programs (97%) reported having attracted a variety of types of funding from sources other than the program itself, most commonly as cash contributions by universities, government and private organizations (80%), followed by funds awarded to students (78%). Two-thirds of STIHRs (66%) reported receiving in-kind administrative and teaching (research) support from their institutions, while 50% reported receiving cash contributions from industry, foundations, and donors. Other sources of funds included in-kind access to infrastructure (e.g., equipment, space, labs) (22%), fundraising from fees, sales or donations (19%) and cash contributions toward equipment, infrastructure and lab facilities (3%).

Of the partners/collaborators identified by surveyed PIs, those that provided the most in cash contributions included provincial governments and private organizations (>$1M), while those that provided the most in in-kind contributions included Canadian universities and/or research centres and Canadian federal organizations (>$1M). PIs indicated that the most common ways in which host institutions supported their STIHR was by providing working space (63%), followed by facilities and materials (56%) and additional funding (50%). Host institutions provided close to 20% of their overall program funding (M = 17.19%, SD = 22.65%; PI Survey).

Data from all lines of evidence converged on disappointment with the decision to no longer launch open STIHR funding opportunities. When reporting on future plans, although 66% of the 59 STIHRs described approaches for ensuring sustainability after their CIHR funding ended (e.g., seeking funds from other funders; working to merge/link with other STIHRs or entities; integrating training programs within their university’s existing system/program offerings, engaging in succession planning), at the time the progress reports were completed and analyzed, only 38% (on average) of the funding needed to continue each program after the end of the grant had been obtained. The proportion of contributions obtained to continue programs after the end of the STIHR grants from CIHR totalled more than $11M, or approximately $200,000 per STIHR program.

PIs indicated very few STIHR-funded programs would continue (as is) after the grants ended; 50% said host institutions would not continue to provide support. When asked which program components would continue after CIHR funding ends, not surprisingly, the most frequent were relationships with participants (47%) and courses (44%), which are elements that are the least likely to require substantial or ongoing funding, and are more naturally occurring in an existing system. Interviewed PIs expressed confusion in the expectation that programs would eventually be self-sustaining and others felt there was a lack of support in sustainability planning process, specifically since most plans were still under development when funding ended. Several barriers to sustainability were outlined by the interviewees including limited funding sources, a lack continued commitment from partner Universities and mentors, a lack of trainee interest and applications, and administrative effort for program management and maintenance.

Successful Implementation of STIHR Programs

According to progress reports, STIHR programs were implemented as intended. Implementation challenges identified most often by PIs included issues related to funding sustainability for training programs. A small number of PIs reported other challenges, consistent with those from the 2008 Evaluation, including: cross-institutional administrative collaboration, trainee and mentor recruitment and participation (i.e., difficulty attracting a large enough pool of candidates and/or achieving a desired balance across regions, disciplines, pillars or languages), and developing recruitment or eligibility criteria to optimize participation. A few challenges were noted with program development and design, including ensuring trainees had adequate time to participate in their training program activities over and above regular program commitments.

Despite the implementation challenges noted above from the progress reports, surveyed PIs were very satisfied with the recruitment process and felt there was a medium sized and very diverse pool of highly qualified trainees. Comments provided by PIs highlighted that once programs were established the difficulty with trainee recruitment was alleviated.^{Footnote 19} PIs considered many factors when recruiting trainees, the most common being publications/research achievements, awards, and interest in the STIHR program subject area. An average of 18% of participants who applied were accepted per STIHR (SD = 24.58%). Taken together, these findings suggest that becoming a STIHR trainee was a competitive process and that those selected were of high caliber.

In terms of program design, the majority of PIs (85%) felt the 71% stipends/29% expenditures ratio was appropriate (PI Survey). Program expenditures included salary, administrative expenses, travel, etc. The ratio of CIHR program expenditures to partner expenditures was $1.00:$0.10. Limited indicators related to cost efficiency were included in the scope of the current evaluation due to the decision to sunset the STIHR program.

Most PIs, University Representatives, and trainees interviewed identified many advantages to indirectly funding trainees through an organized training program (although the importance of both indirect and direct funding was emphasized). The most frequent advantages identified by PI survey respondents was greater program coherence and the ability for a more strategic focus versus investigator driven approaches associated with direct funding through individual awards (both 79%). Other advantages identified across target groups^{Footnote 20} included: the strategic allocation of funds, enriched learning experiences, and the creation of a learning community with support for budding researchers. The majority of PI survey respondents indicated that there were no limitations associated with CIHR indirectly funding trainees/students compared to direct funding through individual awards (48%).

Evaluation Issue 2: Program Relevance

Key Findings

• There is continued need and interest for an initiative similar to STIHR.
• STIHR is aligned with CIHR and federal government priorities.

Continued Need for Training Programs Similar to STIHR

There was high interest in STIHR, with PIs and trainees motivated to apply for a variety of reasons. The most frequent reasons why surveyed PIs applied for STIHR funding included the opportunity for capacity development in the Canadian health research community (84%) and the opportunity to develop an innovative training program (78%). The most common reasons surveyed trainees applied was either interest in the research area/topic of STIHR programs (61%) or the associated funding (i.e., financial stipend) (55%). These findings were comparable to those from the 2008 STIHR Evaluation.

The majority of surveyed trainees (83%) were already living in Canada when they decided to participate in programs, while 17% came from another country. Of those trainees who came from another country, 39% said STIHR programs were one of several reasons they chose to come to Canada, followed by 34% who said STIHR programs were the primary reason. These findings were relatively comparable to the 2008 STIHR Evaluation. Taken with those above, these results suggest that STIHR programs were most successful in attracting high quality trainees from Canada, who were in the later stages of their studies (doctoral or postdoctoral) or the early stages of their academic careers (research faculty), conducting health related research and intending to continue conducting research in academic, government or private sectors.

Surveyed trainees felt that the STIHR training very much complemented other academic training already offered (M = 4.13, SD = .99), with minimal duplication and/or overlap (M = 2.02, SD = .98). Interviewed trainees identified several ways STIHR program training complemented their degree program, including modifications to their own research due to: STIHR program involvement (e.g., defined/redefined research question, changed research focus, identification of research supervisors); content exposure to (e.g., more general topics in addition to the subject area of the STIHR program such as trends in the field, career considerations, etc.); and, skills development (e.g., grant writing, intellectual property, laboratory methods).

Consistent with the 2008 Evaluation, the majority of surveyed PIs also felt similar programs were not available at their universities (88%) or in Canada more broadly (81%), and that there was high demand for health researchers with multidisciplinary training and skills both in the subject areas of their STIHRs (81%) as well as in the health community as a whole (69%). Taken together with the findings presented in the sections above (e.g., the high interest in the initiative, trainees’ high ratings of usefulness, satisfaction and improvement in skills related to STIHR programs, benefits to PIs), it is suggested that initiatives providing support for training that goes beyond what is delivered through degree programs can build capacity in targeted areas among trainees.

There is Continued Desire for Funding for Training Initiatives in Health Research from the Federal Government and CIHR

All key informant interviewees felt that financial support, for students completing health research in Canada, from the federal government as well as from CIHR is necessary and should continue. Specifically, trainees and PIs feel that CIHR should continue to provide various awards and funding opportunities for training and capacity building. The most common methods of support identified, for both groups, included doctoral awards, post-doctoral fellowships, and STIHR funding (or similar programs).

More specific and additional suggestions for CIHR’s role in funding and training and capacity building were provided by key informant interviewees. Across groups, key informants felt that CIHR should learn from and share results from STIHR and continue to ensure training programs continue to have a transdisciplinary focus that aims to build capacity in health research. All PIs, trainees, and university representatives interviewed also added that federal financial support is crucial for health research training in Canada. Such support emphasizes the need for high quality research training and federal responsibility speaks to the value that Canadian society puts on research and innovation. One university representative added that researchers are particularly vulnerable in the first 5-7 years after postdoctoral training where gains can be lost without support.

Several key documents were reviewed in order to examine alignment of the objectives of STIHR with CIHR’s priorities. As part of CIHR’s mandate, CIHR was designed to respond to the evolving needs for health research and seeks to transform health research in Canada by:^{Footnote 21}

• funding both investigator-initiated research as well as research on targeted priority areas;
• building research capacity in under-developed areas and training the next generation of health researchers; and
• focusing on knowledge translation that facilitates the application of the results of research and their transformation into new policies, practices, procedures, products and services.

The objectives of STIHR relate directly to CIHR’s mandate and objectives. Within the CIHR Act, a specific objective (4j) of CIHR is to build the capacity of the Canadian health research community through the development of researchers and the provision of sustained support for scientific careers in health research.^{Footnote 22}

CIHR’s Strategic Plan (2009/10-2013/14), Health Research Roadmap outlined four strategic directions, the first of which was investing in world-class research through sustaining a healthy research foundation by: (1) training, attracting and retaining the best talent in health research; (2) providing increased focus on trans-sectoral and multidisciplinary training; and (3) preparing young researchers for non-academic labour markets. Under the activity of providing capacity to attract and retain the best researchers, STIHR was designed with the aim of creating opportunities for successful research teams to strengthen and expand training in cutting-edge research domains by offering training support for health researchers. The ongoing commitment, in CIHR’s current strategic plan (Health Research Roadmap II), under strategic direction 2, focuses on mobilizing health research for transformation and impact. It is CIHRs intent to build, shape and mobilize research capacity to address critical health issues that are important to patients and Canadians, and to maximize health, social and economic impacts through targeted and partnered investments. This will also be achieved through T-SAP and continue to align with the objectives of STIHR.

Conclusions and Recommendations

The evaluation finds that the STIHR model was effective. The convergence of results from multiple lines of evidence (e.g., data sources and methods), despite the heterogeneity of the STIHR programs (e.g., pillar, subject, skills targeted) and the participating trainees (e.g., across degree levels and STIHR programs), strengthened the findings. An early understanding of the strengths of the STIHR model (from the 2008 Evaluation) and findings from the present evaluation were used to inform the development and implementation of CIHR’s T-SAP. Given the decision to no longer launch open STIHR funding; the following recommendations are focused on continuing to inform the T-SAP and future training initiatives.

1. As CIHR implements its Strategic Action Plan on Training (T-SAP), future CIHR approaches to indirect training should be informed by the STIHR model and there should be increased use of the STIHR tool within CIHR’s priority-driven funding.

The evaluation findings suggest that the strategic directions of CIHR’s T-SAP can be well-informed by the STIHR model. The three strategic directions are to develop:

1. research leaders of tomorrow who can lead high-impact, multidisciplinary research in a rapidly evolving environment of advancing technology and globalization;
2. leaders across knowledge sectors who can apply their scholarship and talent to lead innovation across different sectors of Canada’s knowledge-based economy; and
3. experts in critical priority areas who can establish and fill Canadian priority areas of specialized expertise and advance the frontiers of science.

The STIHR model demonstrated success in developing and implementing strategic transdisciplinary training programs, targeting specific skill areas using multiple and mixed training approaches (e.g., face to face, distance, group and tailored approaches), while effectively integrating high quality mentors into the training environment. Specifically, the transdisciplinary training programs provided a variety of collaborative and experiential learning opportunities to trainees (e.g., structured activities requiring trainees to work together, deliberate transdisciplinary supervision/mentoring, exposure to multiple facets of health issues, networking, hands-on/real world experiences), who in turn reported improvements in related skills (i.e., networking and transdisciplinary research). These findings hold promise for T-SAP directions aimed at building research leadership while capitalizing on technology.

Through requirements embedded in program design, almost all STIHR programs ensured training on the ethical conduct of research and related ethical issues, knowledge translation and professional skills (e.g., scientific communication, grant writing, career planning), all areas of interest to CIHR’s mandate and the T-SAP. This high compliance suggests that the STIHR model could be adapted to expand the scope of professional skills delivered to trainees in order to address T-SAP directions related to leadership across knowledge sectors. STIHR programs resulted in modest improvements in collaborative research with private, government and not-for-profit sectors as well as international research collaborations. Given T-SAP directions, these areas could also be deliberately expanded in any future iterations of a STIHR-like initiative.

One of the elements of the STIHR model included indirectly funding trainees (via stipends distributed by grantees, in addition to direct funding via awards and scholarships trainees may be receiving external to STIHR). The advantages of this approach were greater program coherence and the ability for a more strategic focus versus an investigator driven focus associated with direct funding through individual awards. The STIHR model has also demonstrated success in reaching a large number of high caliber trainees (both in funded and unfunded capacities), from a variety of disciplines, actively engaged in scientific production, contributing relatively large numbers of a variety of outputs, and intending to continue conducting health research in academic, government or private sectors. Given the interest of T-SAP in building capacity in critical priority areas, the STIHR model may be an effective mechanism to support strategic training objectives using a distributed approach to training.

2. CIHR should continue to provide support for training that adds value beyond stipends and what is delivered through degree programs.

Although the evaluation did not include a comparison to other training models, the strength and convergence of findings suggest that trainees experienced considerable benefit by being engaged in STIHRs above and beyond what they would have received through their degree program alone. In addition to providing transdisciplinary training in transferable skills, access to multiple mentors, and additional funding, STIHR programs offered different types of opportunities to support the employability of trainees, which were rated as being very useful. Just over half of the programs systematically exposed trainees to career options and preparation activities. PIs felt that as a result of STIHR programs, trainees headed into academic careers were provided with opportunities beyond regular degree programs (e.g., working with multiple supervisors/mentors, networking, and access to additional funding). These findings suggest that the purposeful integration of activities and resources related to employability and career development add value and benefit trainees’ research and career development.

Beyond trainees, PIs perceived several benefits from the leadership of STIHR programs through the development of teaching resources and research productivity, with the main challenges being an increased workload in order to manage their programs, and the level of effort needed to maintain them (e.g., ensuring the recruitment and participation of mentors, administrative burdens), unexpected costs (e.g., travel, securing funds for sustainability) and challenges working across departments and faculties. Future initiatives could therefore consider incentivizing support and resources for the leadership and/or management of the delivery of training activities.

3. Future CIHR training initiatives must have stronger performance measurement.

CIHR is committed to strong performance measurement and evaluation activities, as demonstrated in its current strategic plan.^{Footnote 23} In order to meet these commitments, future training initiatives that take a distributed approach to funding trainees (i.e., where grant dollars are given to PIs to administer to trainees), should be required to report trainee-level contact information and data to CIHR for tracking and follow-up in evaluations. Going forward, CIHR should also develop common reporting processes to ensure consistency between reporting templates used for midterm and end of grant reports both within and across granting opportunities. Performance measurement strategies should extend beyond outputs, when appropriate and possible, to ensure that outcomes and impacts can be measured and used to inform evaluation plans and indicators. This approach would ensure similar data across different funding approaches and would enable comparative evaluation going forward.

Although only a small sample of partners was interviewed, the evaluation found that partners had considerable interest in the outcomes of STIHR programs and that there were different approaches to communicating with partners about progress and results at the initiative and program levels. For future initiatives, CIHR could consider adapting communication approaches used by programs to ensure partners are engaged and well-informed.

4. CIHR should identify, plan for and communicate sustainability expectations and/or requirements beyond grant funding.

On average, STIHR programs secured about 38% of the total funding needed to continue the existing programs after the end of the grants from CIHR in their current form. There was unanimous disappointment in the decision to no longer launch open STIHR funding opportunities and the sustainability of programs (in their current form), beyond courses and relationships built, was highly unlikely for the majority. The continuity of courses and relationships for some STIHR programs was more successful due to the fact that there were existing systems in place that made this transition easier (e.g., a university setting with formalized processes for delivering education is an existing system whereby new courses can easily be incorporated). Barriers to sustainability in the current evaluation included accessing financial resources/limited funding sources, and continued commitment (or lack thereof) from partner universities and mentors.

Sustainability and/or renewal intentions should be clearly communicated to grantees by CIHR. The process of sun-setting STIHR by CIHR and transitioning to a broader training strategy, as well as the desire of researchers and trainees for the continuation of training programs aimed at capacity building and transdisciplinary research, highlighted the need for purposeful planning, definition, and clear communication about program objectives including sustainability plans and requirements. This includes offering clear definitions for sustainability and parameters for CIHR funding renewal plans in the design of training initiatives. Within funded programs, this should include whether and how individual grant-level partners, including host institutions, will be involved in supporting sustainability of programs overall or of specific elements (e.g., courses).

Appendix A – STIHR Investment Details

Ratio of CIHR:Partner Contribution - $1.00:$0.10

Appendix B – STIHR Partners

Note. Partners were identified through mentions by PIs in the progress reports and/or in the PI Survey. In progress reports, roles included co-funding, trainee funding, provides funding, collaborator, funding workshop/training session.

References

Canadian Institutes of Health Research, Health Research Roadmap II: Capturing Innovation to Produce Better Health and Health Care for Canadians, Strategic Plan 2014-15 – 2018-19, 2015.

Canadian Institutes of Health Research, Canadian Institutes of Health Research Act, S.C. 2000, c.6, 2000.

Canadian Institutes of Health Research, Our Mandate, 2013.

Canadian Institutes of Health Research, Strategic Training Initiative In Health Research (STIHR) 2001-2006: Final Evaluation Report (August 2008), available upon request.

Treasury Board Secretariat, Evaluation Policy Suite, April 2009, 2012.

Date modified:

2017-04-07