Evaluation of the Strategy for Patient-Oriented Research

Final Report

Table of Contents

• Executive Summary
  • Overview
  • SPOR Context and Profile
  • Findings
  • Recommendations
• 1. Introduction
  • 1.1 SPOR Profile
  • 1.2 Core Elements of SPOR
    • 1.2.1 SUPPORT Units
    • 1.2.2 SPOR Networks
    • 1.2.3 Capacity Development
    • 1.2.4 Clinical Trials
    • 1.2.5 Patient Engagement (PE)
  • 1.3 Target Populations
  • 1.4 Stakeholders
  • 1.5 Governance
  • 1.6 CIHR Institute Involvement in SPOR
  • 1.7 Resources
  • 1.8 Evaluation Purpose and Scope
  • 1.9 Evaluation Questions
    • SPOR Evaluation Issues and Questions
  • 1.10 Report Structure
• 2. Methodology
  • 2.1 Data Sources
    • 2.1.1 Document Review
    • 2.1.2 Performance Data Review
    • 2.1.3 Key Informant Interviews
    • 2.1.4 Case Studies
    • 2.1.5 Governance and Administrative Network Analysis
    • 2.1.6 International Comparative Review
    • 2.1.7 Cost efficiency analysis
  • 2.2 Data Analysis
  • 2.3 Strengths and Limitations of the Evaluation
    • 2.3.1 Strengths
    • 2.3.2 Limitations
• 3. Evaluation Findings
  • 3.1 Relevance
    • 3.1.1 Key Findings:
    • 3.1.2 Analysis
      • 3.1.2.1 Addressing the need for evidence-informed health care (Evaluation Question #1)
      • 3.1.2.2 Roles and responsibilities (Evaluation Question #2)
      • 3.1.2.3 SPOR’s alignment with government priorities (Evaluation Question #3)
    • 3.1.3 Conclusions
  • 3.2 Design and Delivery
    • 3.2.1 Key Findings
    • 3.2.2 Analysis:
      • 3.2.2.1 Design and delivery at the strategy level (Evaluation Question #4)
      • 3.2.2.2 Design and delivery of core elements (Evaluation Question #4)
    • 3.2.3 Conclusions
  • 3.3 Performance
    • 3.3.1 Key Findings:
    • 3.3.2 Analysis
      • 3.3.2.1 Progress towards Immediate Outcomes (Evaluation Question #5)
      • 3.3.2.2 Movement toward longer term impacts (Evaluation Question #6)
    • 3.3.3 Cost-efficiency Analysis (Evaluation Question #7)
    • 3.3.4 Conclusions
• 4. Recommendations
• Appendices
  • Appendix A – SPOR Logic Model
  • Appendix B – CIHR Institutes: Engagement with SPOR

This report has been prepared by KPMG LLP (“KPMG”) for the Canadian Institutes of Health Research ("Client") pursuant to the terms of our engagement agreement with Client dated August 10, 2015 (the “Engagement Agreement”). KPMG neither warrants nor represents that the information contained in this report is accurate, complete, sufficient or appropriate for use by any person or entity other than Client or for any purpose other than set out in the Engagement Agreement. This report may not be relied upon by any person or entity other than Client, and KPMG hereby expressly disclaims any and all responsibility or liability to any person or entity other than Client in connection with their use of this report.

© 2016 KPMG and the KPMG logo are registered trademarks of KPMG International, a Swiss entity

List of Acronyms

ACAHO

Association of Canadian Academic Healthcare Organizations

ACCESS

ACCESS Open Minds

CCTCC

Canadian Clinical Trials Coordinating Centre

CD

Capacity Development

CIHR

Canadian Institutes of Health Research

CLAHRC

Collaboration for Leadership in Applied Health Research and Care, UK

CT

Clinical Trials

EAC

SPOR's External Advisory Committee on Capacity Development

ED

Executive Director (typically for SUPPORT Units)

FP7

Seventh Framework Programme of the European Union

GBF

Graham Boeckh Foundation

HC

Health Canada

KI

Key Informants

MRC

Medical Research Council, UK

NICE

National Institute for Health & Clinical Excellence, UK

NSC

SPOR National Steering Committee

PAA

Program Alignment Architecture

PCORI

Patient-Centered Outcomes Research Institute

PE

Patient Engagement

PHAC

Public Health Agency of Canada

PIHCI

Pan-Canadian SPOR Network in Primary and Integrated Health Care Innovations

PMS

Performance Measurement Strategy

POR

Patient-Oriented Research

REB

Research Ethics Board

SPOR

Strategy for Patient Oriented Research

TPMI

Newfoundland Translational and Personalized Medicine Initiative

Executive Summary

Overview

This report presents the findings from the evaluation of the Canadian Institutes for Health Research (CIHR) contributions to Canada’s Strategy for Patient-Oriented Research (SPOR). Although many of the SPOR partners were engaged in the evaluation, the evaluation was of CIHR’s contribution to SPOR only.

The SPOR evaluation covers the period from 2010-11 to 2015-16 and was scoped to address the evaluation elements outlined by Treasury Board related to program relevance, design and delivery, and performance to meet accountability requirements and additionally to inform program decision-making. Due to SPOR’s early stage of evolution, the evaluation focused on the deployment of SPOR core elements and short-term progress towards objectives.

SPOR Context and Profile

Patient-oriented research (POR) refers to a continuum of research that engages patients^{Footnote 1},^{Footnote 2} as partners, focuses on patient priorities and improves patient outcomes individually and in communities such as vulnerable populations. This research, conducted by multidisciplinary teams in partnership with relevant stakeholders, aims to apply the knowledge generated to improve healthcare systems and practices. It involves ensuring that the right patient receives the right clinical intervention at the right time, ultimately leading to better health outcomes.^{Footnote 1}

Announced in 2011, SPOR is a ten-year national Strategy. It encompasses five core elements:

1. SUPPORT Units: specialized, multidisciplinary research service centres located in provinces and territories across Canada;
2. Networks: pan-Canadian research networks that represent a collaboration of patients, health service providers, policy/decision makers, and health researchers;
3. Capacity development: targets activities for training and support for POR;
4. Patient engagement: supports efforts to engage patients in a meaningful way through active collaboration in governance, priority setting, and the conduct of research; and,
5. Clinical trials: an element targeted to improve the clinical trials environment in Canada, primarily through the Canadian Clinical Trials Coordinating Centre.

While CIHR plays a leadership role within SPOR, it engages a broad coalition of stakeholders from across Canada.

Each SPOR element has been developed and implemented within varying timeframes. Although SPOR planning, design, partnership development, and foundational elements have been in place since its creation in 2010-11, several SPOR core elements have been in place for less than two years, with most only being active for one year at the time of the evaluation. Overall, SPOR is still in early stages of being implemented to scale and the results of this evaluation, particularly results related to intended outcomes, need to be interpreted with this condition in mind.

Findings

The evaluation found that SPOR is relevant to addressing Canadian health systems needs for integrating research into care. Key findings include:

• SPOR encompasses the elements and activities required to address areas in need of support for POR, such as enhancing access to common resources, increasing collaborations among stakeholders and facilitating patient involvement.
• SPOR is consistent with federal government and CIHR priorities, being a direct extension of the Canadian Institutes of Health Research Act, which specifies CIHR’s mandate to improve the health of Canadians.
• SPOR is aligned with international trends towards employing POR models and approaches.
• It remains important for the federal government, through CIHR, to play a role in advancing the POR model in Canada.
• SPOR complements POR activities undertaken by the provinces and territories, helping to fill gaps in current programming, resources, methods and capabilities.

Some elements of the design and delivery processes for SPOR were found to be effective, but areas for further improvement were also identified.

The design and delivery elements found to be effective include:

• National governance mechanisms are generally strong with key success factors being the national composition that facilitates inter-jurisdictional learning and promotes a “let’s discuss together” approach.
• The consideration of provincial/territorial priorities and needs in the development of the SUPPORT Units in particular are a strength of the flexible approach CIHR has taken.
• Stakeholder engagement is believed to be “exceptional” in the views of most stakeholders and SPOR has been successful at developing relationships and reaching a broad group of stakeholders across sectors and participant types.
• Peer review processes are strong with the inclusion of patient representatives being a positive and novel approach.
• The extensive and iterative process for Network development and selection has been effective and has continuing effects on even those projects that were not selected for funding (i.e., groups not funded are still working together).

There are six areas in which effectiveness can be improved, including:

• Communications overall require improvement, including the need for further clarification of the mandates for each of the SPOR core elements, communication of support available from the SUPPORT Units, consistent and common definitions of POR and patient engagement (PE), and tailored communications targeted to different stakeholders.
• Most stakeholders identified a level of uncertainty regarding “what’s next,” particularly with respect to the limits of the initial the five-year funding term. This highlights the need for greater communication regarding plans for future funding beyond initial commitments.
• Patient engagement is still a work in progress with enhancements needed to support training and mentoring, as well as determining best practices for engaging patients. This includes strengthening SPOR’s PE core element through increased work on capacity development (CD) (patients learning how to participate in research, and researchers learning how to effectively engage patients) as well as expanding national efforts to support more consistent and well-communicated development of PE and CD.
• There has been a low level of understanding and slow uptake by some areas/members of the research community who are not convinced of the value of POR or PE, mainly related to the lack of hard evidence as to its impact on the quality of research or the importance of the resulting clinical outcomes.
• There is a lack of clarity regarding SPOR’s many elements at both the national and provincial/territorial levels. As a result, the alignment of priorities and activities, how various capabilities and services are to be integrated and leveraged, and how all elements work together going forward could be better defined and are confusing to many participants.
• Performance measurement was consistently identified by stakeholders as needing attention. It is not clear the right things are being measured (i.e., considerable measurement of activities and limited measurement of outcomes) and the collection and reporting of performance data is believed to be quite onerous for funded recipients.

Although SPOR is in its initial phases of implementation, it has advanced toward achievement of its stated immediate outcome areas. The main areas of achievement include:

• SPOR is now committing funding at full scale to its core components and has leveraged partner dollars, meeting is own requirement for 1:1 matching and in most cases exceeding this ratio.
• Six SUPPORT Units have been established, many of which are putting high priority on building data platforms. SUPPORT Units have increased their dataset holdings from 253 to 534 over a two-year period and have collectively provided consultation and research services reaching 1,872 stakeholders.
• Demonstration projects undertaken by the SUPPORT Units are showing early signs of POR research outputs, with examples of success in using Big Data in Newfoundland & Labrador, in engaging patients in Ontario, and in the development of a best practice clinical service protocol in Manitoba.
• The Canadian Clinical Trials Coordinating Centre has been operational since early 2014 and has delivered against two of its key objectives with the completion of a model clinical trials agreement and a clinical trials asset map.
• Three national research Network funding opportunities (Mental Health; Primary and Integrated Health Care Innovation, and Chronic Diseases) have been launched, resulting in seven Networks funded.
• SPOR has engaged stakeholders across sectors and across participant communities with extensive efforts at the national and provincial/territorial levels, bringing researchers, clinicians, policy makers and patients together across sectors and geographies.

In terms of progress toward intermediate outcomes, SPOR is early in its influence on the health research landscape and shifting the culture towards POR. The alignment and integration of the activities of SPOR’s core elements, enhanced engagement of patients, stronger communication of key concepts, future plans and priorities, and early wins will help to increase buy-in for POR and support the further implementation and achievement of intended results.

Recommendations

Overall, SPOR is relevant, addresses an ongoing need for POR, and is demonstrating expected performance given the stage of implementation, supporting the need for continued involvement and investment in SPOR by CIHR; however, some improvements are required to further strengthen its design and delivery.

The following recommendations were identified to improve program performance:

1. CIHR should increase efforts to strengthen SPOR’s role in a common agenda for change to patient-oriented research.
   • CIHR needs to continue to focus on increasing buy-in of POR and changing the culture, identify and communicate best practices in POR and enhance communication to clarify definitions of many POR terms, including POR itself.
   • SUPPORT Units need to increase communications and outreach to their broad stakeholder community in relation to SUPPORT Unit services available and initiatives undertaken.
2. CIHR should provide strategic guidance regarding how SPOR elements are to work together toward achieving SPOR’s intermediate and long-term outcomes.
   • CIHR, in collaboration with its established SPOR governance structures, should enhance guidance on operationalizing SPOR elements, in particular, clarifying how elements are expected to work and coordinate together.
3. CIHR should communicate plans for moving beyond the initial five year funding period to manage sustainability expectations for CIHR investments in SPOR.
   • CIHR needs to provide clear communications regarding SPOR funding, and options beyond the current five-year funding commitment to some elements.
4. CIHR should strengthen approaches to enable cross-learning, sharing of best practices, and collaboration; this should occur within and across SPOR elements and between CIHR and Canadian and international organizations.
   • CIHR should re-examine the structure, operations and effectiveness of working groups, and encourage cross-provincial initiatives, particularly among SUPPORT Units.
   • CIHR and all SPOR elements should encourage ongoing interaction/connection and relationship building with other POR initiatives.
5. CIHR should continue to support effective management and administrative functions within funded SPOR SUPPORT Units and Networks and across these elements.
   • CIHR should require SPOR SUPPORT Units and Networks to be supported by CEO/COO-type management positions, if not already present, to help manage operational obligations, administrative requirements and the high corresponding workloads in these areas.
   • CIHR should review the funding model in place, and adjust funding flow based on the stage of development/need of the element.
6. CIHR should revise the existing SPOR performance measurement strategy to balance administrative/operational outputs with outcomes/impacts.
   • Indicators should be re-oriented from tracking primarily activity-based or output indicators toward outcomes and impacts; consider applying a “collective impact” lens.
   • CIHR should improve its financial monitoring and coding for SPOR grants and awards expenditures (including partner contributions) and for operating and maintenance expenditures.

1. Introduction

This report presents the findings from the evaluation of the Strategy for Patient-Oriented Research (SPOR). This report includes a summary description of SPOR, the approach and methodology used for completing and analysing the data sources supporting the evaluation, and the presentation of concluding findings from all lines of enquiry. Key findings are drawn against the applicable evaluation questions relating to SPOR’s relevance, design and delivery, and performance.

1.1 SPOR Profile

Patient-oriented research (POR) refers to a continuum of research that engages patients^{Footnote 1},^{Footnote 3} as partners, focusses on patient-centric priorities and improves patient outcomes individually and in communities. This research, conducted by multidisciplinary teams in partnership with relevant stakeholders, aims to apply the knowledge generated to improve healthcare systems and practices.^{Footnote 4}

POR can range from initial studies in humans to comparative effectiveness and outcomes research, and the integration of this research into the health care system and clinical practice. The goal of POR is to better ensure the translation of innovative diagnostic and therapeutic approaches to the point of care (i.e., to patient diagnoses and treatments), so as to help ensure greater quality, accountability, and accessibility of care. It aims to ensure that “the right patient receives the right clinical intervention at the right time, ultimately leading to better health outcomes.” While clinical research has provided the basis for the development and application of health interventions, comparative evaluations of these interventions have lagged in their ability to provide guidance as to when and for which patients to apply them. Such results in clinical research have led to a push internationally towards POR, which has become a high priority for international peer organizations including those in the United States, United Kingdom, and Australia.

Various factors in the Canadian environment throughout the turn of the century have added impetus to the need to move forward on the development and implementation of a comprehensive national strategy for POR. First, while investments in health research have led to the development of a vast array of preventive, diagnostic and treatment interventions for health, there has been increasing impatience among clinicians, policy makers and patients with the slow pace at which scientific discovery has resulted in new products or interventions. Second, significant gaps in high quality evidence on comparative effectiveness have shown that it can be difficult to establish guidelines for appropriate care. Third, in tighter economic times, funders of basic biomedical research, including federal and provincial governments and health charities, have been anxious to see and to explain to taxpayers and donors the public benefit of the billions of dollars invested in scientific research.

In response to this identified need, Canada’s Strategy for Patient-Oriented Research was developed to set out a vision about how Canada’s POR model will be designed. SPOR’s Strategy document depicts the Canadian landscape as, in effect, facing a dual challenge in the research-to-practice continuum. The strategy states this often being referred to as the two "Death Valleys". Valley 1 refers to the decreased capacity to translate the results of discoveries generated by basic biomedical research in the laboratory to the bedside or careside as well as to successfully commercialize health discoveries. This negatively impacts Canada's clinical research and knowledge base and its international competitiveness. Valley 2 refers to the limited capacity to synthesize, disseminate and integrate research results more broadly into health care decision-making and clinical practice.^{Footnote 1}

Exhibit 1 – The Two Valleys of the Research-to-Practice Continuum

Low capacity to translate the results of discoveries generated by basic biomedical research in the lab to the bedside as well as to successfully commercialize health discoveries. Bridging Valley 1 by fostering public-private partnerships and by providing clinical validation of research results.

Limited capacity to synthesize, disseminate and integrate research results more broadly into health care decision-making and clinical practice. Bridging Valley 2 by ensuring that research results are integrated into the healthcare system.

Adapted from Steven Reis, University of Pittsburgh and Harold Pincus, Columbia University.

In response to these pressures, Canada’s Strategy for Patient-Oriented Research was developed with the vision to demonstrably improve health outcomes and enhance patients' health care experience through integration of evidence into the health care system and clinical practice. SPOR was first announced in August 2011. It is expected that through the integration of research evidence into clinical practices, the act of putting patients first would support Canada in ensuring that research will have a greater impact on treatments and services provided in clinics, hospitals and doctors' offices throughout the country. SPOR represents a shared agenda among federal, provincial and territorial partners dedicated to the integration of research into care.

The immediate and intermediate outcomes defined for SPOR include:

An important principle of SPOR is the leveraging of non-federal partner funds to match, at a 1:1 ratio, the investments made from the federal government toward the core elements of SPOR. To date, $357 million^{Footnote 5} has been invested or committed by CIHR.

Several additional principles are in place for the federal investments in the core elements of SPOR^{Footnote 6}.The principles include:

• Patients need to be involved in all aspects of research to ensure questions and results are relevant and integrated into practice;
• Decision-makers and clinicians need to be involved throughout the entire research process to ensure integration into policy and practice;
• Effective patient-oriented research requires a multi-disciplinary approach;
• SPOR is focused on first-in-human (and beyond) research designed to be transformative in nature and improve patient outcomes and/or the effectiveness and efficiency of the health care system; and,
• SPOR is outcome driven and incorporates performance measurement and evaluation as integral components of the initiative.

1.2 Core Elements of SPOR

SPOR consists of five core elements, with each element having been structured to address specific challenges identified as having delayed or prevented the translation of high quality research to improvements in patient outcomes within Canada. These elements include:

• SUPPORT Units
• SPOR Networks
• Capacity Development (CD)
• Improving the Clinical Trials Environment
• Patient Engagement (PE)

SPOR was initially launched in 2011, with the release of the formal Strategy document. In the five years since that time, each SPOR element has been developed and implemented^{Footnote 7} within varying timeframes. For the most part, SPOR core elements have been in place for a maximum of two years, with most only being active for one year.

Official program documents indicate there was a planned delay through the first two years of SPOR with activities identified for 2011-12 to include the establishment and engagement of SPOR’s National Steering Committee, the determination of priorities, the creation of the funding opportunities for the Networks and SUPPORT Units, as well as some specified work in the Clinical Trials area.

Additionally, there are a number of relevant and important programs preceding SPOR that helped to set the stage/prime the Canadian clinical and patient-oriented research community for the launch and implementation of SPOR. SPOR has a number of Foundational Investments associated with it (e.g., operating grants, catalyst grants, knowledge synthesis grants. These Foundational Investments are aligned with SPOR; however, may have begun prior to SPOR being announced, maintained during the design and implementation of SPOR, or were sunsetted post-SPOR implementation.

For example, there are a number of Foundational Investments that were, and continue to be, important, as most play a role in the development of SPOR-relevant research capacity and are directly aligned with SPOR. For example:

• One of the stated vision-achieving goals of SPOR is to grow Canada's capacity to attract, train and mentor health care professionals and health researchers, as well as to create sustainable career paths in patient-oriented research. Through the Foundational Investments, CIHR continued to support:
  • MD/PhD students through the MD/PhD Program Grants;
  • Students pursuing health professional degrees (often at the undergraduate level) to engage in research (usually during the summer) through the Health Professional Student Research Awards program;
  • Trainees at the doctoral and post-doctoral level as well as new investigators through priority announcements on the relevant CIHR open award programs in Clinical Research; and
  • Mentoring of clinical trial trainees through the Randomized Controlled Trials Mentoring program.

These Foundational Investments are not included in the evaluation’s scope.

Exhibit 2 depicts the progressive implementation of the core SPOR elements.

Exhibit 2 – Timeline of SPOR Elements and Key Documents since 2011

The description and evolution of each core SPOR element is described in the following sections.

1.2.1 SUPPORT Units

Support for People and Patient-Oriented Research and Trials (SUPPORT) Units are locally accessible, multidisciplinary groups of specialized research resources, policy knowledge, and patient perspectives. SUPPORT Units have been created to provide support and expertise to those pursuing POR. Additionally, they are tasked with facilitating decision-making within the health services setting, fostering the implementation of best practices, and promoting collaboration among researchers engaged in POR.

SUPPORT Units have been or are being established in collaboration with the provinces and territories, who have a significant role in directing the work they carry out.

The concept of SUPPORT Units was mapped out at a one-day meeting in March 2011 that included researchers, clinicians, policy makers, and CIHR representatives. In 2012, CIHR made available to all provinces/territories, a funding opportunity that outlined a set of instructions to assist with the development of jurisdictional SUPPORT Unit business plans. The first SUPPORT Units were approved for funding in 2013 and spent much of that year finalizing strategies and work plans.

Each SUPPORT Unit has committed to the following six key components (these are CIHR requirements):

• Data Platforms and Service that enhance access to large administrative data sets, train and provide data analysts to respond to requests for de-identified data, provide a central platform for primary data collection (Alberta), and provide data ambassadors to deal with data sharing agreements, privacy, ethics and consent.
• Methods Support and Development that enhance access to expertise such as biostatistics, epidemiology, clinical trial design, survey methods and health economics.
• Health Systems Research, Implementation Research and Knowledge Translation which include activities that put knowledge to action, enhancing uptake.
• Pragmatic Clinical Trials that support common services such as ethics review and streamlined legal services.
• Career Development in Methods and Health Services Research that offer capacity building in health services research fields (e.g., biostatisticians, health economics) through academia, mentorship and leadership supports.
• Consultation and Research Services which provide service support to researchers in areas such as methodological development, administrative needs, project management, and data management.

Each SUPPORT Unit must also undertake a demonstration project or projects. These are meant to show the benefits of POR, and demonstrate quickly that the POR model can work.

To date, CIHR has invested $68M^{Footnote 8} in SUPPORT Units (Alberta, Manitoba, Maritimes, Ontario, Newfoundland & Labrador, and Quebec) and will invest $127M^{Footnote 9} over the next five years to roll out the remaining SUPPORT Units (British Columbia, Saskatchewan, and three territories). The implementation stage of the SUPPORT Units as of February 2016 is:

• SUPPORT Units implemented:
  • 2013: Alberta, Manitoba, Maritimes, Ontario
  • 2014: Newfoundland & Labrador, Quebec
• SUPPORT Units approved and moving towards implementation:
  • 2015-16: British Columbia, Saskatchewan and Northwest Territories
• SUPPORT Units in ongoing discussions:
  • Yukon and Nunavut.

A SUPPORT Unit Council (SSUC) was established in 2013. The mandate of the SSUC is to provide an opportunity for information sharing and collaboration among SUPPORT Units throughout the development and implementation of this element of SPOR. It does not have powers to compel actions.

The SSUC is comprised of two co-chairs, one from CIHR and one from the SUPPORT Units, one lead person from each SUPPORT Unit and two other members from CIHR. The SSUC Terms of Reference identifies they have been meeting four to five times per year.

The SSUC focuses its attention on the following:

• Sharing information, best practices, tools and lessons learned;
• Advocating for the Strategy for Patient-Oriented Research;
• Directing communications between SPOR partners and SUPPORT Unit stakeholders;
• Serving as a sounding board for CIHR and other SPOR partners for new initiatives;
• Coordinating SUPPORT Unit responses to multi-jurisdictional issues arising when interacting with SPOR Networks; and
• Directing the work of the SSUC working groups^{Footnote 10}.

SSUC Working Groups (WGs) were formed in 2014 and 2015 to address issues of common interest and are designed to focus on a particular issue. The WGs report to the SSUC and have been established as follows:

• Performance Measurement Working Group (2014);
• Knowledge Translation Working Group (2014);
• CD Working Group (2015);
• PE Working Group (2014); and
• Data Working Group (2015).

1.2.2 SPOR Networks

SPOR Networks are national collaborative research networks involving the full range of SPOR stakeholders (patients, health professionals, decision makers, health researchers and other stakeholders). They focus on specific health challenges identified as priorities in multiple provinces and territories. They are intended to pursue research and generate evidence and innovations designed to improve patient health and health care systems.

In January 2011, CIHR hosted a round-table meeting to discuss the vision for the pan-Canadian Networks. The meeting’s goal was to determine how Canadian clinical research networks should be best organized, governed, and resourced. The specific goals of the roundtable were:

1. To understand what constituted a Canadian Patient-Oriented Research Network and how it might best function.
2. To share best practices and realities in developing and running national research networks.
3. To provide CIHR with guidance on the development and implementation of future Canadian Patient-Oriented Research Networks and information needed to develop funding opportunities related to them.

Through the SPOR development process, CIHR solicited stakeholder input on research areas of greatest need. Based on the input received, the SPOR National Steering Committee (NSC) then helped to refine the areas in which SPOR funding should be targeted for the creation of Networks.

The application process has been different for each of the Networks. The process may include a combination of the following: expressions of interest, letters of intent, strengthening workshops and full applications. The purposes of the strengthening workshops are to strengthen applications and facilitate dialogue between applicants, potentially identifying synergies among applicants. In some instances, funding is allocated to applicants through the letter of intent stage to facilitate development of the full application.

To date, seven networks^{Footnote 11} in three different areas, described below, are being implemented to deliver on the SPOR objectives. These networks are pan-Canadian initiatives.

From 2010-11 to 2020-21, CIHR has committed $83.2 for Networks, of which $12.6M had been spent during the timeframe of the evaluation (2010-11 to 2015-16).

SPOR Network in Youth and Adolescent Mental Health – ACCESS Open Minds

The SPOR Network in Youth and Adolescent Mental Health, with delivery through ACCESS Open Minds^{Footnote 12}, aims to bring about transformational change in addressing adolescent and youth mental health and well-being. The Network seeks to improve the care provided to young Canadians with mental illness through assisting in connecting patients and youth with researchers, health care professionals, and decision-makers in order to foster the translation of research findings into practice and policy.

The funding opportunity for the Youth and Adolescent Mental Health Network was launched in January 2013. In June 2014, the ACCESS Open Minds Canada research network for youth and adolescent mental health was announced. The Network is a collaborative effort between CIHR and the Graham Boeckh Foundation.

Pan-Canadian SPOR Network in Primary and Integrated Health Care Innovations (PIHCI)

In November 2013, the funding opportunity for the Primary Integrated Health Care Innovations (PIHCI) Network was launched. PIHCI entered Phase I implementation in 2014. Phase 2 of PIHCI proceeded through 2015 with Quick Strike research projects^{Footnote 13} and applications for full membership in the pan-Canadian Network.

This pan-Canadian Network (a network of networks) is intended to support an alliance between research, policy and practice to create responsive learning networks. It is intended to deal with delivery of care both within and across sectors of health care (e.g., public health, home and community care, primary, secondary, and tertiary care) and outside the health sector (e.g., education, social services).

PIHCI is a key CIHR initiative under SPOR and also the Community-Based Primary Health Care Signature Initiative (CBPHC). A core requirement for PIHCI Network membership necessitates members to link with CBPHC Innovation Teams in a member’s jurisdiction.

SPOR Networks in Chronic Disease

In October 2014, the funding opportunity for the Networks in Chronic Diseases was launched. The focus of these networks is on the translation of existing and new knowledge generated by basic biomedical, clinical, and population health research into testing of innovations that can improve clinical science and practice and foster policy changes, leading to transformative and measureable improvements in patient health outcomes, and in efficient and effective healthcare delivery within five years.

Applications were submitted in early October 2015, with results released in late February 2016. Five networks were funded in March 2016. As these networks were in the applications phase at the time of this evaluation, they were purposely excluded from the evaluation scope.

1.2.3 Capacity Development

Capacity development is intended to grow, support and sustain the capacity for a collaborative, interdisciplinary and innovative patient-oriented research environment capable of addressing evolving health care questions, contributing to enhancing patients' health care experience and improving health outcomes.

In August 2012, an External Advisory Committee (EAC) was mandated to develop a report identifying the deficiencies in patient-oriented research within Canada. In 2013, a national workshop was held to collect feedback on barriers to patient-oriented research. The EAC drafted guiding principles and suggestions for implementation and released its report “Training and Career Development in Patient-Oriented Research” in June of 2013. Further consultation with SPOR stakeholders including those that are involved directly in building capacity in a variety of sectors took place through a workshop on capacity development in March 2014, with a final CD Framework being released in August 2015.

The SPOR CD Framework^{Footnote 14} was designed to encourage a shared vision, key principles, and considerations for capacity development in POR. In alignment with this framework, training, mentoring, and career support is to be integrated into the SPOR Networks and SUPPORT Units, with each Network and SUPPORT Unit being required to articulate a training and capacity development strategy.

1.2.4 Clinical Trials

An important goal of SPOR is to strengthen organizational, regulatory, and financial support for clinical trials in Canada and enhance patient and clinician engagement in these studies. Implementing and funding multi-centre clinical trials is difficult in the current Canadian context, and Canada is perceived to be losing its competitive edge. To improve Canada's competitiveness in conducting clinical trials, SPOR has developed this element, which is designed to help overcome a number of identified barriers. The first ever Canadian Clinical Trials Summit was held on September 15, 2011 in partnership by CIHR, HealthCareCAN (formerly the Association of Canadian Academic Healthcare Organizations), and Innovative Medicines Canada (formerly Canada’s Research-Based Pharmaceutical Companies (Rx&D)). The purpose of the Summit was to include various public, private and academic participants in the development of an action plan to “further guide academia and clinical sites, government, and industry on a common path” in order to “help Canada regain its capacity to attract clinical trials.” The first recommendation of the Action Plan called for the development of “a national headquarters for clinical trials improvement activities”. The Canadian Clinical Trials Coordinating Centre (CCTCC) was subsequently established through a joint public, private, and academic partnership between CIHR, Innovative Medicines Canada, and HealthCareCAN in 2014. The CCTCC is working to implement the recommendations from the 2011 Clinical Trials Summit Action Plan^{Footnote 15}, which were to:

1. Establish implementation and coordination headquarters and resources;
2. Measure, monitor, manage, and market clinical trial performance improvements;
3. Integrate health system and research infrastructure to ensure quality and sustainability;
4. Improve efficiencies of ethics reviews and advance strategic issues;
5. Develop a database of registries and consider a national patient recruitment strategy;
6. Adopt common Standard Operating Procedures, training and certification;
7. Improve and use the common clinical trials contract;
8. Optimize intellectual property protection policy, scientific research and experimental development tax credits; and
9. Signal Canada’s interest globally.

Additionally, the clinical trials element is incorporated as a core function of each provincial/territorial SUPPORT Unit, as well as through the clinical trials activities conducted across the SPOR Networks, and through the engagement of clinicians and clinical researchers across the SPOR governance and reporting structures.

1.2.5 Patient Engagement (PE)

By encouraging a diversity of patients to tell their stories, new themes may emerge to guide research. Patients are expected to gain many benefits through their involvement in research, including increased confidence and mastery of new skills, access to information they can understand and use, and a feeling of accomplishment from contributing to research relevant to their needs.

A key objective of Canada's SPOR is for patients, researchers, health care providers, and decision-makers to actively collaborate to build a sustainable, accessible and equitable health care system. Experience has shown that the priorities of clinical researchers and healthcare systems may not always perfectly match the priorities, concerns, and needs of patients.

Engaging patients is therefore a vital element to be integrated with the development and implementation of all elements of SPOR, such as SUPPORT Units^{Footnote 16} and Networks. There is also activity in sharing knowledge on PE across SPOR elements. For example, the SUPPORT Units are noted to be actively sharing and involved in CIHR-sponsored workshops on the topic.

In response to feedback regarding the need to clarify SPOR’s approach to PE from a variety of sources, including the research community, SPOR's National Steering Committee requested the development of a PE Framework. The PE Framework has been designed to establish key concepts, principles, and opportunities for PE in identifying health research priorities and in designing and conducting research projects.

In January 2014 and the spring of 2014, CIHR hosted a workshop and consultations to develop the SPOR PE Framework. This Framework was published and disseminated to stakeholders in June 2014.

The SPOR PE Framework^{Footnote 17} elaborates on what patients contribute to the research process and why it is needed:

“Patients bring the perspective as "experts" from their unique experience and knowledge gained through living with a condition or illness, as well as their experiences with treatments and the health care system. Involvement of patients in research increases its quality and, as health care providers utilize research evidence in their practice, increases the quality of care.

Engaging patients in health care research makes (investments in) research more accountable and transparent, provides new insights that could lead to innovative discoveries, and ensures that research is relevant to patients concerns. The international experience with engaging citizens and patients in research has shown that involving them early in the design of studies, ideally as early as at the planning stage, leads to better results."^{Footnote 18}

As a partner in SPOR and seeking to align itself with the SPOR PE Framework, CIHR developed a citizen and patient engagement (CPE) implementation strategy with a number of cross-cutting components, some with direct implications for SPOR. These particular components are also at various stages of implementation:

Examples of further work by CIHR include:

• Patient Engagement Collaboration Grants: 11 projects were funded (as of March 31, 2016) with objectives to: identify and implement inclusive engagement mechanisms, processes and approaches that value patient perspectives, experiences and skills throughout the research process; and facilitate opportunities for researchers and knowledge users, including patients, to work together to identify problems and gaps, set priorities for research, and produce and implement solutions. At the time the evaluation was designed, these grants had been funded for less than six months and so were excluded from the scope of this evaluation.
• The Foundational Curriculum for Patient-Oriented Research: Development of a resource to build capacity for all who are engaged in patient-oriented research (prepare citizens and patients to be active partners). The curriculum modules use a co-learning approach to foster collaborations between researchers and patients. CIHR intends to work collaboratively to situate this curriculum within a broader system of learning, and co-development of content is to take place. Ultimately, the curriculum will be rolled out nationally and eventually be made open access.

1.3 Target Populations

All Canadians are expected to benefit from SPOR, as it is hoped to lead to:

• improved health for Canadians by helping to ensure that the best research evidence moves into practice, enhancing the health care experience for patients and improving health outcomes for Canadians;
• economic benefits by optimizing spending on health care systems, reinvesting resources where the evidence shows greatest impact, and attracting private investments in evaluative research;
• innovation in patient-centred care in areas such as e-health, implementation science, and clinical practice;
• more clinical research by improving the environment for clinical trials in Canada; and
• collaboration among provinces and territories by providing jurisdictions with opportunities to learn from each other, translating best practices in patient-oriented care across Canada.

The patient should benefit from SPOR through receiving the right care in the right place at the right time. Patient-oriented researchers should benefit from training, research support services and an improved environment for clinical research. Health care professionals and policy makers should benefit from the timely and efficient translation of research innovations from the research setting to patient care settings, as well as the evaluation and synthesis of existing knowledge and its proper transfer to the clinical setting. Finally, the provincial governments and health care administrators should benefit from a more cost-effective, efficient, and affordable health care system.

1.4 Stakeholders

SPOR maintains a broad range of stakeholders, involved and interested in the development and implementation of its objectives including:

1.5 Governance

The governance structure of SPOR consists of a National Steering Committee and a CIHR SPOR Working Group. The governance structure is supported by the Priority-Driven Research Branch within the Research, Knowledge Translation and Ethics (RKTE) Portfolio of CIHR. In addition, each SUPPORT Unit and SPOR Network is required to have a governance structure, which includes appropriate mechanisms for PE.

SPOR National Steering Committee

The National Steering Committee (NSC) was established in 2011 and oversees SPOR development and implementation.

The committee is co-chaired by the Deputy Minister of the Ontario Ministry of Health and Long Term Care and the President of CIHR. Members include patients, federal/provincial/territorial governments, provincial health research funders, academic institutions, health care organizations, researchers, health charities and industry. The National Steering Committee’s composition also respects a nationwide geographical representation. CIHR provides secretariat services for the Committee.

SPOR Working Group

The SPOR Working Group was established in 2012 and provides scientific leadership within CIHR for the development, implementation and co-ordination of CIHR’s activities and initiatives related to SPOR. The Working Group also provides on-going monitoring and identifies refinement of activities and initiatives, as needed. Ad-hoc external advisory committees are engaged for advice on various specific issues as necessary.

The Working Group is chaired by the Chief Scientific Officer of CIHR, and consists of five Institute Scientific Directors, each being a champion of one of the five SPOR elements. It also includes the Associate VP Research, Knowledge Translation and Ethics, the Director General of Priority-Driven Research Branch, and Manager of Major Initiatives.

1.6 CIHR Institute Involvement in SPOR

CIHR integrates research through a unique interdisciplinary structure made up of 13 "virtual" institutes. Each Institute is dedicated to a specific area of focus, linking and supporting researchers pursuing common goals. All CIHR Institutes are engaged in SPOR, though to varying degrees. Institute engagement with SPOR can be characterized in the following ways, which are not mutually exclusive (see Appendix B for details):

• Providing scientific leadership (e.g., serving as a Champion for a SPOR element; co-leadership for designing SPOR Networks; presentations to SPOR governance structures; shaping scientific content of SPOR funding opportunities; contributing to the development of performance measurement frameworks).
• Serving on SPOR governance structures (e.g., SPOR Working Group) or consulting Institute governance structures (e.g., Institute Advisory Boards on SPOR elements or SPOR overall).
• Making financial contributions from Institute Strategic Initiative budgets (e.g., to Network development grant funding opportunities) or Institute Support Grant budgets (e.g., funding strengthening workshops).
• Ensuring strategic linkages to other Institute priorities (e.g., linking SPOR to other Institute strategic priorities or ensuring alignment where possible) or stakeholders (e.g., partnership development).

1.7 Resources

An overview of CIHR grants and awards expenditures on SPOR over the evaluation timeframe is provided in the table below:

*Summary of figures provided by CIHR current as of March 21, 2016.

1.8 Evaluation Purpose and Scope

SPOR is a complex and complicated program and reflects a large system of stakeholders, delivery partners, activities, and linkages to other programming efforts both within CIHR and in the provinces and territories. Although many of the SPOR partners were engaged in the evaluation through interviews, the evaluation was of CIHR’s contribution to SPOR core elements only.

The SPOR evaluation covers the five-year period 2010-11 to 2015-16 and was scoped to cover issues of relevance, design and delivery, and performance. The purpose of the evaluation is twofold:

• To meet the CIHR’s accountability requirements as a federal government agency (Treasury Board’s Policy on Evaluation and Financial Administration Act).
• To provide an independent and objective assessment of the implementation and performance of SPOR to date to inform program decision-making.

In order to prioritize the focus on the implementation of the core elements and assess progress toward immediate outcomes as outlined in the SPOR performance measurement strategy, the following calibration considerations were made:

• Scope and resources for the evaluation were designed to complete the evaluation in eight months.
• The evaluation was designed to meet information needs of the program through the assessment of the implementation of SPOR core elements, especially the SUPPORT Units, and to inform senior management on progress to date, course corrections needed, and evidence of early/quick wins.
• The evaluation aligned with TBS requirements to demonstrate impact from funds secured through TB submissions which are mapped to the Performance Measurement Strategy submitted to TBS.

Given these scoping considerations, the evaluation focused on the deployment of SPOR core elements and short-term progress towards objectives, and used the program profile contained in the SPOR Performance Measurement Strategy as the baseline implementation plan.

The focus on Foundational Investments was limited to outlining the investments made through the evolution of SPOR from 2010-11 to 2015-16 as well as considering these investments in the assessment of efficiency under the performance issue. In terms of assessing the outcomes of the Foundation Investments, these were scoped out given they were not deliberately designed at the outset to achieve SPOR outcomes as stated in the logic model, resources to assess the investments were not available, and some of the investments were in the scope of previous, ongoing (e.g., scholarships) and planned (e.g., operating support) evaluative activities.

The scope of the evaluation is summarized in the following table:

Further elaboration of some of the components excluded from the evaluation scope can be found in section 2.3 on study limitations.

1.9 Evaluation Questions

A set of evaluation questions were prepared and vetted by CIHR. The following seven questions were addressed in the evaluation:

SPOR Evaluation Issues and Questions

Relevance

1. To what extent does the research funded under SPOR address the need for evidence-informed health care?
2. To what extent is SPOR aligned with federal roles and responsibilities?
3. To what extent is SPOR aligned with federal government and CIHR priorities?

Design and Delivery

4. To what extent has SPOR been implemented as planned?

Performance

5. To what extent has SPOR made progress toward the achievement of expected immediate outcomes?
6. To what extent has SPOR made progress toward the achievement of expected intermediate outcomes?
7. To what extent is SPOR being delivered in a cost-efficient manner?

1.10 Report Structure

The findings and analysis for each of the seven evaluation questions are provided in the following sections of this report. Each section defines the specific evaluation question(s), summarizes the key findings against each of the issue areas, provides details on the analysis and evidence and provides supporting conclusions. There is a final chapter on recommendations for improvement.

2. Methodology

To provide cross-cutting representation and feedback from the various stakeholders involved in SPOR, the evaluation was designed to use multiple lines of evidence. With a very complex program design and implementation that is early in its lifecycle, along with variations by jurisdiction and degrees of progress, the evaluation design relied heavily on interview and case study techniques. Overall, the methodology included the following data sources:

• Document and data review.
• Key informant (KI) interviews.
• Case studies that incorporate:
  • Document review, and
  • Interviews.
• Governance and administrative structures network analysis.
• A comparative review of three somewhat similar international POR initiatives.

The data sources have been mapped to each of the evaluation issues in the following table:

Each of the data sources are described in the sections following and the supporting instruments are provided in the appendices to this report. The number of people interviewed for the key informant interviews and the number of people interviewed for the case studies are presented collectively in the following Exhibits 3 (by method) and 4 (geographically).

2.1 Data Sources

2.1.1 Document Review

Approximately 40 documents were reviewed, mainly in support of the evaluation questions related to relevance. These included CIHR strategic documents (such as its annual reports and the SPOR overarching strategy document), federal and provincial level strategic documents and reports, industry evaluations, SPOR element specific reporting and other grey literature. The document review was also used to form the contextual description of SPOR utilizing early planning documents to report on the background and vision of the strategy.

2.1.2 Performance Data Review

The evaluation compiled a summary of activity data presented in the 2013-2014 and 2014-2015 SUPPORT Unit Annual Performance Reports as well as the 2014-2015 Annual Performance Report pertaining to the ACCESS Open Minds Network. This was the entire scope of performance reporting available during the evaluation timeframe. The performance data selected for presentation in this report aligns to the performance indicators as identified in SPOR’s performance measurement strategy. The data is reported at an aggregate level representative of SUPPORT Unit activities collectively. The data presented in this report has not been validated by KPMG and is presented as submitted to CIHR by the provincial SUPPORT Units and the ACCESS Open Minds network.

2.1.3 Key Informant Interviews

A key informant interview sample of 19 individuals was developed that included representatives from CIHR management as well as a sample of SPOR partners, stakeholders and users as part of the evaluation work. The sample was purposefully selected to include individuals with broad knowledge of SPOR. Considerations for interviewee selection included knowledge across all SPOR elements as well as (to the extent possible) other factors such as the academic/institutional/healthcare/patient involvement dimensions, and provincial/territorial representation. The sample of 19 individuals was identified by SPOR program personnel. Fifteen interviewees participated out of the sample (15/19=80% response rate).

2.1.4 Case Studies

Five case studies were developed that focused on the implementation of the five core elements of SPOR in a cross-cutting way against all of the SPOR programming. Two sets of questions were addressed with case study interviewees. The first set of questions related to the specific core element of SPOR being studied and the second set of questions related to the overall strategy of SPOR. The five cases conducted, ordered alphabetically, include:

• Capacity Development: A sample of seven interviewees were identified by SPOR program personnel that included members of SUPPORT Units, as well as members of the External Advisory Committee. This sample was purposefully selected to include individuals with specific knowledge of CD. Two interviewees participated out of the sample (2/7=29% response rate).
• Clinical Trials: An initial interview sample made up of five individuals was developed to include representatives of the Clinical Trials National Advisory Group including public, private, and academic sectors. This sample was later expanded to a total of seven individuals, including representation of the CCTCC Executive Committee as well as additional clinical trials researchers. The sample was purposefully selected to include individuals with specific knowledge on clinical trials. The sample of five individuals was identified by SPOR program personnel, with the two additional personnel being recommended through the interview process. Four interviewees participated out of the sample (4/7=57% response rate).
• Patient Engagement: The scope of the PE case study centred on CIHR’s efforts at a national level in development of the PE Framework as well as PE activities undertaken or connected to the other SPOR elements. A sample of 13 interviewees were identified by SPOR program personnel, including patient representatives, on the basis of their involvement in PE (e.g., through CIHR’s curriculum development and collaboration grants), patient representatives from SUPPORT Units and Networks, and CIHR personnel. This sample was purposefully selected to include individuals with specific knowledge of PE. Ten interviewees participated out of the sample (10/13=77%).
• SPOR Networks: Two Networks were included in the scope of the case study: ACCESS Open Minds and PIHCI. It was considered too early to conduct interviews with direct network participants for the SPOR Networks in Chronic Diseases, since this network was still in the competition process in the fall of 2015. Interview and survey data collection thus focused exclusively on ACCESS Open Minds and PIHCI.
  • A sample of seven interviewees for the Networks case study was selected with six interviewees participating (6/7=86% response rate). Although interviews were initially planned with one PIHCI representative from each of three areas (research, policy, clinical), this data collection activity was subsequently expanded to target all PIHCI Interim Leadership Council members through an email survey rather than the individual interviews, with notification given to the Interim Leadership Council during a meeting on October 19, 2015. Note that a more restricted set of questions was asked in the PIHCI survey than during the full interviews. The survey response rate was low at 19% (or 6 out of 32).
• SUPPORT Units: A sample of 54 interviewees for the SUPPORT Unit case study was selected. The sample reflected the Executive Director (ED) and component leads within each SUPPORT Unit, clinicians and/or scientists (usually those involved in one or more of the SUPPORT Unit’s demonstration projects), and representatives of provincial/territorial funding agencies. Some interviews were conducted with individual respondents, although, where possible, interviews were conducted in group fashion with similar respondents from similar groups (i.e., a given SUPPORT Unit’s Executive Director and component leads were, when possible, all interviewed together). A total of 68 interviewees participated, surpassing the selected sample and providing a 125% completion rate for the target interviews. The increase in number of participants is partly due to a snowball effect in the group interviews, where additional people may have been invited to attend, or could not attend but were suggested as important respondents, as well as additional partner interviews being conducted.

2.1.5 Governance and Administrative Network Analysis

The governance and administrative network analysis was employed as a minor supporting method in the evaluation. Data were collected for each SPOR component having information available through a public internet site. Data were collected for each SUPPORT Unit and the SPOR National Steering Committee in this manner. Contact lists provided from the SPOR program in support of the evaluation were also used to compile data for the PIHCI network, SPOR SUPPORT Unit Council and CIHR’s internal SPOR Working Group.

Web sites were mined for all information related to each component’s governance structure. This included governance, management, and advisory committees. A total of 296 data points were used in the network analysis.

The data elements collected to form the initial edge list^{Footnote 19} included the person’s name, the organization with which they were associated, the SPOR element with which they were associated, their province/territory, role, and the part of the governance or corporate/administrative structure to which they belonged.

This data were then cleaned and standardized before importing to two separate social network analysis software packages. Two packages were used to obtain the graphical functionality and the mathematical processing functionality required for analysis and presentation. Data were imported at the organizational level, not at the level of each individual. For example:

If the following ties were identified:

• John Smith, University of Alberta, connected to the SPOR National Steering Committee (NSC)
• Harvey Beam, University of Alberta, connected to the Alberta SUPPORT Unit

The following data were imported:

• University of Alberta - SPOR NSC
• University of Alberta - Alberta SUPPORT Unit

A tie represents a connection from one organization to another. The ties in this example are between the University of Alberta and the SPOR NSC and the University of Alberta and the Alberta SUPPORT Unit.

NodeXL^{Footnote 20} was used to create the network graphs and assemble groups based on the network attributes of “province/territory” and “role”. UCiNET^{Footnote 21} was used for the mathematical calculations and analysis of degree centrality^{Footnote 22} and betweenness^{Footnote 23}.

Typical terminology found in social network analysis has purposely not been used (to the extent possible) in this report for ease of portraying the information to readers unfamiliar with the methodology or network theory.

2.1.6 International Comparative Review

The international comparative review was undertaken to explore how other countries are approaching POR initiatives with similar objectives to SPOR. The objective of the review was to gather information on the relative merits of other approaches and models and to obtain perspectives on the pros and cons of the SPOR model versus others. A direct comparison with other initiatives was not undertaken due to the uniqueness of SPOR (large and nationally based).

Through scoping interviews, three initiatives from two organizations were mentioned most frequently as being of most interest to SPOR. These organizations were selected for their experience in POR. The two main international comparison organizations which were reviewed are: The Patient-Centered Outcomes Research Institute (PCORI) in the US and the UK’s National Institute for Health Research with two initiatives: INVOLVE and the Collaboration for Leadership in Applied Health Research and Care (CLAHRC).

The comparative review was conducted in two stages. First the websites for each of the three initiatives were mined for descriptive information on the models/approaches employed. This was followed by interviews with one representative from each of the initiatives. The contacts for PCORI and INVOLVE were identified through the SPOR program. The SPOR program could not identify contacts for the CLAHRC so respondents were identified through the interviews conducted with key informants and case studies. The main purpose of the interviews was to obtain opinions on SPOR’s key strengths and weaknesses as compared to a respondent’s patient-oriented initiative and to identify lessons learned through their experience in implementation and delivery.

2.1.7 Cost efficiency analysis

By assessing not only a program's relevance and performance, but also the resources the program uses, value for money can be determined.

For the SPOR evaluation, both allocative and operational efficiency were examined in order to determine: (a) whether resources requested from TBS were spent on the activities for which they were requested; and (b) whether this spending occurred at the level and in the timeframe for which it was requested. Resources are considered in relation to the outputs or outcomes achieved, respectively.

The SPOR evaluation was designed to focus only on the SPOR core elements – SUPPORT Units, Networks, CD, PE and Clinical Trials. “Foundational Investments” as well as other investments that are part of SPOR (e.g., Studentships; Fellowships, Clinical Trials funding) were excluded from the evaluation of effectiveness and relevance due to feasibility (available budget, timeframe of evaluation vis a vis funding flow) and direct and deliberate relevance of the timing of these investments relative to SPOR. In many cases, the Foundational investments were well-aligned^{Footnote 24} with SPOR, however may have begun prior to SPOR being announced, maintained during the implementation of SPOR, or had sunsetted post-SPOR implementation (e.g., operating grants, catalyst grants, knowledge synthesis grants). However these investments are included in the cost-efficiency analysis to help ensure a complete accounting of resources used to deliver on SPOR.

Allocative and Operational Efficiency

Data for the cost-efficiency analyses were provided from CIHR’s Finance and Administration Branch, and Performance Measurement, Reporting, and Data Unit and confirmed with the SPOR Program Manager and staff. These data were examined in relation to the Treasury Board submissions for SPOR over the same period of time. Data for the cost-efficiency analysis included:

• All grants and awards expenditures by CIHR from 2010-11 to 2015-16 (the evaluation time frame) related to the SPOR Elements (SUPPORT Units, Networks, CD, PE, and Clinical Trials) and the Foundational Investments.
• All direct and indirect administration costs for Operations and Maintenance (O&M) as follows:
  • Direct salary costs: As SPOR has been ramping up since the first Treasury Board submission in 2010-11, the FTEs dedicated to SPOR, as obtained and reported to TBS since then, was used as the base to calculate the direct salary. CIHR continuously increased the number of FTEs dedicated to SPOR as follows: CIHR obtained three FTEs in the first submission (2010-11), four in 2011-12, four in 2012-13 and 4 in 2013-14 (at which time CIHR noted to TBS that it would fund 5.75 additional FTEs from non-salary funding received). As a result, direct salary costs increased from 2010-11 to reach the steady state of 20.75 FTEs since 2013-14.
  • Direct operations and maintenance costs, including: travel and other related expenses for SPOR meetings, such as peer review meetings and the SPOR Summit (e.g., meals, accommodations, space), equipment, training and conference fees for SPOR staff.
  • Internal services (indirect costs): As a result of the CIHR exercise completed in November 2015 to calculate CIHR’s Internal Services allocation in compliance with TBS’s Guide on Internal Services, it was determined that CIHR spends $0.03 on internal services for each G&A dollar. This methodology was applied to the total G&A disbursed (new and ongoing, including the Foundational Investments) in the year. For CIHR, such internal services include: Corporate and Governmental Affairs, Communication and Public Outreach, Senior Executive offices, and staff from the Resource Planning and Management Portfolio (e.g., Finance, HR, IT, and Evaluation).
  • For direct salary and internal services, employee benefit plan (EBP) costs calculated at the Treasury Board rate of 20 per cent of total salary costs.
  • For direct salary and internal services, accommodation costs calculated at the Treasury Board rate of 13 per cent of total salary costs.

A fidelity assessment (O’Connor, Small & Cooney, 2007) was conducted to determine the allocative efficiency and assess the degree to which SPOR was implemented as planned and to determine the extent to which these implementation variances impacted the outputs, outcomes, and costs. In the case of the SPOR cost-efficiency analysis, the fidelity assessment is defined generally as a qualitative approach to assessing operational efficiency. This approach focuses on assessing the degree to which a program was implemented according to its initial plans, identifying variances in implementation, and examining these in order to determine the rationale for the variances and the effect they had on costs or on the achievement of outputs or outcomes. These approaches also allow for an assessment of the changes in delivery made by the program in its implementation approach and the impact that these had on costs or on the production of outputs or achievement of outcomes.

A proportion for operational efficiency was calculated annually for SPOR by dividing the total administration costs (direct and indirect) by the total SPOR expenditures (administration costs and grants and awards expenditures) (Exhibit 23).

As SPOR core elements have stated requirements for contributions for partnership funding (see Section 1.1), the extent of external partner contributions to SPOR were examined by tabulating applicant level partner contributions from 2010-11 to 2020-21, as provided in the applications. Partner contributions included cash and cash equivalent in-kind contributions for the full timeframe of the respective investments. A ratio of CIHR to partner contributions was calculated by SPOR element.

2.2 Data Analysis

Technical reports were prepared for each data source. For the data sources that included interviews, interview notes were prepared for individual interviews conducted and were then summarized and consolidated into an Excel database. Summarized notes were organized by question as presented in the interview guide and then mapped to the corresponding evaluation question. Questions were laid out in rows and the interviewee names formed the columns. A multi-stepped approach was used for the analysis. The first step was to complete a review of all interview summaries. This review provided the analyst with a preliminary understanding of the issues arising from the interviews. The next step in the process was a detailed review of the interview summaries looking at each interview question, or logical groupings of questions, noting keywords and statements and identifying common themes through the content analysis. Finally, findings were aligned by summarizing findings for each evaluation question.

In the development of the findings associated with the interviews, the analyst took into consideration the number of people or percentage of people who provided a specific response, comment or discussion along the same theme to understand shared or similar opinions across the group. Additionally, where there were rating questions associated with open ended questions, the analyst compared the qualitative findings with the quantitative findings to qualify coherence in the results and assess internal consistency.

For the purpose of this report, the following terminology is used to specify if there were single ideas or ideas shared by more than one individual:

• None (0 or no)
• A few (~20%)
• Some (~40%)
• Many (~60%)
• Most (~80%)
• All (100%)

Caution should be applied in over interpreting the magnitude of specific perspectives; a lack of response by a participant may not mean they do not have an opinion or similarly, an opinion by a few participants may not be shared by many more participants. Additionally, not all interviewees answered all questions. The number of respondents differs for each question and the analysis was performed based on the responses received for each question separately. As a result, the magnitude of responses presented in this report is based upon the sample of interviewees providing feedback for each relevant question. The average response ratings have been calculated as a standard weighted average based on the number of respondents.

A similar process was used to prepare the overall evaluation report, where each technical report was reviewed to identify logical groupings of findings across data sources noting keywords and statements and identifying common themes through the content analysis. Finally, findings were aligned by summarizing findings for each evaluation question and then rolled up to reflect the three evaluation issue areas.

2.3 Strengths and Limitations of the Evaluation

2.3.1 Strengths

Early stage of implementation. The evaluation was conducted at a time early enough to collect real-time feedback illuminating potential areas of opportunity to influence ongoing delivery. The evaluation was designed to focus on learning, while still measuring progress towards outcomes (measurable change).

Coverage of stakeholders. All SPOR stakeholder groups were reached through implementation of the study methodology; this included researchers, policy makers, health system decision makers, clinicians, patients, and industry. Additionally, a large cross section of respondents was reached based on geographical location, with interviews being conducted with stakeholders situated in every province and territory across Canada.

Depth of enquiry to evaluation questions. The evaluation methodology relied heavily on interview and case study techniques, which are much finer instruments for delineating the “how, what, why, and extent” of effects taking place. Data collection instruments were designed specifically to support inquiry into SPOR as an overarching instrument in addition to specific probing at the SPOR element level. Extensive personal interviewing to discuss progress, achievements, and impacts in depth, allowed for follow-up questioning. The open-ended questions were supported by quantitative rating questions that assisted the study team in understanding the general tenor of responses and aided in the roll up of data for analysis.

Rigorous challenge and review process of evaluation findings and conclusions. All evaluation reporting, technical line of evidence reports and draft findings reports were presented to and reviewed with CIHR’s Evaluation Unit, the SPOR program, and in some instances CIHR’s senior management team and the SPOR Working Group. These exercises provided the opportunity to challenge the evaluation findings, provide additional evidence and confirm contextual data being presented. 

2.3.2 Limitations

Potential bias in opinion data. In developing the key informant and case study interview samples, the number of stakeholders who had broad enough exposure to SPOR and its core elements was limited. During the design phase of the evaluation, it was determined that the individuals holding the broadest knowledge of SPOR and all its core elements included only those individuals involved in SPOR’s governance structures and those involved in operationalizing SPOR’s core elements. The opinions and perspectives of these sample groups may be somewhat biased due to their closeness to the development and implementation of SPOR. However, the possibility of this limitation is believed to be minor as the study team acknowledges the open, direct, and candid nature of the interviewee’s responses.

Absence of comparison groups. The evaluation design did not include a comparison group – either of those not engaged in SPOR or of other initiatives. The evaluation focused at the strategy level and in most instances CIHR is one step removed from direct users. A survey (or other far reaching activity) was not possible due to limited ability to identify all individual connections to all SPOR elements, in addition to the budget and timing allocated to the evaluation to enable the identification of all such connections (particularly the research community). However, through the key informant interviews and case studies all stakeholder groups were represented, including researchers and clinicians in the SUPPORT Unit case studies and patients in the PE case. Note also that many individuals involved in designing and implementing SPOR are also either end-users themselves, or representatives of end-user groups.

The evaluation does not include a comparison with another initiative due to the uniqueness of SPOR (large and nationally based). However interviews with international stakeholders were conducted to explore how other countries are approaching initiatives with similar objectives were undertaken.

Validity of self-reported data. The evaluation reports the observations and opinions of the individuals contacted. Within study resources, and because SPOR is so new, the accuracy of opinions cannot be confirmed at this time (e.g., it is too early to really determine if all needs are being met). However, most of the individuals interviewed are very experienced in their respective domains.

Early stage. The analysis is mainly limited by the very early stage of the operations of SPOR. The evaluation does not cover the overall added-value of SPOR or the coherence across the SPOR elements (SUPPORT Units and Networks in particular) due to early and varying stages of implementation (funding) and operationalization. Thus, the interview findings should be taken as early considerations that may affect the future design and delivery of the SPOR elements and how they interact with other SPOR elements.

Internal validity. The analysis of performance reporting is limited by challenges in interpreting the information in performance reports and confirming the internal validity^{Footnote 25} of the data, particularly in separating out existing networking, collaboration, and cooperation around patient oriented research topics and/or evidence-based healthcare that pre-existed the creation and implementation of SPOR. The evaluation scope, timeframe and allocated resources for the study did not allow for mitigation of this effect in its design.

Ratings. The results of the rating questions used within the interview guides are likely not fully replicable as presented in the report, mainly because the early stage of development often meant that respondents were uncertain how to answer. In some cases, there are relatively few respondents who provided ratings. Thus, small changes in the number of respondents in a given rating category would, in many cases, significantly change the apparent distribution of responses and the average ratings, easily leading to misinterpretation and/or over-interpretation of findings based on these quantitative data.

Quotes. Some anonymized quotes are provided to illustrate the perspective of respondents. These have been selected to be representative of commonly-held opinions, or in a few cases, less common opinions that may represent unique and important insights. The report has indicated which line of enquiry the quote is aligned with and in some cases the position of the individuals quoted. However, because of the group nature of some interviews with SUPPORT Unit EDs and component leads, it was not always possible to determine this – in these cases the quotes are attributed to “ED or component lead”.

Basic form of network analysis. The data compiled for the network analysis were limited to gathering publicly available information and data available through documentation provided by the SPOR program that reported on key SPOR component governance and administrative structures. The data collected do not include all participant information at the operational level (e.g., researchers accessing SUPPORT Unit services, patients involved in projects). Therefore, the data do not represent the SPOR network in its entirety.

The analysis applied to the SPOR governance structure network is basic in its form. This is a limitation of the scope of the evaluation, the data available, as well as the software used for analysis. As the data used for analysis are not complete (i.e., not all relationships were captured), more extensive mathematical measures (underpinned by graph theory) would not produce valid results and would skew the interpretation of findings and therefore were not completed. As a result, the data and analysis presented in this report should not be interpreted to alone explain any of SPOR’s outcomes.

Cost-efficiency analysis limitations. Given the nature of SPOR, which is a large-scale strategy composed of numerous elements rolled out at different points in time and supported through unique design features described above, there is not a suitable comparator within CIHR against which to benchmark the operational efficiency proportion so no conclusion as to the adequacy of the cost-efficiency ratio has been provided.

The grants and awards expenditures and the operating and maintenance costs were collected at different points in time, though within the same Fiscal Year. Operating and maintenance costs for the core elements were challenging to capture due to re-organization of human resources within the portfolio where SPOR is managed.

For the applicant level partnership contributions, as noted above, the data were pulled from applications and, as such, reflect commitments confirmed at that time as opposed to actual commitments made through the full course of the investment, however, the program monitors and validates these commitments on an annual basis. As a result, there is the potential that applicant level partner contributions could be an underestimate or an overestimate of those actually made. Further, for those programs where partnership matching was not a requirement, the partnership data were not consistently entered into CIHR’s funding database across programs and/or over time. As a result, partner contributions to these programs could be an underestimate due to data entry inconsistencies.

High heterogeneity. This limitation is at the level of the individual SUPPORT Unit element. Each SUPPORT Unit represents a different response to the healthcare needs and pressures within its own provincial/territorial environment, including: different levels of existing POR initiatives; different levels of baseline capacity in the research, clinical, and policy communities; and varying levels of existing interactions among researchers, service providers, provincial/territorial healthcare authorities, policy makers, patients, and other stakeholders. Where findings appear to be consistent across most or all SUPPORT Units, they are discussed jointly; where they appear to differ significantly across different SUPPORT Units, some discussion of differences is provided – most of the differences relate to the size and existing research and clinical capacity of each province/territory. Note that while there are a number of common aspects, for many aspects of design and implementation, these SUPPORT Units have faced quite different challenges (and opportunities) to date as a result of the variability across provincial/territorial contexts. As a result, SUPPORT Units yet to be implemented will need to interpret the relevance of best practices and lessons learned in relation to their own context. This limitation also applies to the ACCESS Open Minds and PIHCI networks, being very different initiatives, where lessons learned within these two instances may not translate well to future Networks in development.

3. Evaluation Findings

3.1 Relevance

3.1.1 Key Findings:

• SPOR aligns with Canadian public policy, Canadian public opinion and international perspectives on the need for patient-oriented research.
• In support of POR, SPOR is facilitating an increase in collaborations among relevant stakeholders, enhancing access to common resources, facilitating patient involvement in research, and providing the platform for the facilitation of translation of knowledge into clinical practice and policy.
• SPOR does encompass the right elements and activities to address areas in need of support for POR and evidence-informed health care.
  • Each strategy element is meeting needs (operationally) to various extents, and all appear to be focused in the right areas.
    • Capacity Development was designed to address three key areas of need: enhancing patient participation in research, increasing the engagement of researchers conducting PE, and, supporting the development of careers regarding patient-oriented research.
    • Clinical Trials was designed to address the need for enhancing the ability of Canada to further improve the clinical trials environment in order to remain competitive within the global market. This includes reducing the length of time to conduct clinical trials, enhancing resourcing and infrastructure, and improving the national model.
    • Patient Engagement is an integral design element to assist SPOR, across the nation, in engaging patients in research, at all levels and in all phases, to improve outcomes of health care systems across the country. The PE element of SPOR is seen as setting the tone for what PE within research would mean, both within the scope of SPOR specifically, and more broadly across other efforts in Canada.
    • SPOR Networks are addressing a significant need for large, multi-faceted, POR initiatives that intend to be successfully scaled up and translated across sites and regions; however, the networks in place are not operating at full scale so needs are still being defined.
    • The SUPPORT Unit model is providing enough flexibility to develop the provincial/territorial response to needs within POR. SUPPORT Units are coordinating support for POR through extensive consultation with (mainly) provincial/territorial stakeholders as to priorities, existing capabilities, and areas of focus.
• SPOR aligns closely with the roles, responsibilities and the priorities of the federal government to support patient-oriented research as well as evidence-based approaches to improve healthcare systems and practices. SPOR objectives can be mapped to the mandate of CIHR through the CIHR Act, in addition to the Canada Health Act, and the objectives of other federal departments, such as Health Canada and the Public Health Agency of Canada.
• CIHR is viewed by all stakeholders to be playing an integral role in the implementation of SPOR and critical for the development of POR in Canada. CIHR’s credibility, national approach, and ability to bring commonality in design and delivery were identified as key.
• SPOR complements POR activities undertaken by the provinces and territories. SPOR strengthens and coordinates other disparate POR initiatives and helps to fill gaps in current programming, resources, methods and capabilities.

3.1.2 Analysis

3.1.2.1 Addressing the need for evidence-informed health care (Evaluation Question #1)

The nature of the ongoing need of evidence-informed health care has been long-established in Canada. As early as 1986, the federal government has highlighted evidence-based, patient-oriented, and pan-Canadian approaches to research.

In 1984, the federal government replaced two previous acts with the Canada Health Act, a law that set pan-Canadian standards for hospital, diagnostic and medical care services.^{Footnote 26} In 1986, the Canadian government played an important role in establishing the globally influential Ottawa Charter for Health Promotion. This declaration focused on the process of enabling people to increase control over, and to improve, their health.^{Footnote 27} Additionally, through a review of the 2004 Health Accord, the Standing Senate Committee on Social Affairs, Science and Technology was notified that the development of benchmarks was not sufficiently evidence-based or patient-centered. The Committee made various recommendations, inclusive of the development of strategies to address such evidence-based and patient-centered pan-Canadian benchmarks.^{Footnote 28} Further, patient-centered research has remained a topic of discussion for various federal agencies, inclusive of Health Canada, the Public Health Agency of Canada, as well as CIHR.

Trends in public opinion polls indicate a view for a need for Canadian health care reform, as well as demonstrate support for patient-oriented approaches to research. These results indicate that, at a minimum, there is a perception that reform should be undertaken, research is perceived to be an integral component of health care, and that patients would like to be engaged in the research. In a survey conducted in 2010, 61 percent of the patients sampled identified that the Canadian health system was in need of fundamental reforms or to be rebuilt completely, which was lower than in Australia (75%) and the United States (68%) but higher than France (58%), Sweden (53%) and the United Kingdom (37%).^{Footnote 29} In another report, 90 percent of Canadians have said that health and medical research makes an important contribution to health care and 70 percent of Canadians have said they are interested in participating in health and medical research, though only 24 percent reported being aware of opportunities to do so.^{Footnote 30}

There remains an ongoing need for Canada to stay current in terms of health care and health research approaches in order to address national needs and to be competitive within the global market. Also, an international trend towards evidence-informed health care and patient-oriented research provides Canada with the ongoing requirement to remain relevant in this field.

In order to gain insight regarding the perceptions surrounding the evolution of patient-oriented research as a model on an international scale, international interviewees contacted as part of this evaluation were asked for their views on the context for patient-oriented research evolving internationally. Generally, it was felt by all international interviewees that there is a strong trend internationally towards patient-oriented research models. A few countries are perceived as being “quite vibrant” in moving towards this trend, including the UK, Canada, the US, Denmark and Australia. All interviewees identified CIHR SPOR as playing an integral role in Canada’s position in support of implementing a patient-oriented approach to research.

The PE case study also identified how successes internationally in patient-oriented research have only served to strengthen the need for such a national strategy in Canada. Indeed, various studies have supported the theory that patients who are active as participants in the health care decision making process results in enhanced health care outcomes at lower costs than less active patients^{Footnote 31}. The PE case study additionally identified that engagement of patients within the research process contributes towards an enhanced ability to translate research into practice, as it helps to ensure uptake by the end user – the patient. Experiences internationally would serve to support this theory as well, with considerable evidence suggesting that PE can improve patient experience and satisfaction and also can be effective clinically and economically.^{Footnote 32}

Documentation reviewed from international sources supports the findings in the PE case study. A few examples of international support towards patient-oriented research and evidence-informed decision making include, but are not limited to, the following:

• The Seventh Framework Programme (FP7) of the European Union. This research-funding instrument stresses the importance of patient and public involvement.^{Footnote 33}
• The Patient-Centered Outcomes Research Institute in Washington DC, which has allocated US$68 million to a research network centered on the principle that “the interests of patients will be central to decision making.”^{Footnote 34}
• The World Health Organization, which released a paper in 2008, noted that “Strategies to support patient education and engagement should… be a fundamental plank of health policy.”^{Footnote 35}

All key informant interviewees acknowledged the need for change in the manner in which health research is conducted in Canada and the valuable opportunities available for continued development. The common areas of need being addressed by SPOR were expressed by most interviewees as:

• Increasing the ties between research and delivery to facilitate translation of knowledge into clinical practice and policy in a timely and practical manner.
• Developing a common framework for and understanding of meaningful patient involvement in research and research that is attuned to user needs.
• Increasing and enhancing access to common resources, infrastructure (data), methods and expertise within and across Canada.
• Increasing and augmenting collaboration (and multidisciplinary collaboration) among researchers, patients and decision makers to extend funding and speed research.

The extent to which SPOR elements are addressing key needs varies by area. All key informant and case study interviewees responded to this question within the context of the current stage of each SPOR element, identifying that it is “still early days for SPOR.” Exhibit 5 depicts the results of all interviewee responses to the rating question.

Exhibit 5 – Extent to which SPOR’s core elements meet key needs

*Rounding has been forced to add to 100%.

*Average rating has been calculated as a standard weighted average based on the number of respondents.

The SUPPORT Units and PE elements have the highest average rating, with the majority of key informant and case study interviewees rating these elements as meeting needs between a moderate to great extent (average rating of 3.3).

Key informant interviewees identified SUPPORT Units as being in various stages of implementation across the country and most interviewees were unsure of the extent to which needs were being met. However, all SUPPORT Unit respondents believed their SUPPORT Unit was designed and operating to meet needs very well due to their explicit mapping of SUPPORT Unit initiatives against provincial/territorial priorities and needs. Many SUPPORT Unit case study participants noted that, although the concepts of PE and POR aren’t new, the SUPPORT Unit provides a formal place to go where stakeholders can talk about it and ask questions. Many respondents noted that this ability is very new and is in large part due to SPOR.

The PE case study identified the PE element as integral to assisting SPOR in engaging patients at all levels and in all phases of research to improve outcomes of health care systems across the country. PE case study interviewees recognized the PE element of SPOR as setting the tone for defining PE within research, both within SPOR and across Canada, as researchers become increasingly familiar with PE through SPOR grant applications.

The SPOR Networks and the Clinical Trials environment were rated as meeting needs to just around a moderate extent, with aggregate average ratings of 2.9 and 2.7 respectively. Key informants were mostly unsure of the extent to which the Networks were meeting needs, while the case study identified ACCESS Open Minds topics (i.e., mental health) to be an area in significant need for large, multi-faceted, initiatives to demonstrate how youth mental health care could be transformed. The Network case study respondents noted that necessary key activities are being undertaken at present, however implementation has been slower in some areas than expected, and there is considerable site-by-site variation that must be overcome. Respondents to the PIHCI survey noted that PIHCI is intended to address both national and in-province/territory needs, e.g., cross-jurisdictional needs identification and learning; integration with clinicians and policy makers; regional linkages across academia, clinicians, healthcare, and policy makers; and developing common languages among sectors. PIHCI addresses the diversity across the country and scale of primary care, with the opportunity for secure and shared data. Meeting needs in the Clinical Trials environment was identified through the case study to be mainly centred on the progress being made in the CCTCC, although it was noted that needs would not be fully met through this entity alone.

CD is the SPOR element identified as meeting needs to some extent (aggregate average rating of 2.4); lower than any of the other elements, and the area where the greatest number of interviewees identified this element to be not meeting needs at all. The CD case study identified three core areas of need including: enhancing patient participation in research, increasing the engagement of researchers conducting PE, and, supporting the development of careers regarding patient-oriented research. These core areas were seen to be lagging behind in implementation and it was identified through the key informant interviews that some key stakeholders were missing from the table to be able to move it forward. Key informant interviewees did acknowledge the release of the SPOR CD Framework but did not identify outcomes in this area to be sufficient beyond this. It was perceived that there is no plan in place to move CD forward and that there has not been sufficient work with partners (e.g.., the universities). Key informant interviewees identified a further need to undertake targeted activities for capacity development to increase understanding and drive action in this area. It appears it is not well known what the SUPPORT Units are doing in this area and there is an expectation that SPOR needs to be doing more.

3.1.2.2 Roles and responsibilities (Evaluation Question #2)

Though the federal government enacted the Canada Health Act in 1984 to set pan-Canadian standards for hospital, diagnostic and medical care services, there continue to be emerging health challenges across the provinces and territories. In Canada’s Strategy for Patient-Oriented Research, published in 2011, noted that such challenges speak to the need to develop a fulsome, collective strategy at the federal level in order to move forward.

SPOR is clearly aligned with the mandated role and responsibilities of CIHR in its founding instrument, the Canadian Institutes of Health Research Act, assented 16 years ago in April 2000. The Act states CIHR’s main objective is to “excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system.”^{Footnote 36}

The federal Minister of Health is responsible to Parliament for CIHR and the government has a stated objective of making Canada one of the five leading health research nations in the world.^{Footnote 37} Health Canada provides evidence-based, authoritative information to Canadians and key stakeholders, including health professionals such as physicians, pharmacists and natural health practitioners, to enable them to make informed decisions. Health Canada’s emphasis on evidence based information echoes CIHR SPOR’s focus on evidence based research. SPOR has been referenced directly within federal level priority statements, such as those delivered in Canada’s Economic Action Plan, and CIHR Departmental Performance Reports (DPRs), Reports on Plans and Priorities (RPPs), and Strategic Plans. Additionally, patient-oriented initiatives have been reflected throughout various federal priority statements, such as within Health Canada and Public Health Agency of Canada (PHAC) objectives, thus demonstrating the federal commitment and role in supporting patient-oriented and evidence-based approaches.

“[CIHR is the] only national group trying to put solutions to this [patient-oriented research].”

All key informant interviewees and case study interviewees stated that it was critical CIHR play a role in advancing patient-oriented research. Many suggested existing efforts would not be present if CIHR were not supporting SPOR, not only financially, but also from a structural and coordination perspective. Pockets of POR activities may have been active, but not to the national extent triggered by SPOR. A number of reasons were cited for the criticality of CIHR’s involvement, including:

• Being key to getting POR on the provincial/territorial scene with Health Ministers and Provincial/Territorial Premiers.
• Having international and national credibility and the ability to gather information and input from other entities, and bring communities together more easily than individual provinces/territories or sectors.
• As the national health research funding agency, CIHR is the logical agency to drive collaborative involvement of the research community.
• Being directly aligned with CIHR’s mandate of improving health for Canadians.
• Having existing processes (e.g., peer review, granting and funding mechanisms, familiarity for the research community), albeit modified, to support the roll-out of a different strategy.

In recognizing the role of CIHR, a few key informant interviewees also identified the importance of provincial/territorial involvement and buy-in as being necessary as well, as they are the ones managing the respective health care systems.

3.1.2.3 SPOR’s alignment with government priorities (Evaluation Question #3)

In October 2013, the President of the Treasury Board approved a new Program Alignment Architecture (PAA) for CIHR that took effect April 1, 2014. The PAA consists of one Strategic Outcome and three Programs (including Internal Services) that support the Strategic Outcome. Within the updated PAA structure, SPOR objectives have been identified predominately within “Priority 2: Priority Driven Health Research.” The goal of this program is to advance health knowledge and its application, in specific areas of research identified by CIHR in consultation with other government departments, partners and stakeholders, in order to improve health systems and/or improve health outcomes in these priority areas.

Additionally, the over-arching strategic objective for CIHR did not experience significant changes through the update to the PAA structure, with the objective being for Canada to be a world leader in the creation, dissemination and application of health research knowledge.

The mandate of CIHR as reflected in its Strategic Plan 2014-15 to 2018-19 – Roadmap II^{Footnote 38}, particularly pertaining to knowledge translation, is “improved health for Canadians, more effective health services and products and a strengthened Canadian health care system.”^{Footnote 39} This is aligned to, and complementary of, SPOR’s objective to “foster evidence-informed health care by bringing innovative diagnostic and therapeutic approaches to the point of care, so as to ensure greater quality, accountability, and accessibility of care” which was first identified in 2010.

The CIHR Strategic Roadmaps have undergone various iterations, but have remained highly aligned throughout the years. The Roadmaps consistently set out a high-level vision with relatively consistent strategic directions and supporting objectives. Key areas notable for SPOR are depicted in the Exhibit 6.

Exhibit 6 – CIHR’s Strategic Roadmap 1 and 2

• 2009-10 – 2013-14 (Roadmap 1)
  • Enhance patient-oriented care and improve clinical results through scientific and technological innovations, increase focus on collaborations between researchers and users of research knowledge to increase uptake of findings.
• 2014-15 – 2018-19 (Roadmap 2)
  • Strategic Direction 1: Focuses on promoting excellence, creativity and breadth in health research and knowledge translation – broader disciplinary mix of researchers culture of excellence in knowledge creation and translation.
  • Strategic Direction 2: Focuses on mobilizing health research for transformation and impact – health issues important for patients.
  • Research Priority A: Enhanced patient experiences and outcomes through health innovation.

Most recently the Report of the Advisory Panel on Healthcare Innovation (the Naylor Report) identified three priorities for innovation including PE and empowerment, health systems integration with workforce modernization (research to practice), and technological transformation via digital health and precision medicine (right treatment, right person, right time). The Report specifically identifies SPOR as synergistic with some of the objectives delineated in its report.

“SPOR is complementing discussions in the provinces on how do we do things differently. For example, efforts in the clinical trials area. How do we really change the environment to help entice more activity back to Canada? What SPOR brings to the table complements what jurisdictions are already embarking on."

Additionally, and key to SPOR’s approach, is the alignment to provincial/territorial objectives and priorities in health care. The concepts of PE, POR, and evidence-based healthcare are not new, and there are other initiatives within Canada that address them to some degree. Most key informant interviewees and interviewees across all case studies noted that SPOR is complementary to provincial and territorial initiatives in patient-oriented research and there is joint interest between federal and provincial/territorial priority areas. The creation of synergies and focal points to address priorities and improve the conduct of health care delivery were key reasons. In part through leveraging and connecting with existing provincial/territorial organizations already working on delivery of programs and policies that will improve the conduct of health care delivery. It was also cited that SUPPORT Units, in particular are aligned with provincial/territorial efforts, through their extensive initial needs assessment exercises and ability to build services in a flexible manner aligned to provincial/territorial needs.

3.1.3 Conclusions

SPOR is relevant to the Canadian health care system and is responding to key needs intended to drive research into practice with the main objective to impact health outcomes of Canadians and enhance patient experiences in health care systems across the country. SPOR is “in play,” being part of the international trend towards employing patient-oriented research models, and is perceived to be one of the “quite vibrant” initiatives being implemented. SPOR is aligned with Canadian public policy, in the government’s long standing drive for patient-centered research, and measures up to Canadian public opinions (i.e., as measured by Research Canada) that research, supported by government efforts, is an important contributor to improving health outcomes in which the public has a desire to be involved.

The definition of the core elements of SPOR is complete and relevant at this time; meeting needs as requirements have evolved^{Footnote 40}. Each element of the strategy has been designed to meet key needs in areas of increasing collaborations among stakeholders, enhancing access to common resources, providing support for research that can be scaled up and translated across sites and regions, facilitating patient involvement in research, enhancing the clinical trials environment and building capacity around patient-oriented approaches with both researchers and patients. The extent to which SPOR is meeting needs is highly influenced by the current early state of implementation of each element.

There is clearly a role for CIHR to play in SPOR. First and foremost, SPOR is a direct extension of the Canadian Institutes of Health Research Act. The Act states CIHR’s main objective is to “excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system.” Full support for CIHR’s involvement was furnished by all categories of stakeholders, from researchers to policy makers, acknowledging its logical place in delivery as the national health research funding agency in Canada and its international and national credibility. Successful coordination with the provinces and territories was believed to be due to CIHR’s national structure and ability to bring commonality in design and delivery. In addition, respondents noted SPOR’s complementarity to provincial and territorial initiatives in patient-oriented research as there is joint interest between federal and provincial/territorial priority areas. The creation of synergies and focal points to address priorities and improve the conduct of health care delivery were key strengths for the successful coordination of SPOR with pan-Canadian priorities CIHR’s integral role is also acknowledged by the international community.

3.2 Design and Delivery

3.2.1 Key Findings

• There were significant efforts undertaken by CIHR to bring provincial partners on-board, including a two-year engagement and planning process with subsequent time to mobilize partner resources. The buy-in of provinces and territories to the objectives of SPOR is critical.
• The national SPOR governance model is a strong mechanism. The national composition with lateral structures across provinces and territories for inter-jurisdictional learning, and the corresponding structure that includes representatives external to CIHR, promotes a “lets discuss together” approach. However, areas for improvement were noted, including:
  • The need for further enhancement of PE inclusion across the governance structures; and,
  • A need to increase the national approach to governance and delivery in support of all core elements to enhance alignment, integration and limit potential duplication.
• CIHR’s engagement efforts are viewed by stakeholders as being “exceptionally” active in developing relationships and reaching a broad group of stakeholders across sectors. However, there is room for learning with respect to best practices in the integration of patient representatives.
• Funding is generally seen to be timely and flexible where required (e.g., eligible costs) and there were minor points raised regarding challenges encountered with operating funds. However, major concerns were raised around the sustainability of funding following the initial five-year funding period, where many interviewees identified risk to ongoing operations and sustainability of activity and momentum for continued uptake of POR. The 1:1 funding requirement was also raised in connection with anticipated potential issues in sourcing funds from the same origins as other SPOR elements (e.g., provincial/territorial Ministries of Health) and the potential for mismatch of provincial/territorial vs. national priorities.
• Communications need improvement. There is a perceived lack of advancement in communications surrounding mandates, available support, consistent and common definitions of POR and PE, and the tailoring of communications towards target audiences.
• Performance measurement was consistently identified as needing attention. It is not clear that the right things are being measured (activities vs. outcomes) and the collection and reporting of performance data is perceived to be quite onerous for funded recipients.
• Peer review, with the inclusion of patient representatives, was seen to be a positive novel approach and the process for Network development and selection was noted to have continuing effects on even those projects that were not selected for funding. However, PE in peer review still requires increased efforts for training and clarification of roles and responsibilities.
• Provincial/territorial priorities and needs are heavily considered in the SUPPORT Unit development process – this is likely the most important early outcome of the SUPPORT Unit implementation and is considered a great strength of the flexible approach CIHR has taken. However, consideration for the provincial/territorial priorities and needs has impacted the length of time for implementation for the SUPPORT Units (and Networks) and most are still working towards integrating and coordinating all required SPOR components.
• A key area of design concern is the anticipated interactions of the SUPPORT Units and Networks and the Networks with other Networks. It is not clear how these elements are to work together, nor to what extent. There is also uncertainty with regards to how funding of services will be handled and whether possible double-billing will occur, for example, sourcing funding for the same project that may be supported by multiple SPOR elements and seen as “double-dipping” at the project level.
• A dedicated operational leadership position for the funded entities within SPOR elements (e.g., SUPPORT Units and Networks), tasked with program/project and administrative management, was identified as a best practice and one that is necessary for efficient operations.

3.2.2 Analysis

3.2.2.1 Design and delivery at the strategy level (Evaluation Question #4)

SPOR was initially launched in 2011, with release of the formal Strategy document. In the five years since that time, each SPOR element has been developed and implemented within varying timeframes. There were significant efforts undertaken by CIHR to plan and operationalize the SPOR vision and activities that would support its objectives (i.e., bringing the theory of SPOR into practice), as follows:

• provincial/territorial partnership development, including a two-year engagement and planning process with subsequent time required to mobilize partner resources;
• funding opportunity design and delivery;
• establishing SPOR operational and governance structures;
• broad stakeholder engagement and communication about SPOR; and
• funding early capacity development activities.

According to the data collected on the CIHR grants and awards expenditures on SPOR, it took three years after launching the strategy in 2010-11 to begin expending funds to the SUPPORT Units and Networks, with a gradual ramp up of funding over the 2013-14 through 2015-16 period (see Exhibit 21 in the cost-efficiency section). Official program documents indicate there was a planned delay through the first two years of SPOR with activities identified for 2011-12 to include the establishment and engagement of SPOR’s National Steering Committee, the determination of priorities, the creation of the funding opportunities for the Networks and SUPPORT Units, as well as some specified work in the Clinical Trials area. The unplanned delays are further articulated in section 3.3.3.

Key informant interviewees specifically mentioned the federal-provincial/territorial engagement, cooperation and collaborations, including the buy-in of provinces and territories to the overarching objectives of SPOR, as key to the current successes of SPOR. Key informant and case study interviewees identified a number of elements designed well and delivered in an effective manner with the implementation of SPOR. For the most part, process and structural areas were rated to be effective to a moderate extent or higher with the exception of one area – performance measurement.

The results of the effectiveness ratings, by area, are presented in Exhibit 7 and further elaborated with discussion of each area following the chart.

The areas rated highest, on average, for effectiveness include SPOR governance structures, SPOR engagement of stakeholders, and delivery of SPOR funding. These three areas were seen to be effective to a great extent by respondents.

Performance measurement was rated lowest, viewed to be effective to almost a moderate extent on average, and was the one category with the greatest number of respondents identifying that the performance measurement area was not effective at all.

Exhibit 7 – Extent of effectiveness in delivery of processes and structures – Strategy level

*Rounding has been forced to add to 100%.

* Average rating has been calculated as a standard weighted average based on the number of respondents.

SPOR Governance

The SPOR governance structures received one of the highest average ratings. The main characteristics of governance effectiveness were noted by key informant interviewees to be the national coordinating body with lateral structures across provinces and territories for inter-jurisdictional learning and the corresponding structure that includes representatives external to CIHR (i.e., a “lets discuss together” approach). The buy-in of the provinces and territories was noted to be critical in moving SPOR to the position it is now. Most key informant and case study interviewees noted that SPOR did have the right people in the right places for effective governance.

Through the CT and SUPPORT Unit cases, CIHR staff were noted to be supportive and open, and viewed CIHR as striking a balance between being hands-off in the involvement of the work conducted by each of the SPOR elements while maintaining support for the elements.

“SPOR has lots of hierarchy and needs more coordination. There are 5 separate ships going in separate directions, then seven or eight working groups, we don’t know what they all do."

However, many of the case studies (CD, PE and CT) found there was a perception for the need to strengthen a more national approach to governance in support of all elements to enhance alignment, integration and limit duplication, in particular for the SUPPORT Units and Networks. Recognizing there are mechanisms in place that are intended to enable cross-jurisdictional exchange and collaboration (e.g., the SSUC and its working groups), the case studies identified that direction at the national strategy level could be enhanced with clear central leadership. One specific example provided in the PE case was that the SPOR SUPPORT Unit Council PE Working Group is still in the process of defining itself, and would require increased clarity in definition moving forward. A lack of consistency in approaches to PE was noted across the SUPPORT Units, and it was suggested a stronger national approach would be beneficial to accompany the SPOR SUPPORT Unit Council PE Working Group. It was also identified through the SUPPORT Unit case study that the roles and actions of the SUPPORT Units versus the SSUC Working Groups were not well defined. Interestingly, the international key informants also identified that with SPOR having SUPPORT Unit operations governed at the provincial level, there may not be enough interaction at the federal level. Each of the international comparative initiatives are more targeted towards a national POR approach.

Another area identified as having potential for enhancement is the engagement of patients across the various levels of governance. This was identified in the PE and CT case studies and is consistent with findings in the performance area (see section 3.3.1.1), where it is noted there is a need for further efforts to enhance PE across the various levels of governance. This includes support for patients to engage as equal stakeholders at these levels, inclusive of governance training.

This perception, whether accurate or not, is supported by the results of the governance and administrative network analysis. The patient component of the Network, albeit connected to many of the governance bodies, is one of the smallest pieces of the structure. The network analysis was based on publicly available information and even though does not paint a complete picture of SPOR, does imply that the communicated patient involvement may be misunderstood.

Engagement of stakeholders

All key informant and case study interviewees noted very extensive efforts in SPOR’s engagement of stakeholders, identifying SPOR as being effective in engaging a broad group, from the public, private, and academic sectors. Particularly in the CT and PE cases, respondents noted that “all kinds of efforts” have been made at both the national and provincial/territorial levels. Additionally, the ability to bring various stakeholders together at the same table is an asset of the CCTCC that was noted to be unique within Canada.

Further elaboration on the engagement of stakeholders is documented in the section on Performance (see section 3.3.1.1).

Delivery of funding

Key informant interviewees identified that the requirement to secure matching funding contributed to the variation in timing of establishing some of the SPOR elements, particularly the SUPPORT Units.

SPOR funding was seen to be timely and flexible where required. Both the SUPPORT Unit and Networks case studies identified initial negotiations were required to determine categories of eligible costs, as the SPOR model is outside the bounds of traditional CIHR eligibility with an increased need for operating and service delivery funds. However, there is a concern in the CT area where it is not seen that SPOR will cover all expenses associated with running clinical trials. Additionally, the SUPPORT Unit case study identified that funds were said to be intended to support infrastructure and there are no operating funds, so the SUPPORT Units are sometimes searching for funding for specific projects^{Footnote 41}. Given the perceived high competition rates for CIHR Open Grants if researchers have no connections with large industry (e.g., pharma) to find matching funding, the SUPPORT Unit case study identified funding as a significant challenge.

Through the Networks case, it was identified that consideration should be given to the timing for distribution of funds. CIHR’s funding model assumes that at the moment funding begins, participants already have the required infrastructure and delivery models in place, neither of which was true in the case of ACCESS Open Minds – some ramp-up period is required. Key informant interviewees also identified the ramp-up activities required to establish infrastructure and new corporate entities.

The biggest issue raised around funding is the uncertainty surrounding the ability of CIHR to continue to provide financial support for SPOR, and hence the sustainability of partner funding was also raised, despite there being a requirement for the SUPPORT Units, in particular, to develop sustainability plans^{Footnote 42}. The five year funding term was a concern in the CT, PE, and SUPPORT Unit cases. Key informant interviewees also shared this viewpoint. Key informant interviewees suggested the short timeframe to build sustainability around the concepts and results associated with SPOR was considered to be the biggest challenge. For example, if CIHR funding was to disappear at the end of the funding term, would the focus on patient-oriented research and conducting research in this different way still be embraced? This was acknowledged to be a challenge not only in regards to sustaining a major paradigm shift in how research is done, but being able to do this within a very short period of time. Concerns were also expressed around the risk of escalating costs (i.e., as POR takes more time and resources) or decreasing funds and the ability of funding partners to maintain the levels of financial commitment required. None of the provincial government respondents interviewed suggested that their ministries would provide 100% funding for SPOR initiatives, rather than the 50% now being provided. Some SUPPORT Units are already trying to build various kinds of fee-for-service models into their programming, e.g., for training, but this is considered unlikely to be able to maintain SPOR momentum. The issue around the short funding term was also noted to potentially impact the adoption of POR approaches due to uncertainty of the sustainability and longevity of support.

Coordination of communications

Issues around the coordination of communications was identified by the KI interviewees, the PE, Networks, CT and SUPPORT Unit cases.

“Communications is a big pet peeve – the initiative is not well communicated or well understood – people have a “blank stare” when you talk about SPOR.”

Communication of concepts and results were perceived to be highly important and noted to take dedicated resources and efforts to accomplish. Knowledge translation, best practice dissemination, and sharing of successes were noted to be required to keep momentum moving and for SPOR to achieve its greatest potential. Without an enhanced communication structure effectively outlining SPOR objectives and deliverables, and success stories of POR or PE achievements there is a risk of misunderstanding “what SPOR is, and what SPOR is not.” Further, it was noted there is a need to clarify definitions, such as that of PE and patient-oriented research, to reduce confusion and contribute towards enhancing adoption within all stakeholder communities, including the public. Lessons learned included the tailoring of communications towards target audiences (also noted by the US international interviewee), and potentially enhancing website and social media communications by CIHR and the SUPPORT Units and Networks^{Footnote 43}. Respondents identified the need for consistency of communications and messages from the many possible national and provincial/territorial sources was critical.

“[We’d love to see an announcement for a webinar for each platform, each platform lead could talk about what they could do for [researchers] . . . more information, how/ when/ where – and costs would be useful.”

Some key informant interviewees and the SUPPORT Unit interviewees also noted the increased need for communication with stakeholders by SUPPORT Units. It was identified that SPOR could help to make SUPPORT Units better known through increased engagement with the research community, including through the engagement of foundations, research institutes, and charities.

The international interviewees noted that convening the community and a need to maintain visibility of the POR agenda is one of the most important elements. Bi-annual conferences are held in the UK and were noted to be getting bigger and more diverse in the mix of attendees participating. As the UK is also structured with a mix of many entities focused on patient-oriented research (or parts of it), it was identified that they attempt to inform participants/stakeholders as to what one another are doing to ensure some consistency – particularly in relation to training – but did not elaborate on how they do this.

The US initiative noted that it spends a lot of time on communications and particularly dissemination of results. The US interviewee identified that researchers are typically not necessarily the best disseminators, hence the importance of integrating stakeholders into the research, so ownership is established and more enthusiasm for dissemination of results is generated. Having partnerships with organizations that reach patients, clinicians, and health organizations is important as it is these that reach the decision-makers.

Performance measurement

A performance measurement strategy does exist for SPOR as a whole. Linked to this is a common performance framework for the SUPPORT Units with annual performance reporting requirements as well as performance reporting requirements for ACCESS Open Minds.

Although the program stated that progress reporting requirements were developed and refined collaboratively through consultations with partners and funding recipients, issues with current performance measurement processes, tools and targets were consistently identified across the key informant interviews and the case studies, including the SUPPORT Unit, Network, CT (specific to the CCTCC) and PE cases. The issues around performance measurement were identified by individuals who receive the reporting (advisory members and funders), as well as those required to submit the reporting (SPOR element leads). There is a need to continue to develop relevant performance measures.

The key informant interviewees perceived a lack of attention to specific outcome performance measures and the uncertainty surrounding what is required and expected in the longer term. In addition the varying requirements of all partner funders was an issue. It was noted there is no consistency in requirements or expected outcomes across partners which was identified to be quite burdensome for the SUPPORT Units in particular. The SUPPORT Unit and Network case study support this viewpoint with the SPOR performance metrics being identified as extensive and onerous to complete. Few SUPPORT Unit case study respondents believed it was effective in measuring outcomes and impacts, and several respondents commented it was very difficult to understand exactly what the SUPPORT Units and SPOR were accomplishing, especially given other non-SPOR PE and POR initiatives within the provinces/territories. In addition to the workload, a few respondents commented that the static “activity” metrics will not contribute to SPOR’s sustainability as they will not prove the value of POR/PE approaches, as opposed to a scientific approach to measuring what works best and when.

One specific area identified through the PE case study is a need for detailed evidence of the use of the PE framework by SUPPORT Units supported by proof of meaningful engagement of patients to better inform the measurement of the progress and impacts of the initiative. As an example, it was suggested a measure of patient opinions on levels of engagement in preparing applications for research grants could be considered.

The importance of robust performance measurement framework was highlighted by the international key informant interviewees as well. All international interviewees highlighted the importance of ensuring outcome driven performance measures, set in a way to encourage alignment of the initiatives to delivery on outcomes. Each international interviewee felt it was integral to report on the impacts and the overall progress of the initiative. As an example, performance measures under CLAHRC are viewed as being directed towards delivery of their research results. The metrics serve as quality drivers for the translation of research through application to customer uptake (based on public/patient opinion). One key area of improvement for SPOR, as identified by international interviewees, was to ensure data being captured includes applicants, reviewer comments, milestones, and progress to date against plan. All international interviewees perceived SPOR as not clearly defining the impacts it intended to make and how these should be measured from the start of the initiative. International interviewees believed SPOR requires enhanced clarity on how it would assess its overall progress to date, as stakeholders need visibility of the agenda in order to maintain the momentum of the initiative and support for it. Further SPOR was encouraged to define measures focused on whether the initiative was “driving better outcomes.”

Consideration for robust performance measurement was a lesson learned in the UK and is still something to be resolved. One international respondent noted that their initiative began without defining what clear impacts it should be measuring. This is an ongoing challenge for the future to be able to answer this question more appropriately, however, the UK has set three measures:

• Reach: how research is reaching into the population?
• Relevance/Priority Setting: where is the public impacting research but also evidence to show what is funded is what the end user really wants.
• Refinement: how is research more effective and efficient?

Risk Management

There were very few key informant interviewees or case study findings that further elaborate on the effectiveness ratings provided by respondents. Basically, there was not enough exposure to this area for respondents to comment knowledgeably. For the key informant interviews, it was identified that for the most part, the risk management area was impacted by other processes, such as slowness of implementation, communication and concerns related to the robustness of the performance measurement structure, and generally the uncertainty of how risks are managed.

Implementation of peer review

The engagement of patient stakeholders in the peer review process was seen as positive and a novel approach within the CD, Networks and PE case studies identifying the effectiveness of this process. However, it was suggested in both the CT and PE cases that there is still increased support required for patient representatives to be fully engaged. The need for increased training for patient representatives to assist in developing skills to contribute towards the conduct of peer reviews, including enhancing communications in terms of patient roles, responsibilities, and time commitments required, and engaging peer review personnel through increased in-person meetings would assist with this process.

Through the Networks case study the application and proposal review process were identified to have been transformative in terms of generating interest in starting up parallel projects, and in continuing collaborations among some of the teams that did not receive Network funding. One respondent commented that although this process was time and resource intensive, it provided significant dividends for both successful and unsuccessful applicant teams.

3.2.2.2 Design and delivery of core elements (Evaluation Question #4)

Two SPOR core elements have independent implementation structures with facets of design and delivery separate from the overarching strategy. This section covers the design and delivery elements of the SUPPORT Units and Networks. The design and delivery of the PE, CD and CT elements are included in the discussion above on the implementation of the strategy as a whole as PE and CD crosscut all elements and the CCTCC is a shared resource.

Each of the case studies has identified individual elements of design that are working well in addition to areas of challenge that require ongoing attention. Collectively the case study respondents rated their respective delivery structures as being effective to some extent. It is important to note that all SPOR elements are very young, with the earliest implementation being achieved by four SUPPORT Units established in 2013. All other pieces have been launched subsequent to this, so for the most part have only been operational for one year. As these structures are still evolving, it was difficult for respondents to know how well these governance and administrative structures will function once fully developed.

Exhibit 8 – Extent of effectiveness in delivery of governance and administrative structures – by SPOR element

*Rounding has been forced to add to 100%.

*Average rating has been calculated as a standard weighted average based on the number of respondents.

Each SUPPORT Unit is organized and governed somewhat differently, and in these mostly early days it was difficult for respondents to know how well these governance structures would function once fully developed. However, the SUPPORT Unit components were developed through extensive consultation with stakeholders from all sectors to address key research priorities and/or fill critical gaps in capabilities. Thus, these consultations are considered key to implementing the SUPPORT Units effectively. The consultations have helped to ensure that matching funding has been available, and that local stakeholders find the SUPPORT Unit services and priorities to be in line with their own. Of all aspects of governance and administration, this was by far the most frequently cited as important and effective. This is also true of the PIHCI network, where the phased approach to developing and rolling out the network, which allowed for sufficient time to consult on design and ensure it would meet the needs of the community was noted to be a key strength.

But clearly the SUPPORT Units have taken longer – in some cases, far longer – to be implemented than was planned. No single and consistent reason was offered by SUPPORT Unit case study respondents for this situation, but the attempt to be “all things to all people,” coupled with the complexity of the many national, provincial/territorial, and SUPPORT Unit components, along with the complexities of the university research and provincial/territorial healthcare environments, have all certainly played a role. Simply getting appropriate stakeholders together and agreeing on how to proceed has been very time consuming. Once operational, implementing the budget (both cash and in-kind) among the many funding partners was said to be very complex.

This is not unexpected and confirmed by the experience of the international comparative initiatives where it was noted to be a challenge within the CLAHRC’s to to bring together various stakeholder types, inclusive of patients, researchers, and policy makers, together to work towards a shared vision. Similar to SPOR, it was stated that different types of stakeholders tend to work and think in different ways, leading to some struggles in initially learning how to work together.

Similarly, the governance structures for ACCESS Open Minds are seen by respondents as moderately successful to date, but they are still evolving significantly (and frequently) from what was originally proposed, and all parties are working to make them effective. The ACCESS Open Minds case study respondents found it very difficult to predict how long it will take to get ACCESS Open Minds fully operational. Participants originally thought it would take eight months to a year to “get there.” The Network grant came in the middle of 2014, from which it took 3-4 months to set up the Network, with respondents noting ACCESS Open Minds has been about 11 months in development now and it is still not operating at its intended scale. This is considerably slower than the timeline originally projected.

The main PIHCI governance mechanism (at this point in time) is the tri-partite Interim Leadership Council which has one clinical, one research and one policy representative from each province/territory. Although the PIHCI Network is not operating at its intended scale and PIHCI respondents were not asked explicitly about this topic, some participants made comments that indirectly suggest they believe governance and administration will be effective. For example, they commented that PIHCI builds upon existing community-based primary care research infrastructure in several provinces (e.g., Alberta), and PIHCI can leverage that as an interprovincial collaboration unit. The provincial network integrated into a national network (i.e., network of networks model) is considered by some respondents to be a strong design feature that is necessary to encompass the diversity of primary care and to share and learn from each other’s success and failures.

Understanding the early stage of these elements, learnings from implementation experiences to date can be identified, and in some cases, are shared across the SUPPORT Units and Networks. The biggest implementation challenges noted through the SUPPORT Unit and Networks case include:

“One problem is the interaction among SUPPORT Units and Networks, and between SUPPORT Units – I don’t feel terribly engaged yet. We’re not yet developing collaborative, cohesive programs.”

• Interactions of SUPPORT Units and Networks. An issue mentioned many times was how the SUPPORT Units will (and should) interact with SPOR Networks. This was also mentioned in the key informant interviews. Interactions to date has reportedly often been “ad hoc” – respondents said the Networks want to know what the SUPPORT Units can provide, while the SUPPORT Units must consider what is realistic given their own financial, time, and human resource constraints, as well as the fit with provincial/territorial priorities.

  A further issue with respect to the SPOR Networks is one of multiple jurisdictions being involved. For example, if a network is housed in several provinces but not centered in the one where a SUPPORT Unit is being asked for assistance, then how does (and should) that non-central SUPPORT Unit participate? This was said to not be a huge problem to date, but the future situation is unclear. There is the perception of expectations that SUPPORT Units will help, but there is currently no mechanism to decide how many resources come from which SUPPORT Unit (especially as the researchers will be spread amongst several provinces/territories), and the Network asking for assistance may not match the non-central SUPPORT Unit’s provincial/territorial priorities well either.

  Further, it is unclear how (or if) the SUPPORT Units are to bill SPOR Networks for the SUPPORT Unit services, and to date this has been very complicated to figure out. As an example, the following question was raised by a SUPPORT Unit; the Networks are CIHR (and partner)-funded, so if a researcher has a Network or other CIHR grant, what services are the SUPPORT Units obligated to provide, and for what fee? Charging for these services could be seen as double-billing CIHR or potentially the partner funder. Further, the SUPPORT Units are committed to providing many other non-Network services, so their capacity to act as resource centres for Networks – especially as more Networks come on line – was believed by many respondents to be in doubt.

  Through the case studies, networks working with networks was also identified as unresolved. Through the PIHCI survey in particular it was noted that other pan-Canadian Networks are intended to be implemented in the future, and how the PIHCI Network will interact with other Networks is also uncertain. Further, respondents believe these future networks should interface with the (provincial/territorial) PIHCI networks, but with the current limited resources and with support so “thin on the ground,” there is little time to coordinate possible activities, or to even just know whom to consult.

• Matching funding. While the 1:1 matched funding extends the federal/provincial partnerships, it creates issues around communications, expectations, and governance where national and provincial/territorial priorities do not match. Most respondents noted that it was unclear where and how conversations to solve these problems are taking place. On the other hand, for a few of the provinces (and not just the large ones), the matching requirement is actually seen the other way around – because the SUPPORT Units are linked to provincial/territorial priorities, the provinces view they can accomplish their own projects for “50 cent dollars”; i.e., SPOR is seen to be matching them.

  But still, one of the major challenges is ensuring 1:1 matching funding can be found. An example provided by SUPPORT Unit interviewees; since the PIHCI model calls for networks to be formed and supported provincially/territorially, matching funding for core network support and for network projects may potentially be more easily obtained at the provincial/territorial level. However, these provincial/territorial funds also need to support multi-jurisdictional projects, so there is a potential mismatch of provincial/territorial vs. national needs. This issue is also relevant to SUPPORT Unit collaboration and noted above. For example, it is currently unclear how the SUPPORT Unit’s contributions will be taken into account going forward in new PIHCI projects. It is considered unlikely that PIHCI can obtain future partnership matching funds from the same source as for SUPPORT Units (e.g., a health region, or a Ministry of Health), when these sources have already committed their matching fund dollars to the SUPPORT Unit. Further, such matching would potentially be to the same project that the SUPPORT Unit is partnering on with a Network, and therefore respondents believed this could be seen as “double-dipping” at the project level.

• Operational leadership. This challenge was identified in both SUPPORT Unit and Networks cases. Most SUPPORT Units have a single ED and single individuals responsible for SUPPORT Unit components, and report no issues with this other than high workloads. Some have co-leads for the SUPPORT Unit components, and this also reportedly works well as the two individuals (and their two institutions) have different strengths and complement each other. However, SUPPORT Units create a significant extra burden for their leaders, not all of whom have a great deal of time to deal with the workload. Several respondents (from currently funded SUPPORT Units) within government ministries reported doing SPOR “off the corner of their desks.” Further, some are excellent “big picture” strategic thinkers but may not have a great deal of operational experience. Thus, it was suggested that SUPPORT Units would benefit from having a position akin to an Operations Manager – e.g., not getting involved in technical details, but keeping track of time loads, overlaps, communications, etc.

  Similarly, ACCESS Open Minds also had challenges with its operational leadership function. The Executive Committee in the central office has struggled somewhat with the work load and project management and business planning functions. ACCESS Open Minds has just recruited a national director at a CEO level for the network in November, 2015. It is hoped the new CEO will help deal with the administrative work load. Leadership, administration, and business management skills in such a complex initiative are much more important than in a “normal” research grant project. The areas of project management and organization are noted to be the biggest challenges going forward (e.g., national agenda setting, strategic and site-specific planning, communications within and between the sites and central office, knowledge acquisition among sites to avoid duplication of activities, scheduling, funding, and inclusion of all stakeholders as equal partners and participants.)

• Coordination and integration of SUPPORT Unit components.Although the SUPPORT Units have put in place components corresponding to provincial/territorial needs and priorities, integrating and coordinating the required SUPPORT Unit components was reported to be a significant challenge in several jurisdictions. An extension of this also applies to leveraging existing government capabilities. In part this is said to be a problem of managing expectations, including the timing challenge of launching these initiatives all at once, and expectations from SPOR Networks of being able to sometimes obtain instantaneous support. As most of the SUPPORT Units are quite new, integration of the required components is often far from complete, and they do not always work together properly or consider each other’s needs in their planning. The SUPPORT Units reported, however, that after a sometimes “rocky start”, they are finding their way.

• Human resources in smaller provinces/territories. The very limited financial and (especially) human resources in smaller provinces/territories is a significant stumbling block to developing effective SUPPORT Unit operations. Many respondents in these environments noted that the extensive stakeholder interactions expected of them were very difficult to accomplish given these constraints (even flying stakeholders to meetings is extremely expensive in the territories), and SPOR had to be done “off the corner of their desks”. One respondent in the territories noted that the same people are in every meeting – so the capacity in [this territory] is essentially “one person deep”. Another ED in a province with a functioning SUPPORT Unit noted a similar HR problem – when one or two key people left, initiatives were challenged.

• Stakeholder engagement – In ACCESS Open Minds, this has been heavily promoted, and is seen as very progressive by respondents. Discussions have helped Network participants understand what is meant by “stakeholder engagement”. However, the Networks case study found the degree to which patient voices are well integrated into the model and determination of the best ways for patient representatives to participate are ongoing issues.

3.2.3 Conclusions

Some of SPOR’s areas of implementation are on track to becoming effective, but not all. The design of the strategy overall is sound, with national governance mechanisms seen by stakeholders as generally strong and appropriate. Key success factors to effective SPOR governance were noted to be its national composition accompanied by lateral structures across provinces and territories to facilitate inter-jurisdictional learning and the inclusion of representatives external to CIHR, promoting a “let’s discuss together” approach. Stakeholder engagement is seen to be “exceptional” and SPOR has been quite successful at developing relationships and reaching a broad group of stakeholders across sectors and participant types.

Peer review processes are also strong with the inclusion of patient representatives seen to be a positive and novel approach. The extensive and iterative process for Network development and selection was noted to have continuing effects on even those projects that were not selected for funding (i.e., groups not funded are still working together).

The most important early outcome of SPOR implementation is that provincial/territorial priorities and needs were heavily considered in the SUPPORT Unit development process, and this is considered a great strength of the flexible approach CIHR has taken. However, this has impacted the length of time for implementation for the SUPPORT Units and most are still working towards integrating and coordinating all required SPOR components. A dedicated operational leadership position (for the funded Networks and SUPPORT Units, tasked with program/project and administrative management was identified as a best practice and one that is necessary for efficient operations.

However, there are opportunities for improvements in delivery to support the effectiveness of ongoing implementation. Some of the most prevalent opportunities are the requirement for further enhanced engagement of patients within SPOR’s governance structures as well as those of SPOR funded entities and a need to increase a national approach to governance and delivery in support of all elements, but particularly to enhance alignment, integration and limit duplication among the SUPPORT Units.

Communications overall need improvement where further clarification of the mandates for each of the SPOR elements, available support by the SUPPORT Units, consistent and common definitions of POR and PE, and communications targeted to different stakeholders is required. This also includes sorting and communicating how the individual SPOR elements are intended to work together, particularly the SUPPORT Units and Networks, but also applies to how Networks will work with other Networks. It is also not clear to what extent this is intended or expected to occur.

Performance measurement was also consistently identified as needing attention. It is not clear that the right things are being measured (i.e., considerable measurement of activities and limited measurement of outcomes) and the collection and reporting of performance data is perceived to be quite onerous for funded recipients.

There are major concerns surrounding the sustainability of activities and momentum for POR within the initial five year funding timeframe. The 1:1 matching funding requirement also raised concerns with respect to sourcing funds, as many SPOR elements may potentially be approaching the same organizations for funding multiple times, for initiatives which could be seen to be overlapping. As well, there are anticipated conflicts that may be raised by the possible mismatch between provincial/territorial and national priorities and concern regarding how these types of issues may be resolved, let alone funded.

3.3 Performance

3.3.1 Key Findings:

SPOR is in the initial phases of implementation with all core elements at various stages of development and operation, but SPOR appears to be on track to achieving most of its expected immediate outcomes:

• Multi-disciplinary, integrated and accessible research platforms are established. The main areas of achievement include establishment of six SUPPORT Units.
  • Research platforms (i.e., the SUPPORT Units) have been established and are providing support and services in most of their key component areas.
  • Ontario and Manitoba are the most advanced of the SUPPORT Units, with others at various stages of planning, development and operation.
  • Many of the SUPPORT Units are putting high priority on building data platforms within which researchers and policy makers will be able to access and use a wide variety of relevant provincial data. SUPPORT Units have increased their dataset holdings from 253 to 534 over 2013-14 to 2014-15.
  • Consultation and research services provided by the SUPPORT Units centered around data analyses and/or data management activities, knowledge translation support, POR study design (including methods) assistance, and support for research proposal/grant application submissions. SUPPORT Units have reported increasing the services provided from 376 to 1,872 across 2013-14 and 2014-15.
• Pan-Canadian networks are mobilized to address health challenges. The ACCESS Open Minds network is the only active network, with one site operational at December 2015. ACCESS Open Minds reports that central office space has been secured, the main governance structures began to be put in place during the reporting year and key staff have been hired for the central office. PIHCI is still ramping up and the five chronic diseases networks were announced in March 2016. The Networks are too new to have assessed outcomes at the time of the evaluation as they are still undergoing design and delivery adjustments.
• CCTCC is established. The CCTCC has been operational for just over a year and has delivered against two of its key objectives including the completion of a model clinical trials agreement and the completion of the clinical trials asset map. Ongoing work is underway to address the other seven recommendations of the Canadian Clinical Trials Summit Action Plan. The clinical trial environment has not been influenced to any great extent through other SPOR element areas due to the still early efforts of the SUPPORT Units and Networks.
• Key stakeholders, including patients, are supported to engage in research and decision-making processes. Stakeholders have been engaged across sectors and across participant communities. SPOR is noted to have done this extremely well, with extensive efforts being made at the national and provincial/territorial levels. There are a number of ties between and among researchers, clinicians, policy makers and patients. The SUPPORT Units have been successful in engaging numerous stakeholders, from among all stakeholder groups, in all of their core functional areas with an increase in personnel actively engaged from 2013-2014 through 2014-2015. The ACCESS Open Minds network has been able to achieve a high percentage of both youth and family engagement, along with other key stakeholder groups.
  • However, there is room to consider opportunities to continue the development of key stakeholder relationships with other networks and institutions involved in POR (e.g., health charities and patient organizations)to further leverage resources and expertise; and,
  • PE is seen to be a work in progress with enhancements needed to support effective engagement through training and mentoring, as well as determining what are best practices for engaging patients, including how, why, when, and for what needs.
• Methods in patient-oriented research are advanced. There is little evidence that SPOR elements have advanced POR methods to date; however, activities underway are anticipated to bring progress. The SUPPORT Units have demonstration projects up and running and report a number of other research outputs being produced. Additionally, through SPOR, POR activities have been noted to have intensified with more work across sectors, more work based on partnerships, the integration/sharing of research opportunities, and quite importantly, the deliberate engagement of end-users.
• SPOR elements provide capacity development opportunities that align with the SPOR CD framework. CD is lagging in implementation. The need to strengthen a national approach to this element was identified, where currently the SUPPORT Units, SPOR Networks and SUPPORT Unit working groups were all seen to be developing fragmented approaches. Capacity development in PE was noted to be a gap with both patients and researchers. However, most SUPPORT Units identified leveraging the recently released CD Framework issued by SPOR and have undertaken some training activity, albeit patient partnership activities are the least in numbers of completed support or service functions.

Although SPOR is believed to be too new by most respondents to have significantly changed the research environment to date through concrete, measurable outcomes, SPOR is making some impact in influencing a change in the POR landscape and changing the research culture. This has been achieved through the alignment with provincial/territorial priorities, increased involvement of all stakeholders in health research, promoting the importance of PE and the establishment of an environment where the focus is on outcomes of importance to patients.

• However, culture change is a long term endeavor and there are a number of areas that require ongoing effort. Continuing to address resistance within the research community by demonstrating value of POR and PE, through communicating evidence of success and best practice, will remain important. SPOR is not alone in this area as it is a challenge internationally and noted by international key informants that there is a real need for a stronger and increasingly visible generation of evidence supporting the impacts of patient-oriented research. It is essential to provide evidence to researchers, policy makers, and other potential stakeholders that there is tangible and intangible value to implementing a patient-oriented research approach.

SPOR is now committing funding at full scale to its core components and has leveraged partner dollars, meeting is own requirement for 1:1 matching and in most cases exceeding this ratio. SPOR is refocusing its efforts to increase support to the core elements as planned, evidenced by a significant decrease in funding directed towards Foundational Investments.

3.3.2 Analysis

3.3.2.1 Progress towards Immediate Outcomes (Evaluation Question #5)

SPOR’s six immediate outcomes have been defined in the SPOR Performance Measurement Strategy (PMS) (October 2014) and identifies SPOR will undertake programming that results in:

• Key stakeholders, including patients, are supported to engage in research & decision-making processes.
• Multi-disciplinary, integrated and accessible research platforms are established.
• Pan-Canadian networks are mobilized to address health challenges.
• SPOR elements provide capacity development opportunities that align with the SPOR CD framework.
• CCTCC is established.
• Methods in patient-oriented research are advanced.

All key informant and case study interviewees predicated their responses on rating the extent of SPOR’s progress on expected immediate outcomes to the early stage of implementation of all core elements. As identified in Exhibit 9, all of the immediate outcomes have been rated, on average, to being achieved “to a moderate extent,” with the implementation of the SUPPORT Units receiving the highest number of interviewees (31%) perceiving this outcome to be achieved “to a great extent.”

Exhibit 9 – Extent of progress towards immediate outcomes

*Rounding has been forced to add to 100%.

*Average rating has been calculated as a standard weighted average based on the number of respondents.

All key informant and SUPPORT Unit case study interviewees noted the varying stages of implementation of all the SUPPORT Units. Some SUPPORT Units are ahead in implementation while others are still working on establishing administrative and governance structures. This is similar to the perceptions around the implementation of Networks. It was acknowledged that there is one active network (Access Open Minds), another one ramping up (Primary Care) and the next five (Chronic Disease) not yet in place.

Key informant and case study interviewees identified moderate progress in engaging stakeholders when responding to the closed-ended rating question. This moderate rating does not mirror the qualitative elaborations provided. All interview data, supporting performance data submitted by the SUPPORT Units, as well as the governance and administrative network analysis support a very great extent of stakeholder engagement taking place. The SUPPORT Unit case study results have identified that stakeholders have been heavily supported to engage in research and decision-making processes and there are a number of ties between and among researchers, clinicians, policy makers and patients.

However, results from the key informant interviewees and the SUPPORT Unit case study also identified further (and continuing) opportunities for more and better communications to engage stakeholders. There were perceptions that SUPPORT Units are not well known by the research community, and that researchers do not understand what platforms are being established, what services are offered and how researchers can leverage what is being done. There also appears to be more effort required to achieve effective PE.

Advancement in methods and capacity development opportunities were the two outcomes rated the lowest by key informant and case study interviewees. For advancement in methods, again it is the early stage of SPOR that prohibited key informants and case study interviewees in identifying significant advancements in POR being made, however, some examples were provided. Capacity development was the area with the greatest number of respondents (10%) identified that no progress had been made in this area.

Further elaboration on the findings against each of the immediate outcomes follows.

Stakeholders, including patients, are engaged

Virtually all key informant and case study interviewees noted very extensive efforts in this sphere being made both by SPOR at the national level and by SUPPORT Units at the provincial/territorial level. Thus SPOR’s stakeholder engagement activities are seen as strong not only because they ensure the needs of the research, clinical, and policy communities are being met, but that these sometimes siloed communities have increased opportunities to interact with and learn from each other.

The view from international key informants also support this point. Interviewees in the UK viewed SPOR as having been relatively more successful in garnering diverse public engagement than NIHR, and felt that there could be lessons learned for NIHR from SPOR.

The SUPPORT Unit case study identified participation in cross-sectoral discussions on research and government policy that were nearly unthinkable before, including with stakeholders from other jurisdictions and through the Networks case study, ACCESS Open Minds and PIHCI both have engaged stakeholders extensively in their design.

“Never before in my long career have I been around the table with provincial counterparts from other regions. NEVER. This is transformative.”

The total number of personnel reported as actively engaged within the SUPPORT Unit core functions was 2,127 personnel in 2014-2015, an increase from the 409 reported in 2013-2014^{Footnote 44}. At this point the four initial SUPPORT Units launched in 2013 have progressed through their initial set-up and two additional SUPPORT Units are active. All SUPPORT Units active for longer than one reporting period experienced an increase in the number of personnel actively engaged within the SUPPORT Unit core functions over the periods, with the exception one SUPPORT Unit’s Training and CD core function, which reported a decrease from 382 to 62 personnel.

The three most engaged categories of stakeholders, as reported by the SUPPORT Units, are researchers or academics, health system/care practitioners or professionals and patient representatives as shown in Exhibit 10.

Exhibit 10 – Numbers and types of stakeholders engaged with SUPPORT Units 2013 - 2015 (FTEs)

The SUPPORT Unit stakeholders are engaged across all core SUPPORT Unit functions. The main areas of engagement include consultation and research services, health systems implementation research and knowledge translation core functions as depicted in Exhibit 11. Less engagement is occurring in the clinical trials area, methods support and development, and data platform areas.

Exhibit 11 – Numbers of stakeholders engaged with SUPPORT Units by core function 2013 - 2015 (FTEs)

The ACCESS Open Minds network also demonstrates stakeholder engagement across multiple categories. Of particular interest is the reported relatively high percentages of youths and families involved at both the governance and site team level.

The governance and administrative network analysis also shows engagement across and between stakeholder groups. The network graph shown in Exhibit 12 depicts all the SUPPORT Units and the PIHCI research network as having a mix of ties to research, clinical and policy areas. The Ontario SUPPORT Unit and the SPOR National Steering Committee are two structures that include ties to all categorized roles.

Exhibit 12 – Numbers of stakeholders engaged with SUPPORT Units by core function 2013 - 2015 (FTEs)

Legend: Dark blue = Research; Light blue = Clinical; Dark green = Policy; Light green = Industry; Red = Charity; Orange = Patient

In addition to the available public information on the engagement of patients in the SPOR governance structures, the SPOR program provided further detailed information related to the engagement of patients in the SUPPORT Units. It is a requirement of SPOR that the SUPPORT Units have patients engaged in their governing bodies. The total number of patient seats allocated by SUPPORT Units to patient representatives is 74. This ranges from two seats up through 17 seats depending on the SUPPORT Unit. These patient positions are part of different governance structures, such as a steering committee or advisory council supporting the SUPPORT Unit overall or a specific component area (e.g., capacity development). Through data provided by the program, it is reported that 59 (~80%) of the available 74 seats are filled with the rest in recruitment. Note that one patient can fill more than one seat so these data are not reflective of the number of participants.

However, it was identified that the development of all relevant partnerships and connections would be an ongoing process. Key informant interviewees identified that some key institutions/organizations were not connected to SPOR and that the connection to the research community was not clear. Opportunities were noted to be available to better engage with other key stakeholders (e.g., health charities) already working for better care for stakeholder communities through POR. The PE case study interviewees echoed this finding in identifying a need to increase and enhance collaborations with various external networks and institutions. They responded that various external networks, such as the Patient Voices Network and Patients Canada, could be further leveraged by the PE element of SPOR for the provision of resources and expertise. SPOR was encouraged to continue to determine what resources are already available and which stakeholders can be leveraged going forward.

Equally common was the observation that in some regions there is a long way to go to improve the literacy of policy makers and the understanding of researchers as to how to participate fully in the research decision-making process, as well as, quite importantly, supporting the effective engagement of patients.

Part of the challenge is due to definitional issues. What is stakeholder engagement vs PE? Results from the SUPPORT Units and PE case studies indicate the definitions are not clear. PE can mean any or all of: engagement of patients in SPOR governance, engagement of patients in SUPPORT Units governance and decision-making, engagement of patients in individual initiatives (e.g., research projects, SPOR Networks), and engagement of patients less formally through asking them about research priorities (even if they have no other participation in the research) or satisfaction with clinical processes and outcomes. Each of these different aspects likely requires different engagement mechanisms, has best practices, and has different pros and cons.

The CD, CT and PE case studies identified that enhancements would be required to support the effective engagement of patients within the research and decision-making process highlighting the need to provide training and potentially mentoring to patients to enable them to fully contribute. The PE case study particularly identified the need for enhanced support of patients to further enable them in their roles, identifying potential gaps in supporting the representation of patients in the governance structures. The CT case study also identified a need to further involve patients in governance and increase communication and engagement of patient representatives. The CD case identified that there was more discussion needed on how PE should be conducted as opposed to application and uptake by the research community, thus providing room for improvement as SPOR matures.

Extensive “good will” effort has been reported through the SUPPORT Units and Networks case studies in conducting PE, although with highly variable intensity and success. The issuance of the PE Framework nationally by SPOR in 2014 has been used moderately by all SUPPORT Units, mainly by those individuals responsible for PE. It is either as a model for SUPPORT Units to develop their own PE plans, or to (less likely) to refine PE plans already in place. The ACCESS Open Minds Network is aware of the Framework but have not leveraged it to any great extent, mainly because of unique issues involved with youth.

Interviewees in the Networks case study identified that patient engagement efforts to date were incomplete, not sufficiently planned, not fully representative of patients and unclear as to the extent the patient voice is valued and acted upon. At the same time, difficulties in grasping the best ways to involve and educate patient representatives along with struggles in patient capabilities for participation were also raised.

While many respondents across the key informants and case studies believe that PE needs to be strengthened, it was noted that in order to support PE, considerable work needs to be done as to how to do PE, when to do PE, why do PE, and what capacity building needs to be completed – on both the patient and the researchers/clinician sides. Within the SUPPORT Units case, one PE lead, for example, reported that researchers have yet to come to them with questions or identify requirements for help on recruiting patients as partners in research – although this was a SUPPORT Unit operational for only a year and a half.

The Networks case study identifies significant challenges being encountered in developing appropriate, useful, best practices for involving patients and families in the design, review, and conduct of research programs – these are evolving constantly, and while being based on similar underlying principles, will almost certainly require customized approaches for each individual research situation; e.g., taking into account the experience and resources of both the patient and researcher/clinician communities.

Multi-disciplinary, integrated research platforms established

At the time of evaluation data collection, six SUPPORT Units were established and are all at different stages of implementation. During the course of the evaluation, however, an additional three SUPPORT Units were funded at the end of the 2015-16 fiscal year and out of scope for the evaluation. The first year of operation for most SUPPORT Units consists of planning and development. The second year typically consists of the creation of standard policies, procedures and processes for each core function and the beginning of service delivery. Two SUPPORT Units, Ontario and Manitoba, are the most fully developed.

Exhibit 13 depicts all the SUPPORT Units; those approved, and those under discussion. Once completed, there will be SUPPORT Units across all provinces and territories in Canada.

Exhibit 13 – SPOR SUPPORT Units

Through implementation, the SUPPORT Units have been able to establish a number of partnerships that align with the provincial/territorial focus. The majority of the partnerships established are with public funders from government (e.g., the various provincial Ministries of Health), though various SUPPORT Units have also established partnerships with academic institutions, and a couple have maintained private partnerships (IBM and the Cochrane Centre).

The total number of multi-disciplinary and cross-sectoral partnerships being reported across SUPPORT Units in 2014-2015 is 37 (Exhibit 14), up from 17 reported by those SUPPORT Units active during the 2013-2014 reporting period.

Exhibit 14 – Number of multi-disciplinary and cross-sectoral partnerships established, reporting period 2014-2015

Additionally, there is increased activity in SUPPORT Units core functional areas including the provision of supports and services and work around data platforms.

Data provided through the individual Annual Performance Reports for SUPPORT Units was reviewed for the 2013-2014 and 2014-2015 reporting periods. Key results are summarized as follows:

• The total number of data sets reported is 534, an increase from the 253 reported by those SUPPORT Units active for a period of two years or longer.
• The total number of supports and services was 1,872 in 2014-2015, an increase from the 376 reported in 2013-2014. All SUPPORT Units active for longer than one reporting period experienced an increase in supports and services provided by core functions over the periods, with the exception of one SUPPORT Unit’s Consultation and Research Services Core Function, which experienced a decrease in services reported from 40 to zero between the periods. There was one other SUPPORT Unit reporting its Health Systems Research, Implementation Research and Knowledge Translation Core Function experienced a decrease of 19 supports and services.

On the data platform side, the SUPPORT Units case study identified that many of the provinces are putting high priority on building data platforms, facilitating data access and data linkages, within which researchers and policy makers will be able to access and use a wide variety of relevant provincial data (e.g., potentially including clinical data, healthcare administrative information, prescription data and genomics findings.). Additionally, some SUPPORT Units already include non-health data (e.g., environmental variables, justice statistics) while some intend to do so. Many SUPPORT Units and non-SUPPORT Unit initiatives are being planned to use these expanded “Big Data” capabilities (see example impact from Newfoundland & Labrador in section 3.3.2.6).

“We’re trying to “liberate” all data housed in [our region’s] research and clinical labs.”

These initiatives will also be able to link some databases owned by individual researchers, thus providing users with easy access (e.g., cloud-based), interoperability across different data sets, and high levels of security and privacy. Further, the SUPPORT Units provide expert advice as to the nature of these data (because there is so much, it is difficult for researchers to even know what is available), how to access them, and help conduct and interpret the analysis. The Newfoundland and Labrador SUPPORT Unit has the IBM Canada Research & Development Centre as a major funding partner and provider of e-solutions and “deep analytics”. IBM is also involved in the Newfoundland Translational and Personalized Medicine Initiative (TPMI).

The SUPPORT Units case study identifies SPOR to have been a critical factor in at least the smaller provinces in driving these initiatives and allowing them to hire key personnel and make cross-appointments with universities. (Very large provinces usually had significant initiatives pre-SPOR, so the additional effect of SPOR is far less clear). Many (possibly all) of these data platform initiatives are tied to existing institutes that house healthcare data, such as the Newfoundland and Labrador Centre for Health Management and Informatics, or Ontario’s Institute for Clinical Evaluative Sciences. Even where there was an existing initiative, however, in smaller provinces the SPOR funding accelerated the development process: e.g., the Data Science Centre in the George & Fay Yee Centre for Healthcare Innovation in Manitoba was reportedly only a “one person shop” before SPOR funding.

Consultation and research services provided by the SUPPORT Units centered around data analyses and/or data management activities, knowledge translation support, study design (including methods) assistance, and support for research proposal/grant application submissions. Least accessed services included scientific writing experience, literature review and/or metadata review, as well as, perhaps surprisingly, PE support.

All key informant and case study interviewees acknowledged each SUPPORT Unit was at varying levels of maturity and progressing to outcomes at varying rates. Progress was identified at the provincial level, as each SUPPORT Unit is operationalizing its business plan.

Pan-Canadian networks are mobilized

ACCESS Open Minds and PIHCI are established networks, although both have developed more slowly than originally expected and are not fully operational.

The ACCESS Open Minds Network was officially announced in June of 2014 so at the time of the evaluation, has been in development for one and a half years. As identified through the Networks case study, ACCESS Open Minds topics were seen to be in an area of significant need for some type of large, multi-faceted, initiative, to demonstrate how youth mental health care could be transformed. ACCESS Open Minds is intended to gather evidence on how to do so, analyze it, and apply it to communities. Once fully operational, expectations are high among case study respondents that needs in this area will be met and that ACCESS Open Minds will be capable of transforming services for youth with mental illnesses at sites across Canada. Respondents believe the benefits for youth will be earlier and more comprehensive access to services, based on real-world experimental evidence. The principal investigator believes ACCESS Open Minds has been able to put youth mental health services and needs “on the map”.

While ACCESS Open Minds intends to address services for young people in underserviced areas across Canada, the first site will only be fully operational in December 2015 (a First Nation community in Nova Scotia), and, although there is a deliberate multi-site design to this Network, there is significant site-by-site variation in how the model will be implemented.

ACCESS Open Minds has reported in its 2014-15 annual report that central office space has been secured at the Douglas Hospital Research Centre in Montreal, Quebec and main governance structures began to be put in place during the reporting year. This included the establishment of a Youth Council, a Family/Careers Council, Executive Committee and Governing Council. It is also reported that key staff have been hired for the central office including a Scientific & Clinical Director, Senior Project Manager, Training and KT, Research & Evaluation and Inter-site and Governance coordinators, two administrative assistances and one research assistant.

The PIHCI Network is still developing with the launch of its “quick strike research projects” in late 2015. PIHCI survey respondents noted that the function of PIHCI Network still appears to be in its incipient stages, and to date no space or structure has been created for collaboration and discussion between provinces and territories, although some structures are being established informally and more organically (e.g. monthly calls between coordinators).

Both Networks, while somewhat operational, are too new to have assessed outcomes at the time of the evaluation and as noted in Section 3.2 are still undergoing design and delivery adjustments.

Capacity development opportunities are provided

CD was seen to be the least developed SPOR element. Key informant and case study interviewees identified the lack of priority being given to this element as a hindrance, in part due to the only recently released SPOR CD Framework (August 2015). However, the release of the Framework was identified as a clear success in the development of a pan-Canadian approach, vision and framework regarding CD (and also PE). The CD and PE case studies identified the effectiveness of SPOR bringing together CD personnel within provinces, and beginning to open the conversations across provinces.

Findings from the CT and CD case studies indicate progress in this element to be tied to the activities underway at the local level though the SUPPORT Units. Knowledge and use of the SPOR CD Framework varied substantially across the SUPPORT Units and the ACCESS Open Minds Network, with, of course, all component leads responsible for CD being very familiar with it. The ACCESS Open Minds Network has not used the Framework to date, but intends to support capacity development through an extensive training program now in preparation, though very targeted in nature. The intent is for development of certification level courses; e.g., trainees will be certified as an ACCESS Open Minds youth mental health clinician, trained to do mental health assessments.

The PE Case study identified capacity development as an issue within the PE element of SPOR. Specifically a gap in the development and provision of training in PE for both patients and researchers to engage with each other (different than the governance training mentioned above). The PE case study noted that interviewees were unaware of a deliberate link being made between the PE Framework and the CD Framework.

Through the international comparative review, training was noted as an area of stress for NIHR so SPOR is not alone here. NIHR was described as consisting of various entities, each with their own training for researchers, as well as the public/patients. INVOLVE was viewed as having taken the role of ensuring public/patient stakeholders are made aware of the training available, though it was noted that there is no one entity that reviews the public/patient training under NIHR for gaps or duplication and no clear development path for public/patient stakeholders who wish to be involved. This was compared to CIHR SPOR, which was viewed as having a strong approach at the provincial level, but potentially requiring a strengthened national level approach.

However, there are examples of some activity (e.g., studentships, training programs within the SUPPORT Units related to PE and/or POR). SUPPORT Units reported a total of 422 capacity development supports and services across two annual reporting periods (2013-14 and 2014-15). The largest volume of activities during both reporting periods were conducted in the research methods key competencies area, as illustrated in Exhibit 15. Knowledge translation activities experienced the highest level of growth between the periods, with the least degree of growth seen in patient partnership activities.

Exhibit 15 – Number and type of supports and services – Training and CD – total number of activities 2013-2014, 2014-2015

Note: Only one SUPPORT Unit was reporting on these areas in 2014.

CD plans being developed by the SUPPORT Units include elements such as PE and POR studentships, presentations by patients as to how they became involved in research and how they like/dislike being approached, new methods for POR, use of Big Data, and knowledge translation of science and how to apply it.

However, key informant interviewees, the SUPPORT Units and CD case study respondents commented that there was a lack of direction from CIHR as to where they want to go nationally with respect to CD, especially with respect to concrete guidelines. At the moment, these respondents believed this initiative was more loose-knit than one would ideally want, and that the national Framework’s wording was too broad.

Additionally, there is confusion as to capacity development vs. career development. Both are referred to as “CD,” increasing the confusion – different SUPPORT Units appear to be interpreting this component in different ways. Capacity development is mainly interpreted as increasing capacity of patients to participate in research, and of researchers and clinicians to effectively use patient input – with the SUPPORT Units case study identifying that knowledge of best practices here is still poor, as are metrics of success. However, capacity development can also mean the understanding of, and competence of, researchers to conduct POR – assuming they know exactly what “POR” is. Career development appears to mainly involve training students in POR methods, although this is unclear.

Through the SUPPORT Units case study it was noted that building capacity represent a range from dealing with beginning students up to leadership training, and CD in some regions explicitly addresses how to build leaders in POR. For example, one SUPPORT Unit is building a “ladder” in POR techniques, with certificates and diploma programs up to the Masters level, following which leadership training will be provided. It was further noted that individual POR elements such as Knowledge Translation or Clinical Trials would be included within the CD modules.

The main needs to be addressed identified in the CD case study were:

• Enhancement of pan-Canadian coordination and delivery of capacity building in patient-oriented research. It was identified that though CD activities are being developed within each SUPPORT Unit, SPOR should reflect on the potential to strengthen a national approach to CD and the creation of Pan-Canadian training opportunities.
• Enhancement of funding in support of CD activities. There was an identified need for enhanced support in terms of funding towards CD activities, particularly with regards to trainees as well as Network activities. An example provided was the PIHCI Network, which was perceived to be under-funded with regards to CD activities.

CCTCC is established

The CCTCC has been operational for over a year, although still in the preliminary stages of implementation. It is perceived by CT case study interviewees to have made progress as anticipated on the nine recommendations defined in the Canadian Clinical Trials Summit Action Plan. Achievements against the nine recommendation areas are as follows:

It was noted that there may be a need for the CCTCC to further prioritize the recommendations it is tasked to address.

The CT case study documented the difficulty that case study interviewees had in ascertaining the incremental impact of SPOR on the progress of the CCTCC, since in a number of instances it was felt progress in certain areas was completed prior to the establishment of the CCTCC. Thus the specific importance of the CCTCC role going forward in SPOR is less clear.

The Clinical Trials element has also been incorporated within SPOR as a core function of each provincial/territorial SUPPORT Unit as well as through the clinical trials activities planned to be conducted across the SPOR Networks, however very little activity (almost none) has taken place in this area due to the early stages of development.

Methods in POR are advanced

In all cases it was viewed as being too early to identify specific methods being developed that are attributable to SPOR. However, there are activities taking place that will contribute to the eventual advancement of methods and the POR environment.

A few interesting examples were provided through the SUPPORT Units case study that are clearly examples of POR, including:

Ontario

• A cardiovascular (CVS) research project initially focused on the clinician’s primary interest: hospital readmissions and survival related to heart failure. However, the project’s patient panel chose a second outcome of interest to the patients themselves: they asked researchers to also study Emergency Department (ED) visits, caring more about these than readmission to hospital since ED visits can turn the lives of patients and families upside down, and cannot be planned for. This was a complete surprise to the scientists and clinicians focused on hospital readmissions.

Newfoundland & Labrador

• The Big Data work in Newfoundland was able to address several critical healthcare issues simply through data analytics: (1) analysis of prescription data found that three physicians alone accounted for 80% of blood tests requested; once informed, these physicians changed their prescribing habits; (2) a real time multifactor data collection and analysis system could accurately and autonomously predict sepsis in neonatal infants 72 hours ahead of a crisis developing, far faster and better than physicians and nurses could do; (3) because Newfoundland’s population has the cleanest genomics population in Canada, it was possible to predict the chance of serious heart disease from each individual’s genomic signature up to 20 years before the CVS problem became serious; and (4) Newfoundland is now conducting a clinical trial using IBM software for remote monitoring, which goes home with patients who have been readmitted many times to hospital. The intent is to see if this monitoring can detect if patients are getting in trouble, so health services can then intervene.

Manitoba

• The Massive Transfusion Protocol (MTP) Project brought many clinical service providers together to identify best practices to save patients experiencing massive bleeding. These patients need massive blood transfusions, very quickly, within a highly chaotic environment. To solve the problem requires rapid, perfectly integrated responses from a wide range of professionals, from the physicians who first identify the bleeding, to those in blood banks, to those who deliver the blood, to those who test if it is the correct blood type, to those who transfuse it. At least 30 groups of people must respond in these situations - the project developed a system to simultaneously activate and coordinate all these groups, automatically triggered by one telephone call. Mortality in these situations used to be about 50%, and is now much lower (exact figure not yet available).

In the Ontario example, patient feedback and experience directly informed a change in a selected outcome measure for a major clinical study moving the study focus from hospital readmissions, which mattered to the clinicians, to measuring ER visits, which mattered to patients. The home-based patient monitoring technology in Newfoundland and Labrador is offering early detection of issues so health services can intervene with patients sooner providing better patient outcomes; and for Manitoba, getting the right treatment (Massive Transfusion Protocol) to patients faster and more efficiently resulted in decreases in mortality.

There is also research output being produced through the SUPPORT Units. The total number of research output across SUPPORT Units was 203 publications or papers during the 2014-2015 reporting period. As demonstrated in Exhibit 16, the largest volume of research outputs during the period were from scientific publication.

Exhibit 16 – Research output of SPOR Network supported researchers, reporting period 2014-2015

Further, the case studies do document a change in POR activities by those involved in SPOR.

“For myself, I’ve always worked closely with First Nations communities. But our thinking has expanded to involve decision-makers . . . the sit-down with the policy makers is now easier because SPOR forces us all to think more carefully about what you want at the end of the day.”

The SUPPORT Units and Networks case studies identified that many stakeholders involved with SPOR had at least some leanings towards POR before SPOR’s roll-out, although the extent of this varied enormously. With SPOR present, these activities have become even more intense, creating a pressure to do more of this type of work, partnership-based, and across sectors (e.g., with the public health and government departments of health services and health policy.) The SUPPORT Units and CD case identified more discussion on how PE should be conducted, knowing previously that PE is important, but now see it needs a value proposition in order to achieve sustainability.

The SUPPORT Units case study also found a significant shift towards accounting for the priorities and challenges of stakeholders in other sectors and working with research groups they had not worked with before – and in some cases did not know even existed. This includes the integration of research topic areas and the creation of shared research opportunities where previously these tended to be investigated in silos.

The Network case, in particular the results of the PIHCI survey, also identified that SPOR has demanded researchers to think more deliberately about engaging the end-users of the primary health care services in a meaningful way. The national scope of the Network and the requirement for partnership across jurisdictions, has enabled work on projects that involve more than activities in a single province.

For ACCESS Open Minds, patient input was noted to make the research more practicable and able to be effectively applied.

3.3.2.2 Movement toward longer term impacts (Evaluation Question #6)

One of SPOR’s key achievements is making all relevant stakeholders respond to “the pressure of ideas,” expanding the recognition of the value of multi-sector, multi-stakeholder, focus on results to patients; really considering SPOR principals and goals. It is now reported to be very difficult to plan major clinical research initiatives without at least taking POR-related concepts into account; e.g., PE, stakeholder engagement, evidence-based clinical policies and procedures.

Although there are identified gaps and it is re-iterated that SPOR is still very early in its lifecycle, the average rating provided by key informant and case study interviewees indicated, on average, that SPOR has contributed to a change in Canada’s health research landscape to a moderate extent. Many (57%) rated SPOR’s contribution to be at least to a moderate extent or greater and equally many (59%) rated SPOR’s contribution to be moderate or less as shown in Exhibit 17. The SUPPORT Unit case study respondents were those mainly identifying this change taking place to a great extent.

Exhibit 17 – Extent that Canada’s health research landscape changed because of SPOR

*Rounding has been forced to add to 100%.

*Average rating has been calculated as a standard weighted average based on the number of respondents.

The most frequently mentioned area of change, attributed to SPOR, is the increased involvement of all stakeholders in health research. Broadening the scope of stakeholders involved to span the researcher, clinician, policy maker and patient communities was noted to be one of the key change drivers to POR research. This was identified by key informant interviewees as well as through the Networks, CT, CD and SUPPORT Units case studies.

The other area identified was the focus on the relevance of PE. Through the key informant interviews and the CD and PE case studies, SPOR was seen to have fostered and perpetuated a conversation on the relevance of PE in research and the release of the PE Framework was viewed to set the tone and understanding for what the research community could expect regarding patient-oriented research going forward.

It is still very early days, however, and SPOR is seen by virtually all key informant interviewees to have shifted expectations, and built themes and goals that are – if not perfectly coordinated and clear – at least much more widely understood than was previously the case. Because of the SUPPORT Units priority-setting exercises, projects supported by the SUPPORT Units are focused on provincial/territorial priorities, and because of SPOR they focus more strongly on outcomes of importance to patients. This is a distinct change from the traditional CIHR biomedical model.

Similar results are seen when looking at the perceptions of SPOR’s contribution to a culture change towards increased buy-in to POR. All key informant and case study interviewees identified that SPOR has had an influence on the expected culture change required for the conduct of patient-oriented research to a moderate extent on average, with 66% identifying SPOR’s contribution to be moderate or lower and 68% rating SPOR’s contribution to be moderate or higher. Exhibit 18 identifies the proportion of respondents by rating selection.

Exhibit 18 – Extent of SPOR’s influence on culture change to enable POR

*Rounding has been forced to add to 100%.

*Average rating has been calculated as a standard weighted average based on the number of respondents.

However, the qualitative elaborations supporting the ratings indicate a lower level of influence being perceived by key informant and case study interviewees. Culture change was consistently noted to be a long term endeavor and SPOR was noted to be too young for such change to have taken place at this point in time. There were a number of consistent themes that emerged through the key informant interviews and case studies where SPOR would need to continue efforts in order to influence a change in culture, and it was noted that should SPOR “stay the course,” these culture shifts would indeed occur.

Common areas of focus include:

Resistance within the research community

There has been some slow understanding and uptake by some areas/members of the research community who are not convinced of the value of POR or PE, mainly related to the lack of hard evidence as to their value on the quality of the research or the importance of the resulting clinical outcomes. Since POR and PE require significant additional investments of time and energy on the part of researchers, many respondents believe these sceptics will remain unconvinced until hard evidence is available.

Additionally, on the research side, there are disincentives to undertake POR. Institutional tenure and promotion systems may not recognize or reward activities that are primarily directed at practical healthcare challenges, as opposed to fundamental biomedical discovery research and resulting peer review publication. This can be especially discouraging and traditional university systems need to be brought on board to build capacity and leadership in POR areas.

Many interviewees noted that in order to further a culture of PE, it would be beneficial to provide examples of enhanced success due to PE. CIHR needs to keep showing the research community concrete exemplars that POR is better, and why – demonstrating to researchers that POR isn’t a waste of their time. There is a wide-spread belief that these examples are still lacking – there are said to be some pockets of evidence, but more are needed. The SUPPORT Units demonstration projects are helpful, but more of them are needed, perhaps foremost from the SPOR Networks. This also applies to efforts to get patients on board as well. Resistance on the part patients that were unfamiliar with patient-oriented research, both in terms of conduct and potential benefits.

The area of POR understanding and uptake was also identified as a key need internationally to further support the trend of patient-oriented research models. International key informants noted the need for a stronger and increasingly visible generation of evidence supporting the impacts of implementing a patient-oriented research model. It is essential to provide support to researchers, policy makers, and any other potential stakeholders that there is tangible and intangible value to implementing a patient-oriented research model. Further, it was viewed to be essential to provide evidence supporting the optimal level of PE, as it was identified that the optimal level of engagement may vary depending on the type of research being conducted.

Aligned to this were comments on enhancing accountability measures of researchers employing PE. The PE case noted that though the requirement for PE within grant applications is an excellent first step, there is a continued need to ensure accountability of researchers in employing PE. Interviewees noted a concern that researchers are incentivized to employ PE only to the degree that it will fulfill grant application requirements, and there is no accountability to ensure the quality of PE being realized. Suggestions were made to either enhance the performance indicators on the part of researchers, or to allow for patients to report on the level within which they were engaged.

All of these points were raised through key informants, international key informants, and SUPPORT Units, CD and PE cases.

Communicating a clear definition of PE

Despite the definition of PE within the SPOR PE Framework, the SUPPORT Unit and PE case studies noted that those involved in SPOR still operate under different definitions. Interviewees identified that defining the appropriate level of PE is still underway, as each area of research may be optimized through varying levels of PE. However, interviewees also noted that in order to gain further acceptance within the research community, as well as by additional stakeholders such as clinicians and policy makers, PE would need to be better operationalized to provide a coherent understanding of how patients can be engaged in a meaningful way for the enhancement of research outcomes.

Communicating a clear definition of SPOR

SPOR is a very large and complex initiative, resulting in many respondents being uncertain as to how all the “moving parts” fit together, not to mention the difficulties encountered in explaining an initiative of such broad scope in plain language. There is a need to be able to present activities in a way that makes them engaging and meaningful to an audience beyond the academic research community and being able to further penetrate down to the “on-the-ground” research, clinical and policy areas in the jurisdictions. These points came through the key informant interviews and the SUPPORT Units case study.

3.3.3 Cost-efficiency Analysis (Evaluation Question #7)^{Footnote 46}

From 2010-11 to 2015-16, $202.83M was allocated specifically to SPOR by Treasury Board. Resources were targeted for SPOR SUPPORT Units, Networks, a Clinical Research Coordinating Centre, and other relevant research investments. Over this same time period, CIHR spent a total of $356.86M on SPOR grants and awards (Exhibit 21), thus increasing the overall envelope of funding allocated to SPOR and SPOR-related research.

The financial data reflect that all SPOR core elements now have committed funding at scale. When considering the fidelity assessment, the financial data converge with data from elsewhere in the evaluation and confirm the delays in the implementation of the SPOR SUPPORT Units and Networks.^{Footnote 47} Official program documents indicate there was a planned delay through the first two years of SPOR with activities identified for 2011-12 to include the establishment and engagement of SPOR’s National Steering Committee, the determination of priorities, the creation of the funding opportunities for the Networks and SUPPORT Units, as well as some specified work in the Clinical Trials area.

The unplanned delays were attributed by the program to the considerable amount of time needed to engage partners and to design and deliver the elements in a way that would achieve the greatest outcomes as follows:

• provincial/territorial partnership development , including a two-year engagement and planning process with subsequent time required to mobilize partner resources;
• funding opportunity design and delivery;
• establishing SPOR operational and governance structures;
• broad stakeholder engagement and communication about SPOR;
• funding early capacity development activities needed to prepare for larger investments in Networks and SUPPORT Units

SPOR Grants and Awards expenditures are allocated to SPOR core elements as well as Foundational investments. Official program documents identified existing CIHR investment in POR areas prior to the release of SPOR that includes, as examples, funding randomized control trials, various research networks and teams as well as methods centers for clinical research.

There were a number of relevant and important programs already underway that helped to set the stage/prime the Canadian clinical and patient-oriented research environment for SPOR’s launch and implementation. These Foundational investments were and continue to be important as most play a role in the development of SPOR-relevant research capacity and are directly aligned with SPOR. For example:

• One of the stated vision-achieving goals of SPOR is to grow Canada's capacity to attract, train and mentor health care professionals and health researchers, as well as to create sustainable career paths in patient-oriented research. Through the Foundational Investments, CIHR continued to support:
  • MD/PhD students through the MD/PhD Program Grants;
  • Students pursuing health professional degrees (often at the undergraduate level) to engage in research (usually during the summer) through the Health Professional Student Research Awards program;
  • Trainees at the doctoral and post-doctoral level as well as new investigators through priority announcements on the relevant CIHR open award programs in Clinical Research; and
  • Mentoring of clinical trial trainees through the Randomized Controlled Trials Mentoring program.
• The CIHR Clinical Research Initiative spearheaded important dialogue with the provinces and territories about building a pan-Canadian clinical research infrastructure in partnership with the Canada Foundation for Innovation (CFI). The concept of challenging institutions and their researchers to think beyond traditional methods of doing research and engaging with their jurisdictions began with the CRI and continued with SPOR. Although CRI was more focused on the biomedical and clinical pillars of research, SPOR provides a natural continuation with a focus on the clinical and health services pillars of research. CRI also brought forward the concept of regionally and nationally-linked clinical research centres, national multidisciplinary clinical research networks and clinical research technology platforms; again, SPOR enabled a continuation and broadening of the scope to include health services and health system research, yet narrowing the focus to patient-oriented research.
• The Cochrane Collaboration is mentioned a number of times in the SPOR strategy document, specifically as an important foundation to the Canadian expertise in systematic reviews; the Strategy further speaks to SPOR playing a role in providing funding to the Cochrane Collaboration, which it did through funds that are considered as Foundational Investments.

CIHR has planned to gradually refocus this Foundational spending as the core elements of SPOR are implemented.

The investment towards the Foundational elements has been decreasing over the evaluation timeframe. On average Foundational investments have decreased 15% annually, or by 56% in total from 2010-11 through 2015-16. Official program documents note this is a planned refocusing of the spending (Exhibit 20).

Exhibit 20 – Extent of SPOR’s influence on culture change to enable POR

In turn, it is evident there has been an increase in spending on SPOR’s core elements over the last three fiscal years, reflecting the launch of SUs in 6 jurisdictions and funding of 7 Networks across three priority areas (Mental Health; Primary and Int0grated Health Care Innovations; and Chronic Diseases).

As the SPOR core elements have committed funds at planned scale, there have been commensurate increases in administration costs associated with its delivery, reflecting a growth in staff tied to the Strategy. The increases in operating and maintenance costs reflect implementation of the design features of the Strategy such as its governance structures, face-to-face meetings with provinces and territories to support SUPPORT Unit applications, iterative peer review processes, Strengthening Workshops for the Networks delivered between Letter of Intent and Full Application stages and the annual SPOR Summit, intended to enable cross-element learning and collaboration.

SPOR has been able to increase its efficiency by using the existing CIHR program delivery mechanisms to invest SPOR Grants and Awards dollars in Foundational investments. There is no net new incremental salary or O&M expenditures related to the delivery of the Foundational elements, as SPOR retains the tracking and oversight of these investments.

Therefore, in calculating the annual operational efficiency of SPOR no direct salary or O&M data for the Foundational investments have been included. The total internal services (indirect administration costs) have been calculated on the entire Grants and Awards envelope which would cover additional small amounts of effort related to Priority Announcements directly under a core element (e.g., Doctoral Research Awards in Patient Oriented Research – Biostatistics under SPOR’s core element of Capacity Development) where needed.

The operational efficiency of SPOR conservatively ranges between 4.6-7.9% (Exhibit 23).

An important principle of SPOR is the leveraging of non-federal partner funds to match, at a 1:1 ratio, the investments made from the federal government toward the core elements of SPOR. For the SUPPORT Units, Networks, Patient Engagement, and Canadian Clinical Trials Coordinating Centre core elements it can be seen that for every $1.00 invested from CIHR, the same or more was invested by a partner over the full time-frame of the investments. For the investments made in capacity development core element, since these were primarily priority announcements for awards (not grants) for New Investigators, Fellows, and Doctoral Researchers, there were no expectations of matching funds.

For those Foundational Investments where such partnership contributions were not required, partnership funding is still evident, however at a much lower rate. This suggests that requiring partner matching funding is an effective mechanism to extending CIHR investments in SPOR core elements.

3.3.4 Conclusions

Although SPOR is in its initial phases of implementation, it has advanced toward achievement of its stated immediate outcome areas. The main areas of achievement include the establishment of six SUPPORT Units that are providing support and services to the POR communities, establishing partnerships, increasing the availability of data to inform and accelerate research, and beginning to undertake POR projects and producing research outputs. Seven national research Networks have been launched, and are at the stage of undergoing implementation. The CCTCC has been operational for just over one year and has delivered against two of its key objectives, with the completion of a model clinical trials agreement and a clinical trials asset map.

SPOR is now committing funding at full scale to its core components and has leveraged partner dollars, meeting is own requirement for 1:1 matching and in most cases exceeding this ratio. SPOR is refocusing its efforts to supporting the core elements as planned, evidenced by a significant decrease in funding directed towards Foundational investments.

SPOR has actively engaged stakeholders across sectors and across participant communities. Extensive efforts have been made at the national and provincial/territorial levels, tying researchers, clinicians, policy makers and patients together across sectors and geographies. However, there are ongoing opportunities to continue to strengthen stakeholder engagement with other networks and institutions involved in POR to further leverage resources and expertise. More important is the requirement to further strengthen SPOR’s PE element through increased work on capacity development (patients learning how to participate in research, and researchers learning how to effectively engage patients) as well as expanding the national lens applied to more consistent and well-communicated development of PE and capacity development efforts.

Through SPOR’s successes in aligning with provincial/territorial priorities, increasing the involvement of all stakeholders in health research, promoting the importance of PE and establishing an environment where the focus is on outcomes of importance to patients, SPOR has somewhat “moved the needle” in terms of influencing the health care research landscape and shifting the culture towards POR. There is still much to do in this area, with culture change being a very long term endeavor and SPOR’s continuing efforts to address the resistance within the various stakeholder communities are important. SPOR is not alone here; there is an international need for stronger and more visible generation of scientific evidence assessing the impacts of implementing a patient-oriented research model.

In terms of progress toward intermediate outcomes, the alignment and integration of the specialized services and methodological expertise of all SPOR core elements will be critical to providing support services responsive to health system needs, generating and translating research to inform health care policy and practice, and improving the clinical trial environment in Canada.

Underlying these core outcomes is the ongoing need for increased awareness of and expertise among researchers in patient-oriented research and enhanced engagement of patients – essential for patients to become active partners in research and clinical practice thereby strengthening Canadian patient-oriented research capacity.

The alignment and integration of the activities of SPOR’s core elements, enhanced engagement of patients and greater clarity in communicating key concepts, future plans and priorities, and early wins to increase buy-in for POR are necessary to complete implementation and achieve intended results.

There are three overarching issues that must be addressed for SPOR to move forward:

• Clarity of design and linkages in support of SPOR element coherence: SPOR is a complex Strategy, with many elements at both the national and provincial/territorial levels. Many of these are still in the early stages of implementation and activity. Others, such as capacity development, are either not yet fully designed or are being modified and refined at the element level (e.g., SUPPORT Unit components) on an ongoing basis as experience is gained. Further, there are many partner organizations, especially at P/T levels, who have existing initiatives into which SUPPORT Units must be situated and provide value-add. As a result, exactly how all these “moving parts” are aligned in terms of priorities and activities, how their various capabilities and services are to be integrated and leveraged, and how they will all work together and be funded going forward are poorly-defined and confusing to many participants. The flexibility SPOR has allowed individual SUPPORT Units, while a clear strength in allowing the SUPPORT Units to respond to P/T priorities, has also resulted in individual provinces building similar but separate initiatives, e.g., their own, PE approaches, clinical trial initiatives, capacity development plans, etc. that may not be the most efficient means of achieving outcomes.

  Similar challenges are common for major new initiatives, but SPOR’s complexities, including those resulting from differing federal and provincial health research and healthcare mandates, make these problems more pointed. Given that SPOR is, by design, directly embedded in the real world of health systems, clinical practice and delivery and patient experience, SPOR must set out a course, but also be responsive to new opportunities and new problems that naturally arise as SPOR elements mature, and individual elements that may respond to these in different, and possibly inconsistent, ways. Thus, more central direction from CIHR is required as to these considerations in order that SPOR continue to be a collaborative national effort, rather than many disparate and only vaguely-related P/T initiatives that continue to evolve in diverging directions.

• Operationalization of definitions: There remains considerable uncertainty about the definitions of key concepts underlying SPOR. First and foremost, there is confusion about exactly what “POR” is, how it’s done, why it’s done, and what benefits will result from it. Within this, the distinction between POR and PE remains unclear. In part because of this confusion, there is a significant gap in knowledge regarding how best to engage patients in research (and when, how, why), and how to inform researchers on the best ways to do so (and again, when, how, and why). Similar uncertainty exists around capacity development, which some respondents believe means “capacity development” and others believe means “career development” – two quite different concepts, as are the best ways of achieving them.

• Communications: There is a need for better communication both within the SPOR community, and outside it to the more general research and clinical communities, as to both the challenges discussed above, so that SPOR is better understood than it is now. A key component of this increased communications effort is the need to conduct studies that investigate the impacts of POR on improving the conduct of research, and in improving the clinical outcomes of that research – in other words, clearly demonstrating the effectiveness and value-add of patient-oriented research approaches will be critical to building the buy-in and creating the culture change needed to achieve the long-term outcomes of SPOR. And if it does, SPOR, by working with and through its funded entities must communicate it. Without robust, peer-reviewed evidence on the processes and outcomes of patient-oriented research and PE, it will remain challenging to achieve buy-in from the research and/or or clinical communities. Assuming that POR is proven to be valuable, there is a need to communicate plans for funding sustainability beyond the initial five-year funding timeframe as it is far too short to expect significant achievements.

4 Recommendations

Overall, SPOR is relevant, addresses an ongoing need for POR, and performance is on track given the stage of implementation –with some adjustments needed to design and delivery.

The following recommendations are identified that have the potential to improve program effectiveness, performance and design/delivery:

1. CIHR should increase efforts to strengthen SPOR’s role in a common agenda for change to patient-oriented research.

• CIHR needs to continue to increase buy-in of POR and change the culture with all identified stakeholders (researchers, clinicians, policy makers, patients). The identification and communication of exemplars of utility to both research and non-research communities is required to provide evidence of the value surrounding POR. Considerations for continued implementation are strategies to identify demonstrable pieces of research (for example PE vs. non-PE) that can be peer reviewed and published and used for dissemination to provide hard evidence as to POR efficacy in improving either research projects or clinical outcomes. Consideration should also be given to CIHR’s use of “moral suasion” and national position to increase institutional buy-in for the consideration of POR in tenure and promotion systems. Strengthening the validation and acceptance of POR will be critical to increasing Canadian capacity in POR overall.
• CIHR should undertake to identify and communicate best practices for patient engagement and capacity development for POR. There is a need to investigate how best to involve patients in governance, and in research, the required training for patients, and how to enhance communications on the patient side. Simultaneously, similar best practices need to be developed on the clinician/researcher side for effectively using patient input, the required training for researcher-clinicians, and how to enhance communications on how/when to best use patient involvement. There is a continued risk to achieving long term outcomes in POR capacity development without providing the stakeholder community with appropriate means to accomplish it.
• CIHR should take the lead in enhancing communication to clarify definitions of many POR terms, including POR itself. CIHR needs to sort out the distinction for stakeholders between capacity development and career development and provide guidance as to priority areas of focus for SPOR. The distinction between PE and patient-oriented research also needs to be clarified and communicated. The clear communication of these foundational definitions will help more widely establish a common understanding of SPOR enabling stakeholders to actively engage and work collectively to achieve evidence-based improvements.
• CIHR should encourage the SUPPORT Units to increase communications and outreach to their broad stakeholder communities in relation to SUPPORT Unit services and initiatives undertaken. At a minimum, a SUPPORT Unit should develop a clear communication and marketing plan to support its efforts. Where such plans exist, they should address public communications, including a more consistent and up to date web presence across SUPPORT Units, as it is currently very difficult to sort through information and determine what parts of the discussion refer to SPOR and what parts refer to non-SPOR organizations and activities. For all SUPPORT Units, this should include a description of how patients, clinicians, and researchers can become engaged, what the SUPPORT Units governance structure is, and examples of how patients are already actively involved. Often users are referred elsewhere and SPOR linkages are not evident. CIHR should consider promoting the benefits of well-designed informational elements. The absence of targeted communications and outreach limits the awareness of targeted participants and the involvement of the necessary partners required to enable dissemination and integration of research results into practice.

2. CHIR should provide strategic guidance regarding how SPOR elements are to work together toward achieving SPOR’s intermediate and long-term outcomes.

• CIHR, in collaboration with its established SPOR governance structures should work together to enhance guidance on operationalizing SPOR elements, in particular, clarifying how different SPOR elements are expected to work and coordinate together. This is particularly relevant for SUPPORT Units and Network interactions and the resulting implications for obtaining and accounting for matching funds, as well as how to address considerations for allocation to national versus provincial/territorial priorities when connecting across elements. Without operational requirements outlined and communicated, elements will not be acting in concert with each other towards an outcome that optimizes integration of resources and outcomes.

3. CIHR should communicate plans for moving beyond the initial five-year funding period to manage sustainability expectations for CIHR investments in SPOR.

• CIHR needs to provide clear communications regarding SPOR funding, or options, beyond the current five-year funding commitment to some elements. In light of the current stage of implementation and progress made, a reassessment of the five year timeframe for funding of some elements should be undertaken through a strategic planning exercise, identifying the resources and actions necessary to continue the momentum achieved to date. At a minimum, funding recipients should be informed of future SPOR planning. Addressing the perceptions of uncertainty is important, as in its absence, it may limit further progress to embedding the POR model in all jurisdictions and encourage a response that may involve developing additional efforts outside of the strategy.

4. CIHR should strengthen approaches to enable cross-learning, sharing of best practices, and collaboration; this should occur within and across SPOR elements and between CIHR and Canadian and International organizations.

• CIHR needs to re-examine the structure, operations and effectiveness of working groups, and other mechanisms, at the national level to assist in limiting possible duplication of effort and assist with leveraging experience and shared learning. Consideration should be given to resources that could be shared across regions (e.g., CD and PE efforts). There is some risk that full advantage is not being taken in the development of common tools and practices.
• CIHR in collaboration with the established SPOR governance structures, should encourage cross-provincial/territorial initiatives among SUPPORT Units to investigate opportunities for increased cost-effectiveness and cost-efficiency, for example, through harmonization of PE and capacity development initiatives, or direct leveraging of deep pockets of provincial expertise across the country. Not utilizing, aligning and integrating services and methodological expertise may hinder implementation in fostering evidence-based improvements in the most efficient manner.
• CIHR and all SPOR elements should encourage increased and/or ongoing interaction/connection and relationship building with other POR initiatives, e.g., CFHI, CIHI, provincial health quality councils, health charities, foundations, and patient organizations such as Patient Voices Network and Patients Canada as well as international initiatives such as PCORI. These linkages and connections, beyond bilateral relationships, could increase the power of learning, sharing and generating impact.

5. CIHR should continue to support effective management and administrative functions within funded SPOR SUPPORT Units and Networks and across these elements.

• CIHR should require SPOR SUPPORT Units and Networks to be supported by CEO/COO-type management positions, if not already present, to help manage operational obligations, administrative requirements and the high corresponding workloads in these areas. Leadership, administration, and business management skills in such a complex initiative are much more important than a "normal" research grant project. Program management and business planning functions may not be at their opportune levels if there are not dedicated resources driving these activities and researchers and/or clinicians are not typically best placed (in terms of time commitment) to undertake these functions (e.g., agenda setting, strategic and site-specific planning, communications within and between components or sites, scheduling and funding)
• CIHR should review the funding model in place for SUPPORT Units and Networks, including the timing of funding flow, and adjust timing based on the stage of development/need by the elements due to the significant ramp-up periods experienced across all elements. The effect of this is often increased requirements to manage forecasts, spend and adjustments to budgets.

6. CIHR should revise the existing SPOR performance measurement strategy to balance administrative/operational outputs with outcomes/impacts.

• The existing SPOR performance measurement strategy needs to be revised with a stronger focus on measuring outcomes and impacts with logical links back to SPOR’s key objectives of improving patient health and an improved Canadian health care system. This should include a focus on the value-added by SPOR as a strategy, not only the sum of its elements. This can include, but not be limited to the coherence and relationships across SPOR elements.
• Indicators and corresponding reporting should be re-oriented from tracking primarily activity-based or output indicators toward a clear set of measures that will demonstrate the outcomes and impacts of SPOR overall, as well as the impact of each of its elements. Without the ability to demonstrate impacts at the system and, more importantly, patient levels, SPOR risks a continued challenge of generating buy-in for POR in addition to maintaining the interest and commitment of partners. This should include a mechanism to roll-up and report back findings from performance measurement activities within and across SPOR elements to funding recipients and governance structures to inform course corrections. The upcoming SPOR SU evaluations will provide one opportunity to do this.
• CIHR should work to implement improvements in financial monitoring and coding for SPOR grants and awards expenditures (including partner contributions) and for operating and maintenance expenditures, particularly in terms of improving consistency and availability of financial data over time.
• Further, consideration should be given to approaching the revision of the SPOR performance measurement strategy with a view to applying a “collective impact” lens. This considers the partnered approach and the complexity of the issue being addressed (improving health outcomes) which is not an issue CIHR can tackle in isolation. CIHR is in the best position to drive collaborative determination of requirements and expectations across stakeholders. This can assist in supporting Recommendation #1 with provision of evidence to back the POR change agenda. Without some commonality of measurement, SPOR and its partners risk having unaligned goals but also potential missed opportunities to collaboratively learn and increase effectiveness and impact for all participants.

Appendices

• A – SPOR Logic Model
• B – CIHR Institutes: Engagement with SPOR

Appendix A – SPOR Logic Model

SPOR Logic Model

Appendix B – CIHR Institutes: Engagement with SPOR

Date modified:

2016-09-20