CIHR Dementia Research Strategy – Evaluation Report

May 2021

At the Canadian Institutes of Health Research (CIHR), we know that research has the power to change lives. As Canada's health research investment agency, we collaborate with partners and researchers to support the discoveries and innovations that improve our health and strengthen our health care system.

Canadian Institutes of Health Research
160 Elgin Street, 9th Floor
Address Locator 4809A
Ottawa, Ontario K1A 0W9

This publication was produced by Ference & Company Consulting Ltd. in consultation with the Canadian Institutes of Health Research. The views expressed herein do not necessarily reflect those of the Canadian Institutes of Health Research.

Acknowledgements

Special thanks to all participants in this evaluation. Also, thank you to those who supported the evaluation: Sarah Boorman, Meredith Haaf, Don Ference (Ference & Company), the Institute Teams and Scientific Co-Leads from the Institutes of Aging (IA) Dr. Jane Rylett, and Neurosciences, Mental Health and Addiction (INMHA) Dr. Samuel Weiss, and the former Institute of Aging Scientific Director, Yves Joanette, who completed his term in July 2019.

The Dementia Research Strategy (DRS) Evaluation Team

Ian Raskin, Michael Goodyer, Jean-Christian Maillet, Jonathan Gilbert, Ellie Fletcher, Daniela Corno, Carmelle Goldberg, Alison Croke, Jenny Larkin, Sabrina Jassemi, Alexandra Leguerrier.

For more information and to obtain copies, please contact: Evaluation@cihr-irsc.gc.ca.

Table of Contents

List of Figures

List of Acronyms

Acronym Meaning
ADNI Alzheimer's Disease Neuroimaging Initiative
ARC Average Relative Citations
ARIF Average Relative Impact Factor
CCNA Canadian Consortium on Neurodegeneration in Aging
CIHR Canadian Health Institutes of Health Research
CoEN Centres of Excellence on Neurodegeneration
DRS Dementia Research Strategy
EDI Equity, Diversity and Inclusion
EIS Electronic Information System
EU European Union
GBA+ Gender-Based Analysis plus
IA Institute of Aging
ICRH Institute of Circulatory and Respiratory Health
ICRSAD International Collaborative Research Strategy for Alzheimer's Disease
IGH Institute of Gender and Health
IIPH Institute of Indigenous Peoples' Health
IMIS Initiative Management and Institute Support
JPND Joint Programme - Neurodegenerative Disease Research
KTEP Knowledge Translation and Exchange Program
MAP Management Action Plan
NGOs Non-governmental organizations
OIBA Observable Influence Beyond Academia
OST Observatoire des sciences et des technologies
PHAC Public Health Agency of Canada
RIU Results and Impact Unit
PRA Prairie Research Associates
RRS Research Reporting System
SI Specialization Index
SOCI Social Affairs, Science and Technology
TBS Treasury Board Secretariat
WGSD Women, Gender, Sex, and Dementia
WHO World Health Organization

Executive Summary

Program Overview

The Dementia Research Strategy (DRS) is led by the Institute of Aging (IA) in partnership with the Institute of Gender and Health (IGH), the Institute of Indigenous Peoples' Health (IIPH), the Institute of Circulatory and Respiratory Health (ICRH), and the Institute of Neurosciences, Mental Health and Addiction (INMHA). DRS supports research on the latest preventive, diagnostic, and treatment approaches to Alzheimer's disease and other neurodegenerative diseases causing dementia. The DRS is comprised of an international and national component that enables Canadian researchers to lead and participate in a number of new national and international initiatives. The ultimate goal of the DRS is to contribute to the global pursuit of finding a cure or disease-modifying treatment for dementia by 2025.

Evaluation Overview

The objective of this evaluation is to provide management with timely, actionable evidence that will inform CIHR decision-making and planning regarding future activities and investments in dementia research. The DRS program evaluation assesses the relevance, design and delivery, and performance of the initiative during the period from 2009-10 to 2017-18, using a range of methods and data sources to triangulate evaluation findings.

The evaluation covers activities and investments of the international component (e.g., France-Québec-Canada, China-Canada Initiative on Alzheimer's Disease and Related Disorders, Centres of Excellence on Neurodegeneration [CoEN], Alzheimer's Disease Neuroimaging Initiative [ADNI], and Joint Programme - Neurodegenerative Disease Research [JPND]) and national component (e.g., Canadian Consortium on Neurodegeneration in Aging [CCNA]). The DRS activities under the New Directions in Dementia Research suite of programs were not assessed by the evaluation given that these activities were still novel at the time of this evaluation.

Key Findings

Relevance

The social and economic burden of dementia is increasing in Canada and around the world, creating a strong need for additional research on prevention, treatment, and quality of life improvements for those living with the disease as well as their caregivers. Stakeholders believe that the DRS meets this need to a moderate extent and generally complements other dementia research funding opportunities and supports within the Canadian and international research landscapes.

The federal government has a clear role in providing support for researchers to engage in innovative national and international collaboration and leadership initiatives that support dementia research. Federal agreements, legislation, and strategy documents – including the National Dementia Strategy released in 2019 – underscore that supporting dementia research is a key ongoing role and priority of the federal government.

CIHR's role in supporting dementia research directly aligns with its stated objective to contribute to scientific excellence in health and health care systems research as outlined in the CIHR Act (2000) and the CIHR strategic plans in place during the period under review. CIHR's role in providing support for dementia research through the DRS is directly aligned with the CIHR's objective stated in the Act to "excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system." CIHR's past two strategic plans: Roadmap (2009-10 – 2013-14) and Roadmap II (2014-15 – 2018-19) emphasize the need for the Agency to promote innovation, national and international collaboration, and research focused on improving quality of life for individuals living with chronic conditions such as dementia.

It is worth noting that researchers, partners, CIHR Senior and Program Management, and the Senate Standing Committee on Social Affairs, Science and Technology all voiced concerns that the current level of funding of the DRS is insufficient to meet federal priorities and policy commitments. More research, including patient-centered research, is needed to address knowledge gaps (e.g., around understanding dementia onset and the range and extent of its impacts) to help decision-makers develop appropriate research and funding strategies.

Design and Delivery

Overall, the design and implementation of the DRS have been effective in supporting progress toward DRS objectives. In particular, the national and international components of the DRS share the same research themes and overall objectives. However, the oversight and the governance structures differ between both components and there is a lack of documentation supporting the relationship between the two components. There is a need to better integrate the national and international components of the DRS.

The international component supports DRS objectives by facilitating Canada's participation in key international research initiatives and establishing linkages between the international research community and key stakeholders. The CCNA is an innovative model for supporting collaborative dementia research partnerships within Canada, with structures and processes such as cross-cutting activities that support research teams and ongoing communication that increase engagement, facilitate knowledge exchange, and build capacity in the dementia research enterprise.

The evaluation identified the following challenges and areas for improvement regarding the design and delivery of the DRS:

Performance

The DRS has contributed to increasing research capacity in the field of dementia. This is evidenced by the fact that DRS researchers received more CIHR funding in the field of dementia research following the launch of the national and international components. DRS also supported trainees through funding and development opportunities. However, there may be room for improvement in communication and/or processes for awarding trainee funding given conflicting views on the need for more funding and reports of issues with the unsuccessful allocation of partner funds.

The DRS is making progress on a number of expected immediate outcomes. The CCNA supported researchers to form inter-institutional, interdisciplinary, Canada-wide collaborations and share data and resources, including biological tools, new techniques and protocols, technology and equipment, data for developing joint publications, and jointly supervise graduate students and postdoctoral fellows. Key informants also emphasized how the size, reach, and synergistic approach of the CCNA network and the collaboration and community involvement it supports further strengthen innovation in Canadian dementia research.

While it is too early to see the full reach and impact of overall DRS-supported research beyond academia, the DRS is making progress toward some of its expected intermediate outcomes. Evidence suggests some DRS funded research supported knowledge translation and dissemination through peer-reviewed publications, presentations, and methodologies.

The DRS, mainly through CCNA, is also improving consideration of equity, diversity, and inclusion (EDI) factors in dementia research as well as promoting the inclusion and growth of Early Career Investigators and geographical diversity. Improved consideration of sex and gender in dementia research is supported by the Institute of Gender and Health's involvement in the DRS and having a cross-cutting CCNA theme on "Women, Sex and Gender in Dementia" dedicated to the topic. Support for Indigenous-specific dementia research is incorporated into the CCNA, with a research team focusing on issues in dementia care for rural and Indigenous populations.

The DRS supported increased international recognition of Canada as a collaborative dementia research hub by supporting Canadian participation in international events and partnerships, promoting visibility of Canadian research, and providing opportunities for relationship building with international stakeholders.

Recommendations

The evaluation makes three recommendations to improve the performance of the DRS to achieve its expected results.

Recommendation 1:

CIHR should continue to invest strategically in the DRS and re-assess the nature and extent of funding to meet its expected outcomes as well as the needs of the National Dementia Strategy.

Recommendation 2:

CIHR should ensure that the appropriate objectives, resources and structures are in place to better integrate the national and international components of the DRS to work cohesively and synergistically to achieve its overall objectives.

Recommendation 3:

CIHR should examine approaches to enhance communication with, and engagement of, stakeholders, particularly DRS partners, knowledge users, people living with dementia, and their caregivers.

The management response to the evaluation recommendations can be found in the Management Action Plan section of the report.

Program Profile

Program Overview

CIHR's Dementia Research Strategy (DRS) supports research on the latest preventive, diagnostic, and treatment approaches to Alzheimer's disease and other neurodegenerative diseases causing dementia and enables Canadian researchers to lead and participate in a new wave of national and international initiatives. Ultimately, its goal is to contribute to the global pursuit of finding a cure or disease-modifying treatment for dementia by 2025.

The DRS is under the scientific leadership of the Institute of Aging (IA) and is delivered in partnership with the Institute of Gender and Health (IGH), the Institute of Indigenous Peoples' Health (IIPH), the Institute of Circulatory and Respiratory Health (ICRH) and the Institute of Neurosciences, Mental Health and Addition (INMHA)Footnote 1. The ongoing management of the DRS occurs through monthly DRS Integrated CIHR Working Group meetings and weekly meetings between the DRS IA's Scientific Director (accompanied by Institute employees), and Initiative Management and Institute Support (IMIS) staff.

The oversight of the national and international components of the DRS differ from one another. For instance, the Canadian Consortium on Neurodegeneration in Aging (CCNA), as a major initiative within CIHR and key initiative of the national component, has an oversight committee, composed of three CIHR representatives (Director General, Scientific Director leads) and six partnership representatives. Whereas, each initiative currently funded within the international component has its own oversight structure.

DRS objectives over the period of 2014-15 to 2018-19 were to:

In pursuit of these objectives, the DRS supports Canadian researchers to lead and participate in a new wave of national and international initiatives under the following three themes:

Components

The DRS is composed of an international component and national component that were launched at different points in time and are intended to complement each other and capitalize on the recognized excellence of Canadian research.

International Component

CIHR launched the International Collaborative Research Strategy for Alzheimer's Disease (ICRSAD) in March 2009. ICRSAD was renamed as the international component of DRS in 2015. This international component supports collaboration at the international level by creating a global network for generating and sharing knowledge on dementia prevention, therapies, care, quality of life and a cure.

The objectives of the international component include strengthening excellence, optimizing Canadian synergy, with a vision to position Canada as a world leader in research on prevention, early diagnosis, and treatment of Alzheimer's disease and other neurodegenerative diseases causing dementia.

This component facilitates Canada's participation in several key international initiatives through annual calls and alignment in research priorities. It supports Canadian researchers to participate in key international initiatives such as:

Past initiatives include:

National Component

In 2014-15, CIHR launched the CCNA, a ring-fenced initiative, to be the centerpiece of the national component of the DRS.

Resources

CIHR investments in the DRS which are within the period of the evaluation represent a total of $37.7 million between 2009-10 and 2017-18Footnote 2. Of this total investment, $21.0 million has been allocated to supporting the national component since the launch of the expression of interest in 2013-14 (average of $4.2 million per year) followed by launch of the CCNA in 2014-15 and $16.8 million has been allocated to supporting the international component since its launch in 2009-10 (average of $1.9 million per year).

During the same period (2009-10 to 2017-18), CIHR invested a total of $290.6 million in dementia related research, excluding the DRS, averaging $32.3 million per year. Figure 1 shows the funding allocated by CIHR to dementia related research and the DRS, by fiscal year.

Investments in the DRS represent a small proportion of CIHR's total investment in dementia related research. In fact, between 2009-10 and 2017-18 DRS accounted for only 11% of CIHR's total investments in dementia. This proportion increased to 18% with the launch of the CCNA as part of the national component in 2014-15. Figure 2 shows the percentage of investments in DRS compared to the investments in related research.

Description of Evaluation

Purpose, Scope and Context

The purpose of this evaluation is to provide CIHR senior management with valid, insightful and actionable findings regarding the following:

This evaluation covers the period from 2009-10 to 2017-18 and is the first evaluation of the DRS since its inception. The following initiatives were in scope for the evaluation:

The New Directions in Dementia Research suite of programs were out of scope for this evaluation due to their early stage of implementation.

The evaluation of DRS, committed to as part of CIHR's 2018-19 Evaluation Plan, was designed to help inform CIHR's decision-making and planning regarding the activities and investments for future phases of the DRS. The evaluation meets CIHR's requirements to the Treasury Board Secretariat (TBS) of Canada under the Policy on Results and the Financial Administration Act.

It is important to note that the DRS initiative continued to progress in parallel with the ongoing program evaluation. Most notably the CCNA was renewed in 2019 and although the timing of a final evaluation report did not align with the renewal, preliminary evaluation findings informed decision-making related to the funding renewal and continuous improvements to program elements.

The COVID-19 Pandemic

The ongoing COVID-19 pandemic has unprecedented impacts on individuals living with dementia, their carers, and the availability of key dementia supports and services, as well as on the DRS. While ongoing research is being conducted to explore the nature and extent of these impacts evidence shows that "people with dementia are more likely to be negatively impacted by social isolation, anxiety and confusion from worrisome news in the media and be at greater risk of behavioral changes, confusion and delirium during this period – putting them at risk of being hospitalized and hence further exposed to COVID-19"Footnote 3. The CCNA has developed a collaborative task force with the Alzheimer Society of Canada (ASC) to roll out new research endeavors aimed at addressing the social, ethical and biomedical issues surrounding the impact of COVID-19 on individuals living with dementiaFootnote 4. More specifically, CCNA's Team 19 is conducting a number of new research projects within this task force, including an evaluation of the use of healthcare services and the analysis of deaths due to COVID-19 among people suffering from dementia and the organization of healthcare for Canada's elderly in the far NorthFootnote 5.

Methodology

The evaluation addresses the following core issues of the DRS in accordance with the TBS Policy on Results (2016) and information needs of senior and program management:

It also considers equity analyses in accordance with CIHR's commitment to use Gender-Based Analysis plus (GBA+) to develop effective policies and programs.

Evaluation Questions

The evaluation addresses the following evaluation issues and questions:

Relevance:

  1. To what extent is DRS meeting the need to support world class research on dementia?
    • 1.1. To what extent have DRS components been aligned with federal government and CIHR priorities?
    • 1.2. To what extent have DRS components been aligned with federal roles and responsibilities?
    • 1.3. To what extent has DRS continued to address a demonstrated need?

Design and Delivery:

  1. To what extent have the design and implementation of DRS been effective and efficient?
    • 2.1. To what extent has DRS been an appropriate approach to supporting research on neurodegeneration in aging?
    • 2.2. To what extent are the DRS components being delivered in a cost-efficient manner?
    • 2.3. What improvements, if any, can be made to the design and delivery of DRS?

Performance:

  1. To what extent has DRS achieved expected objectives?Footnote 6
    • 3.1. To what extent has DRS made progress toward the achievement of expected immediate outcomes?
    • 3.2. To what extent has the DRS made progress toward the achievement of expected intermediate outcomes?

Evaluation Approach

Consistent with TBS guidance and best practices in evaluation, multiple lines of evidence were used to synthesize and triangulate evaluation findings, including both qualitative and quantitative data as well as primary and secondary data sources.

The evaluation employed the following lines of evidence:

Additional details about the methodology are provided in the Appendix B: Methodology.

Limitations of the Evaluation

It is common for evaluations to face limitations that can influence the validity, reliability, and utility of findings. Key limitations associated with the evaluation are outlined below.

These limitations were mitigated through the tri-angulation of results across data sources as well as consideration of the relative strengths and weaknesses of each data source. The mitigation strategies employed throughout the evaluation help ensure that the evaluation results can be used with confidence to inform program decision making. The limitations and mitigation strategies are discussed in more detail in Appendix B.

Evaluation Findings

Relevance

Key Findings:

The DRS addresses a demonstrated need to support dementia research and builds on Canada's international competitiveness in dementia research.

Dementia poses a significant and growing social and economic burden both in Canada as well as around the world. The World Health Organization (WHO) projects that the global number of individuals living with dementia will increase threefold, reaching 145 million by 2050Footnote 7. Meanwhile, the burden on the Canadian economy is expected to increase from $33 billion annually in 2015 to $293 billion annually by 2040 when both direct medical and indirect costs are combinedFootnote 8Footnote 9.

As of 2015-16, there were already more than 419,000 Canadians aged 65 or older living with diagnosed dementia, approximately two-thirds (65%) of whom were womenFootnote 10Footnote 11. Evidence also indicates that Indigenous populations (e.g., First Nations, Inuit, and Métis) are at a higher risk of dementia compared to the rest of the Canadian population due to the increased prevalence of risk factors associated with its development, many of which stem directly from the impacts of colonization (e.g., physical inactivity, low educational attainment, and chronic conditions such as diabetes, obesity, and hypertension)Footnote 12.

In light of the current and projected burden of dementia on individuals and society, the majority of key informants agreed that the DRS meets the need for dementia research in Canada to at least a moderate extent. To address this need, the DRS is able to leverage Canada's international competitiveness across its priority areas, which include: Prevention and Control of Dementia; Therapeutics of Dementia and Quality of life. To this end a bibliometric analysisFootnote 13 was conducted to help assess Canada's competitiveness in dementia research.

Overall, the findings of the bibliometrics analysis indicated that the annual number of publications by Canadian researchers has increased in each priority area corresponding to the inception of DRS in 2009. Specifically, the number of publications in the priority areas of Therapeutics of Dementia and Quality of life increased steadily after 2010 from 15 to 35 and from 3 to 15 respectively. The number of publications in the priority area of Prevention and Control of Dementia increased from 25 to 51 between 2010 and 2016 (Figure 3). Over this same time period (2000-2016), Canada is ranked 4th among the top 10 most productive nations in the number of publicationsFootnote 14 (Figure 4). The majority of countries in the top 10 most productive nations are partners with Canada on initiatives that are part of the DRS. Additionally, the bibliometric analysis indicates that Canada is more specialized across the three priority areas as whole, when compared to the world average as measured by the Specialization Index (SI)Footnote 15. In fact, Canada ranks 3rd among the top 10 most productive countries for the SI of publications (Figure 5).

Finally, as an assessment of research quality, the findings from the bibliometric analysis indicate that Canada ranked 3rd for the Average Relative Impact FactorFootnote 16 (ARIF) which indicates that Canadian researchers are publishing in journals with a high Impact Factor (Figure 6). This finding aligns with data indicating that the number of citations received by Canadian publications is also well above the world average, with Canada ranking 5th for the Average Relative CitationsFootnote 17 (ARC) (Figure 7).

The DRS is aligned with federal government and CIHR priorities as well as federal roles and responsibilities.

All key informants agreed that the federal government has a necessary role in supporting dementia research, with interviewees citing federal government agreements, legislation, and strategy documents (outlined below) that demonstrate alignment of the DRS with federal roles and responsibilities and showcase the federal government's ongoing priority of supporting dementia research. Specifically, key areas of alignment included the focus on increasing investment in dementia research, supporting innovative national and international research coordination and collaboration, and promoting Canadian leadership in the field, as highlighted in the following:

Further, CIHR's role in providing support for dementia research through the DRS is directly aligned with the CIHR Act (2000), specifically the objective to "excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system"Footnote 26. In particular, the DRS addresses this objective by providing support for researchers to engage in innovative national and international collaborations and leadership initiatives that support dementia research on primary and secondary prevention, improve the quality of life and the efficiency of the health care system.

CIHR strategic plans for Health Research Roadmap: Creating innovative research for better health and health care (2009-10 to 2013-14) and Health Research Roadmap II: Capturing Innovation to Produce Better Health and Health Care for Canadians (2014-15 to 2018-19) outline the need for the Agency to promote innovation, national and international collaboration, and research focused on improving quality of life for individuals living with chronic conditions such as dementiaFootnote 27, Footnote 28.

DRS funding is insufficient to meet all of its objectives as well as the CIHR and federal government priority commitments.

There is a clear continued need for dementia research in Canada, as evidenced by the increasing number of people affected and the projected social and economic burden of the disease on Canadians. Despite the clear alignment of the DRS with federal priorities and evidence that the DRS is addressing dementia research needs to some extent, there are concerns that the current funding level for the initiative is insufficient to meet all of its stated commitments and achieve progress on expected outcomes as intended.

The majority of CIHR Senior and Program Management key informants reported that DRS funding was "not sufficient to meet CIHR and federal government priority commitments". Key informants also mentioned that more funding is required to achieve progress toward finding a cure or disease-modifying treatment to improve the quality of life for individuals living with dementia and their caregivers by 2025.

To complement key informant statements regarding the insufficient level of funding, since the launch of the national component in 2014-15, CIHR investments in DRS represent an average of $7.3 million annually which is part of the CIHR investments in dementia research averaging $40 million per year. Based on the 2013 G8 Dementia Summit Declaration and the recommendation from the Senate Standing Committee on Social Affairs, Science and Technology (SOCI), research investments in dementia research should represent 1% of current dementia care costs, which for Canada represents approximately $100 million toward dementia research each yearFootnote 29.

There is also a need to actively involve people with lived experience of dementia and their caregivers in setting research priorities and identifying knowledge gaps. For instance, several key informants identified knowledge gaps around the understanding of dementia onset and impacts (e.g., dementia types, prevalence, and symptoms), underscoring the challenge of determining the extent of the burden and efficiently targeting resources for research and care in response. Some key informants also noted a lack of involvement of people with lived experience of dementia in the prioritization of research topics. They suggested that this would be addressed through adoption of a more patient-centred approach with better linkages between researchers and individuals with lived experiences. Key informants reported that these unmet needs were limiting factors in developing appropriate research and funding strategies.

DRS generally complements other funding opportunities and initiatives that support dementia research.

Approximately half of key informants indicated that the DRS is complementary to support provided by not-for-profit and private sector organizations, other provincial and federal initiatives, and international funding opportunities. For instance, the DRS complements other initiatives that in some way support dementia research, by establishing and strengthening partnerships that enable coordination and synergism across the Canadian research landscape. For example, the CCNA and CLSA partnered on a CIHR Team Grant exploring the impact of big data on dementia. The project entitled Broad and Deep Analysis in Neurodegeneration (BRAIN) will track the longitudinal trajectory of dementia across several datasets to develop algorithms for delineating cognitive decline in the CLSA. Furthermore, the majority of DRS researchers interviewed reported that they had also applied for additional funding from both CIHR and the Alzheimer's Society of Canada.

Some key informants also noted overlaps between the DRS and other funding opportunities while specifying that it may be justifiable given the need for added funding and attention in certain research areas.

Design and Delivery

Key Findings:

The two components of DRS support the overall objectives, but better integration of the components would enhance cohesiveness and synergy.

The DRS was launched in 2014 when CCNA, the first initiative within the national component, was integrated with CIHR's foundational investment in ICRSAD. The strategy and its components share a common overall objective namely, "to support world-class research on dementia that will contribute to the global pursuit of finding a cure or disease-modifying treatment for dementia by 2025" and same three themes (Primary Prevention, Secondary Prevention and Quality of Life).

Despite having common objectives and themes, a few key informants indicated that they were unaware of, or unclear on the linkages between the two components. These perceptions are further supported by program information which shows that each component has distinct governance and a different approach to structuring funding opportunities.

For instance, the international component is a suite of independent funding opportunities (e.g. CoEN, JPND, ADNI) which are overseen by their own governance structures and focus on international collaborations. Specifically, CIHR, via IA, plays an active role as a full voting member in the governance bodies of JPND and CoEN. CIHR was represented by Canadian ADNI site researchers on the ADNI Private Partner Scientific Board. Conversely, CCNA is comprised of a unified funding structure which allocates funds to teams, themes and programs. CCNA also has a well-defined governance structure which consists of three core components: Research Executive Committee, Strategic Advisory Council, and International Scientific Advisory Board (Figure 8). External oversight of the CCNA is provided by a CIHR CCNA Oversight Committee, which is comprised of three CIHR representatives (Director General and Scientific Director leads) and six partnership representatives involved in CCNA. The CCNA Oversight Committee raised concerns regarding the under-utilization of the International Scientific Advisory Board and the Strategic Advisory Committee. In the case of the Strategic Advisory Committee, it is reported that they have only met together once, although members are consulted on an ad hoc basis.

An overarching governance and oversight structure between the national and international components of the strategy would allow for a more integrated approach between the two components. This would better support the sharing of skills and expertise between national and international researchers; thereby, enhancing capacity building.

Overall, the evaluation found that there was a lack of documentation on the international component of the DRS, as well as on the relationship between the national and international components. In fact, documentation, key informants, financial and grant data for this evaluation were predominately CCNA oriented (given it is the centerpiece of the national component), despite attempts to seek additional information to better understand all the initiatives of DRS. There is a need to better integrate the two components and document how the two components work together to achieve the overall objectives of the DRS.

The international component facilitates Canada's participation in key international research partnerships.

The international component of the DRS facilitates dementia research partnerships between Canadian and international researchers and organizations, through a series a funding opportunities across several different initiatives (see Table 1).

Several key informants stressed the importance of the continuity of the international component of the DRS, highlighting its fundamental role in facilitating partnerships between national and international stakeholders in the field of dementia research. In addition to facilitating partnerships between countries, CoEN demonstrates a focus on establishing linkages between the international research community and industry partners to further accelerate research toward novel therapies.

Table 1: Initiatives Supported by the International Component of the DRS

Initiative Description of Initiative & Canadian Contribution Nature of Support Provided by the International Component
Alzheimer's Disease Neuroimaging Initiative (ADNI) A partnership with the US, the ADNI unites researchers with study data as they work to define the progression of Alzheimer's disease. Its vision is to create an open-access database for all participating researchers using a harmonized protocol. Canada is an integral part of the initiative, with participating centres at Sunnybrook Health Sciences Centre in Toronto, McGill University and the Jewish General Hospital Memory Clinic in Montréal, the University of British Columbia in Vancouver, and Parkwood Hospital in London. In addition to supporting ADNI's four Canadian Centres, CIHR launched a directed grant funding opportunity (ADNI 3) in 2017 to add a coordinated and centralized brain donation and brain banking system (neuropathology infrastructure) for ADNI participants in order to optimize Canada's participation in the ADNI initiative. Brain autopsy programs have become a central feature in ADNI but had been absent from Canadian dementia research until this point.
Network of Centres of Excellence on Neurodegeneration (CoEN) This initiative involves Canada and 8 partner countries in Europe. It aims to encourage collaborative research between recognized national centres of excellence in neurodegeneration through two-year pathfinder grants in order to accelerate research on underlying mechanisms of dementia and identify new therapeutic approaches. CIHR is a founding partner of this initiative and member of the Management Board and Steering Committee. Since 2011, CoEN has funded 11 projects involving Canadian researchers, including recent projects on identifying targets for disease-modifying treatments and mechanisms behind vascular cognitive impairment. In addition to funding, CoEN provides a mechanism for linking industry to Centres of Excellence and developing novel industry partnerships in pre-competitive research.
EU Joint Programme – Neurodegenerative Disease Research (JPND) The largest global research initiative aimed at tackling the challenge of neurodegenerative diseases, JPND aims to increase coordinated investments between participating countries for research aimed at finding causes, developing cures, and identifying appropriate care. It currently involves 27 partner countries. In 2012, with support from CIHR, Canada was the first non-European country to join this initiative and CIHR has been a full member since 2014. Since then, 13 projects supported by JPND have involved Canadian researchers, including four international projects for which Canadian researchers have served in a role equivalent to a Nominated Principal Investigator in Canada. Further, CIHR is also a member of the Management Board and Steering Committee and has played a key role in developing Strategic Research Agendas for the initiative.

CCNA's model is supporting innovative, collaborative research and research capacity, although there are challenges related to its complexity.

At the time of its inception, the CCNA was a novel and complex structure to facilitate and support successful dementia research partnerships across Canada, as acknowledged by its Oversight Committee which characterized the model as a "very new way of doing research"Footnote 30. In its first phase (2014-2019), the CCNA consisted of five core elements, including four cross-cutting Programs, three research Themes, twenty research Teams, and eight Platforms to support teams, as well as an underlying central governance structure. Through these different elements, CCNA has embedded collaboration within its design. For instance, each of the 20 topic-based research team was required to have researchers from more than one university, centre, or province, with an average of 20 members per research team. The Oversight Committee also noted that the model purposefully included "co-leadership and built-in redundancies to the Teams and Themes to ensure corporate knowledge and sustainability"Footnote 31 – elements that support continued functioning (e.g., collaboration) as an initiative evolves.

Key informants and the CCNA Oversight Committee indicated that the structure of the CCNA supported collaborative research as intended. Many key informants expressed that CCNA fostered collaboration through structures and processes such as the cross-cutting activities and ongoing communication required between inter-institutional, interdisciplinary research teams, which served to increase engagement, facilitate knowledge exchange, and build capacity. Program data also reveal that two of the cross-cutting activities were found to be particularly supportive for all of CCNA's research teams. These included the Knowledge Translation and Exchange Program (KTEP) and Women, Gender, Sex, and Dementia (WGSD) cross-cutting components, the latter of which effectively helped to emphasize the importance of sex and gender in dementia research across all CCNA research. This approach represents a unique Canadian contribution to the overall dementia research landscape. The Oversight Committee also reported being pleased with overall progress made by the CCNA throughout its first phase and recommended continuation of funding as per its Terms of ReferenceFootnote 32. Committee members found the CCNA model to be appropriate and effective in supporting the achieving the objectives of the national component of the DRS (e.g., increasing interdisciplinary collaborations, building capacity in dementia research and enhancing the national and international presence of Canadian researchers).

Despite these findings, program documents and key informants identified some challenges with the structure's model. The CCNA Oversight Committee raised various concerns regarding the decision making criteria and the need for clarity around the process to assess CCNA's leadership. Several key informants cited the need for greater clarity around the roles and responsibilities of the CCNA's leadership, as well as more transparency around CCNA's functions (e.g., allocation of funds, structure of teams) as key areas for improvement.

Other key informants questioned the appropriateness of CCNA's staffing structure and centralized leadership model. A couple of key informants stated that the CCNA model was too complex with its three themes, 20 different programs and eight platforms that are attempting to tackle too many elements; this has ultimately led to insufficient resources to meet all stated objectives. In its fifth Progress Report, CCNA stated that it "has never been a 'consortium' in anything but name. It is, in fact, a funded network."Footnote 33 The proposed structure was and remains complex with: 20 topic based Teams; six Platforms; four Cross-Cutting Programs; three Themes and one Central AdministrationFootnote 34. In response, the CCNA Oversight Committee acknowledged the complexity of the CCNA and the inherent challenges in managing this large and multifaceted entityFootnote 35. In response to concerns regarding its structure, the CCNA staffing structure was adjusted in 2017 to meet the needs of the consortium with a new research administrator, administrative coordinator, clinical research assistant, and communication officer. The CCNA also decentralized the leadership of the CCNA by creating Associate Director positionsFootnote 36 to be filled by researchers in order to ensure transparency in decision-making processes in the lead up to the next phase of fundingFootnote 37. A couple of CIHR Senior Management key informants emphasized how CCNA demonstrated adaptability in responding to identified challenges. However, there remained uncertainty about how Associate Directors would be included in the decision-making going forward.

There were mixed views about the extent to which the DRS supports research collaboration.

The majority of partner and researcher key informants agreed that the DRS brought together stakeholders from different disciplines to foster interdisciplinary collaborations – particularly through the CCNA. Approximately half of interviewees felt that research partnerships had improved through an increase in funding and involvement in CCNA. Several key informants identified improvements around ease of interaction and commitment between partners, the frequency and quality of information provided to knowledge users and partners, and organizational support, particularly as a result of DRS meetings and events such as the Partners Forum and annual CCNA meetings. Together, these findings indicate that the DRS supports the development of research collaborations. Furthermore, both components of the DRS were successful in leveraging contributions from CIHR partners. As shown in Figure 9, the CCNA had a total of $8.1 million in contributions from CIHR partners such as the ASC and a provincial research foundation. Whereas, the international component has leveraged a total of $23 million from international research organizations via 17 funding opportunities.

There were mixed views regarding the extent to which the DRS has developed a sustainable model for the formation of collaboration and partnerships as a longer-term indicator of success. For instance, some key informants conveyed that it was too early to tell if the DRS has supported the formation of sustainable partnerships and whether the partnerships have had an impact on the health of Canadians.

The CIHR Oversight Committee identified challenges around CCNA's limited success in generating successful and sustainable partnerships with the wider stakeholder community, particularly industry partnersFootnote 38. Among their concerns were CCNA's inability to generate new partnerships and mitigate the withdrawal of existing partners (e.g., Sanofi, a French multinational pharmaceutical company)Footnote 39 due to low allocation of resources and underutilization of advisory bodies to support partnership developmentFootnote 40. In line with these concerns, program data from a 2017 Partners' Forum demonstrated a significant decrease in partner participation (down from 31 partners attending in 2015 to 19 in 2017 or nearly 40%), with partner feedback highlighting a lack of engagement during both planning and the event itself as well as a limited understanding of the role of partners in the CCNA and its progress overall.

DRS was delivered in a cost-efficient manner since 2012-13.

The percentage of program administrative costs to total program expenditures speaks to how efficiently the DRS program is being run. In the case of DRS, due to some limitations in data availability, particularly in the early years of the initiative, the cost efficiency analysis only compares CIHR direct administrative costsFootnote 41 of DRS against total program investments for the fiscal years 2009-10 through 2017-18Footnote 42. For the period of the evaluation, the ratio of DRS total direct administrative costs to the DRS program expenditures was 6.3%, which is higher than the ratio of CIHR total operating costs to the CIHR total program expenditures of 5.1%, for the same period under review.

The percentage for DRS is characterized by a ramp up period in the first years whereby administrative costs were high and grants and awards expenditures were low. In fact, the percentage of total CIHR direct administrative costs to total program expenditures peaked at 83.7% in 2009-10. This was largely attributable to the positions of an executive director and assistant director for the international component at CIHR from 2009-10 to 2011-12. Both positions were full-time from 2009-10 to 2010-11, before becoming part-time in 2011-12. In 2012-13, the responsibilities associated to the position of the executive director and the assistant director for the international component were transferred to the Institute of AgingFootnote 43. Between 2012-13 and 2017-18, the ratio of DRS total direct administrative costs to the DRS program expenditures remained was 3.7%, representing an average direct operating ratio of 3.7 cents for each dollar of grant funds awarded, while the ratio of CIHR total operating costs to the CIHR total program expenditures was 4.9%. Figure 10 includes the comprehensive cost-efficiency data.

Improvements can be made to the design and delivery of the DRS.

Based on key informants and documentation, some improvements can be made to the design and delivery of the DRS. As outlined below, there is an opportunity to address limitations around the insufficient level of funding and lack of strategic funding inhibiting CCNA's ability to carry out its objectives; there is a need to enhance communication, clarity, and transparency around the CCNA and finally, there is an opportunity to establish structures and processes to support more stakeholder engagement and improve partnership and collaboration.

There is an opportunity to address the level and allocation of CCNA funding to better carry out its objectives.

Findings from an internal evaluation conducted by CCNA report that Research Executive Committee members, theme leaders and team leaders expressed concerns around insufficient funding to achieve all of CCNA's objectives. These concerns stem from initial discussions prior to the funding opportunity which lead to a perception that CCNA was to receive double the amount of fundingFootnote 44 to what was made available in the funding opportunity. These unrealistic expectations resulted in disappointment when this of level of funding did not materialize. It is important to note that the peer review of CCNA's application and budget was deemed feasible with respect to the proposed objectives and total funding amount available.

Although the CCNA's level of funding was deemed appropriate, in its 2018 report, the CCNA Oversight Committee did identified a projected shortfall of almost $150,000 at the end of Phase 1. This raised concerns by CCNA Leadership, funders and the CCNA Oversight Committee regarding the sustainability of CCNA, highlighting the need to consider "alternative avenues through which funding can be sought."Footnote 45 Many key informants cited lack of funding and non-strategic allocation of funding as reasons for gaps in CCNA's design and implementation. For instance, some felt that funding was not strategically allocated across CCNA themes, which resulted in a mismatch between resources and expertise. Program data support the fact that limited funding allocated to CCNA inhibited the advancement of research. As such, "insufficient funding" was cited as the primary roadblock to collaboration in order to fully achieve teams' original goals by CCNA researchersFootnote 46. CCNA also "noted that as synergy and interactions occur and new ideas are pursued, it is becoming obvious that [CCNA is] hampered by a level of funding which is clearly not commensurate with [its] capabilities. Simply put, the urgent need to find the causes of the neurodegenerative disorders requires a higher level of funding and that would be translated into more rapid progress in each team. Searching for additional funding sources will become a major focus of the CCNA Administration in the coming years."Footnote 47

There is a need to enhance communication, clarity, and transparency around the CCNA.

Several key informants identified a need for improved communication and enhanced transparency with partners regarding the CCNA, particularly around its objectives, structures, timelines, and performance measurement. Some noted that higher quality, more frequent communication processes – for instance, regular conferences, meetings, face-to-face discussions, and quarterly status reports – could increase transparency and improve partners' experiences with the DRS. The need for added clarity in these areas was also reported by CCNA's Oversight Committee, which requested clarification on aspects of the CCNA model, including governance and decision making criteria (discussed above) as well as the reporting requirement of the "internal synergies within CCNA to be able to demonstrate the value add of the consortium model that could not take place otherwise."Footnote 48

There is an opportunity for CCNA to improve stakeholder engagement, partnership and collaboration.

In addition to enhancing communication with stakeholders, some key informants also identified a need to engage knowledge users and partners earlier and more often to facilitate interaction between stakeholders and improve partnerships and collaboration. This area for improvement was echoed by the CCNA's Oversight Committee, which felt that advisory bodies had "ample, untapped support and guidance" to offer the CCNA and encouraged the CCNA to further engage and leverage its advisory bodies and partners through more frequent meetings and interactionsFootnote 49. One related suggestion from the Oversight Committee was to establish a role at CCNA to devote more attention to partnership development and alleviate burden on researchersFootnote 50.

Performance

Key Findings:

DRS has contributed to increasing research capacity in the field of dementia by funding researchers and supporting trainees.

The analysis of the funding trajectory of DRS researchers demonstrates that both the national and the international components were successful in increasing research capacity in the field of dementia. The researchers involved in CCNA received a total of $48 million in funding from CIHR in the field of dementia research in the five years before the launch of the CCNA (2009-10 to 2013-14). Since the launch of the CCNA, those researchers received a total of $71 million in CIHR dementia funding (an increase of 48%), of which $29 million is directly attributable to the DRS. Funding for the international component revealed a similar trajectory, with international component researchers receiving $17 million in CIHR dementia research in the five years prior to the DRS. Following the launch of the DRS, the international component researchers received $38 million CIHR in dementia funding (an increase of 123%), of which $17 million is directly attributable to the DRS (see Figure 11).

The DRS focused on building capacity in the research enterprise by providing funding and development opportunities for trainees and early career investigators. For instance, program data show that there are 214 CCNA-affiliated trainees, more than 90 of which are either partially or fully funded by the CCNA. The breakdown of funded trainees is relatively equal across training levels (Masters, Doctoral and Postdoctoral) thereby highlighting the diverse opportunities for development. In addition, program data indicates that 17 of 40 CCNA (43%)Footnote 51 Team, Theme and Program lead and co-lead positions are filled by early career investigators which demonstrates that CCNA provides developmental opportunities for future leaders in Canadian dementia research. Consistent with program data, many key informants believe that the DRS has improved research capacity, they pointed to the exposure new researchers can gain through the opportunity, including access to new knowledge and ways of doing research through shared platforms, as well as developing experience and a reputation in the field by publishing research. Also, CCNA in partnership with the ASC has co-funded the CCNA Training and Capacity Building Program. However, there were some issues leading to an unsuccessful allocation of $1 million dollars in funding from the ASC, which contributed to the CCNA not being able to fully support trainees and which required the CCNA to rely on research groups of investigators to support trainees.

The CCNA has supported the formation of inter-institutional, interdisciplinary, and Canada-wide collaborations.

The constitution of CCNA research teams demonstrates the national component's ability to support the formation of inter-institutional, interdisciplinary, and Canada-wide collaborations. Among the 20 teams, program data show that there was an average of four provinces and nine institutions represented by team members. Further, all but one team included members from at least two Canadian regions, and nearly one third of teams had pan-Canadian representation, reflecting the national scope and extent of collaboration of research supported through the CCNA.

All CCNA research teams reported undertaking at least four collaborative activities, demonstrating that collaboration is a common element underpinning all CCNA-supported research. Many key informants also emphasized the size, reach, and synergistic approach of the CCNA network and its support for collaboration and community involvement as key contributors to the ability of the CCNA and DRS to strengthen innovation in Canadian dementia research.

Program data also highlights considerable collaboration between CCNA's research teams. Approximately 60% of teams reported exploring collaborations with other teams either within and/or across CCNA's research themes of Prevention (Theme 1), Treatment (Theme 2), and Quality of Life (Theme 3). Teams in the Treatment theme tended to explore collaborations with teams in other themes, while teams in the Quality of Life theme tended to explore within-theme collaborations.

Sharing data and resources was the most common type of cultivated collaboration between CCNA's teams across the three themes, as shown in Figure 12. This included sharing biological tools (e.g., specialized cell lines), new techniques or protocols, technology or equipment, data for the development of joint publications, and jointly supervised graduate students and postdoctoral fellows. The collaboration between CCNA's teams in the Treatment theme also reflected a focus on harmonizing outcome measures, such as through joint development of updated protocols or the validation/adoption of a new technique by multiple teams in order to allow comparisons between various experiments.

The DRS has increased capacity for knowledge translation and dissemination through peer-reviewed publications, presentations, and new methodologies.

Many key informants reported that the DRS had a positive impact on knowledge translation. The most commonly cited examples included the number of scientific journal publications and new tools (e.g., common data sharing platforms) generated through the DRS. More than two-thirds of researchers who were interviewed identified publications and presentations as a main academic achievement resulting from their project, while more than one-fifth identified new methodologies as a main achievement.

The interview findings are supported by research outputs documented on CCNA's website. As of October 2019, the CCNA research teams reported producing 123 scientific publications. Of these publications, 89% (109 publications) acknowledged CIHR and/or CCNA. Of these 109 publications, 61% (66 publications) acknowledged both CIHR and CCNA, 22% (24 publications) acknowledged only CCNA, and 17% (19 publications) acknowledged only CIHR (see Figure 13). Given that the majority (89%) of CCNA publications acknowledge CIHR and/or CCNA, the findings of the acknowledgement analysis appear to validate the accuracy of the publications listed on CCNA's website.

In addition to the acknowledgement analysis, a citation analysis of CCNA's publications was conducted using Clarivate's Web of Science Database. Of CCNA's 123 publications, 112 were available on Web of Science (91% of the total). The bibliometric analysis also revealed that a significant portion of this research has since been cited by other researchers. For instance, of the 112 scientific research publications resulting from the CCNA that were available on Web of Science, 66% were cited at least six times and five publications met the criteria for "highly cited papers"Footnote 52 on Web of Science.

Figure 14 shows the full distribution of the number of citations of CCNA-funded research which illustrates that other researchers have considered the implications of, and/or built upon, the CCNA research in their own research.

All 20 CCNA research teams contributed to knowledge translation and dissemination via publications, presentations and knowledge translation activities. CCNA-supported research has been disseminated through knowledge products, such as: public commentary (including publicity, pamphlets, guides, lay language summaries of work, invited talks - other than at scientific conferences, or interviews), interaction with media (e.g., interviews, podcasts), and collaboration with CCNA's partner organizations. In fact, over 600 knowledge translation products (e.g., presentations, academic and non-academic publications, and knowledge translation activities) have been developed on CCNA-supported research.

Both program documents and key informants indicated that it is too early to assess the impact of DRS supported peer-reviewed publications beyond academia. However, a previous analysis of CIHR-supported dementia research revealed that many peer-reviewed publications demonstrated evidence of knowledge translation and impact by informing downstream policy documents and patents, as measured by CIHR's Observable Influence Beyond Academia (OIBA) approach. Based on these findings it is expected that DRS-supported publications would show similar levels of knowledge translation and impact compared to CIHR-supported dementia research publications overall. As such, it would be important that CIHR be in a position to continue tracking knowledge translation activities and impact beyond academia of DRS-funded research.

Key informants highlighted a number of key impacts of DRS funded research including increased participation of knowledge users (particularly among clinicians) in research activities as well as the moderate reach of research findings into the public sphere through media coverage and incorporation in the development of policy and practice (e.g., treatment guidelines).

The CCNA has improved consideration of sex and gender research questions and supported Indigenous-specific research.

As per program data and key informants, the CCNA, notably through its Women, Sex, Gender, and Dementia program improved consideration of sex and gender research questions as well as other EDI factors, although there was some variation in the nature and extent of consideration each received. The majority of key informants believed that the level of priority communicated by the Institute of Gender and Health's involvement in the DRS, as well as the CCNA's theme dedicated to the topic, improved consideration of sex and gender in dementia research. In fact, they considered this to be one of the very important successes of the CCNA. Program data also lend further support to this assertion, showing that 90% of research teams reported factoring issues of sex and gender into their experiments and/or analyses.

Despite these indications of improvement, many CCNA teams – particularly those performing basic science research – appeared to consider sex and gender factors in follow-up analyses rather than during the development of their research questions. Further, some key informants exhibited skepticism that the DRS itself has contributed to improved consideration of sex and gender research questions, believing that this improvement would still have been possible through separately funded research projects not contained within the umbrellas of the CCNA or DRS.

Support for Indigenous-specific research is another way in which the CCNA considers EDI factors in dementia research. Such consideration is evident in the national component, under which one of CCNA's research teams has a specific focus on issues in dementia care for rural and Indigenous populations and four out of seven teams within the Quality of Life research theme reported conducting dementia research related to Indigenous health. This included one research team that formed direct partnerships with Indigenous organizations (e.g., Chiefs of Ontario).

The DRS has contributed to increased international recognition of Canada as a collaborative dementia research hub.

The success of efforts to showcase and increase the international profile of collaborative Canadian dementia research is exemplified by Canadian participation in global collaborative research initiatives and the formation of new international research collaborations.

More than half of key informants indicated that Canada was recognized as a hub for international research collaboration in neurodegeneration in aging, characterizing this status as being at least in part due to support provided through the DRS. Key informants reported that the Scientific Directors of CIHR's Institutes have internationally recognized knowledge and expertise and are well embedded in the research community, contributing to the recognition of Canada as a collaborative dementia research hub. As per program data, the IA, as the lead Institute of the DRS, took several key actions which have contributed to increasing the international recognition of Canada as a collaborative dementia research hub. The IA Scientific Director for period from 2011 to 2019 sat on many international advisory and management boards (e.g., for the EU JPND Research, Global Council on Brain Health), contributed to key international initiatives such as the WHO's Global Dementia Observatory Reference Guide and Canada's contribution to the Global Action Plan on Dementia. This type of participation and leadership in global dementia research enhances Canada's ability to align with global collaborative research plans and priorities and be recognized as a global research leader. Other DRS initiatives, such as hosting a "Canadian Pavilion" at international dementia research events, further support Canada's acknowledgement as a collaborative dementia research hub by promoting visibility of Canadian research and researchers and providing opportunities for relationship building with potential international collaborators.

Beyond raising Canada's research profile on the international stage, the DRS also facilitated the formation of new international collaborations. For instance, CCNA teams have established 42 new international collaborations since the initiative was established, with an average of approximately two new international collaborations per CCNA team. Some key informants added that these types of collaborative research projects of international scope further demonstrate Canada's appeal as an international research partner and its recognition as a hub for dementia research.

Conclusions and Recommendations

Relevance

The social and economic burden of dementia is increasing in Canada and around the world, creating a strong need for additional research on prevention, treatment, and quality of life improvements for those living with the disease as well as their caregivers. The majority of key informants agreed that the DRS meets this need to at least a moderate extent and generally complements other dementia research funding opportunities and supports within the Canadian and international research landscapes.

The evaluation also concludes that the federal government has a clear role in providing support for researchers to engage in innovative national and international collaboration and leadership initiatives that support dementia research – a demonstrated federal priority area. Specifically, there are a number of federal agreements, legislation, and strategy documents from recent years that underscore the federal government's clear and ongoing priority and role related to supporting dementia research, including agreements made at the 2013 G8 Dementia Summit, the National Dementia Strategy released in 2019, and action areas from the WHO Global action plan on the public health response to dementia 2017-2025, among others. CIHR's role in providing support also aligns directly with its stated objective to support scientific excellence in health and health care systems research as outlined in the CIHR Act (2000) and strategic plans.

However, researchers, partners, CIHR Senior and Program Management, and the Senate Standing Committee on Social Affairs, Science and Technology all voiced concerns that the current level of funding of the DRS is insufficient to meet federal priorities and policy commitments.

Design and Delivery

Both the national and international components of the DRS share common elements such as the research themes and overall objectives. However, there are also some challenges related to the integration of the components, which would allow for the strategy to be more cohesive and synergistic. More specifically, the two components of DRS are generally presented as distinct entities, with the common element being the overall theme of dementia research. The oversight structure differs between both components and there is no oversight structure that oversees the DRS as a whole. There is also a lack of documentation regarding the international component and the relationship between the two components.

The international component supports the DRS' overall objectives by facilitating Canada's participation in key international research partnerships and establishing linkages between the international research community and industry partners. As the central initiative of the national component, the CCNA is as an innovative model for supporting collaborative dementia research partnerships within Canada. CCNA structures and processes such as cross-cutting activities and ongoing communication between research teams also increase engagement, facilitate knowledge exchange, and build capacity in the dementia research enterprise.

Despite these effective design elements, the evaluation identifies several challenges and areas for improvement with the DRS:

Performance

The evaluation findings indicate that the DRS is achieving or making progress toward the following expected outcomes:

Program data and key informant feedback also highlight that the DRS, specifically through CCNA's overall structure, is improving consideration of EDI factors in dementia research and conducting research aligned with national priorities, including sex and gender, indigenous populations and rural and remote communities. In fact, increased consideration of sex and gender research questions was viewed as a very important success of the CCNA, supported by the level of priority communicated by the IGH involvement in the DRS as well as the CCNA's cross-cutting theme dedicated to the topic. Support for research specific to Indigenous, rural and remote populations is also incorporated into the CCNA, with a team dedicated exclusively to supporting rural capacity building in dementia care.

While it is too early to see the full reach and impact of DRS-supported research beyond academia, DRS knowledge products are being disseminated and applied beyond academia at a higher rate than CIHR supported knowledge products as a whole. Further, ongoing Canadian participation in global initiatives, supported by the international component, showcase Canadian researchers as leaders on the world stage, facilitate the formation of new international partnerships, and enhance Canada's image as a hub for international research collaboration.

Recommendations

The evaluation makes three recommendations to improve the performance of the program to achieve its expected results.

Recommendation 1:

CIHR should continue to invest strategically in the DRS and re-assess the nature and extent of funding to meet its expected outcomes as well as the needs of the National Dementia Strategy.

Recommendation 2:

CIHR should ensure that the appropriate objectives, resources and structures are in place to better integrate the national and international components of the DRS to work cohesively and synergistically to achieve its overall objectives.

Recommendation 3:

CIHR should examine approaches to enhance communication with, and engagement of, stakeholders, particularly DRS partners, knowledge users, people living with dementia, and their caregivers.

Appendices

Appendix A: Report Figures

Figure 1: CIHR's Annual Investment in Dementia Research, 2009-11 – 2017-18

Figure 1 long description
2000-2001 2001-2002 2002-2003 2003-2004 2004-2005 2005-2006 2006-2007 2007-2008 2008-2009 2009-2010 2010-2011 2011-2012 2012-2013 2013-2014 2014-2015 2015-2016 2016-2017 2017-2018
DRS               $7,500 $42,500 $80,000 $645,868 $1,444,451 $2,795,174 $3,519,832 $7,383,277 $7,354,361 $7,285,000 $7,231,279
Dementia $8,985,859 $13,139,438 $14,610,287 $15,545,782 $17,591,398 $20,757,596 $22,627,021 $22,148,262 $23,758,858 $24,326,286 $29,241,534 $34,110,483 $36,160,698 $34,300,254 $34,007,641 $32,480,827 $32,314,978 $33,671,960
Investment total $8,985,859 $13,139,438 $14,610,287 $15,545,782 $17,591,398 $20,757,596 $22,627,021 $22,155,762 $23,801,358 $24,406,286 $29,887,402 $35,554,934 $38,955,872 $37,820,086 $41,390,918 $39,835,188 $39,599,978 $40,903,239

Source: CIHR EIS data.

Figure 2: Percentage of DRS Investments Compared to All CIHR Dementia Related Investments

Figure 2 long description
2009-2010 2010-2011 2011-2012 2012-2013 2013-2014 2014-2015 2015-2016 2016-2017 2017-2018
Percentage 0.3% 2.2% 4.1% 7.2% 9.3% 17.8% 18.5% 18.4% 17.7%

Source: CIHR EIS data.

Figure 3: Number of Canadian Publications in Each Priority Areas of DRS

Figure 3 long description
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2000-2016
Prevention and Control of Dementia 14 22 10 15 30 14 25 39 18 26 25 32 25 41 28 27 51 442
Therapeutics of Dementia 4 13 8 8 11 15 14 20 17 10 15 17 19 25 32 24 35 287
Quality of Life with Dementia 2 4 2 2 5 5 5 4 4 3 3 7 6 4 8 7 15 86

Source: Bibliometric analysis conducted by OST.

Figure 4: Number of Publication in All 3 Priority Areas of DRS by Countries

Figure 4 long description
Country 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
United States 124 131 158 182 184 227 250 238 264 284 330 269 278 322 331 329 256
United Kingdom 31 38 36 52 48 44 60 63 61 75 99 86 101 115 175 131 160
Germany 19 16 29 26 28 33 23 28 33 41 53 51 78 60 75 69 76
Canada 16 33 16 22 37 30 38 54 32 37 37 49 42 60 64 48 80
Australia 8 5 11 17 18 27 16 22 22 42 46 45 45 66 74 80 100
France 17 11 15 33 23 28 33 30 37 40 44 40 51 56 54 62 57
Italy 4 7 6 12 17 27 35 29 33 33 34 47 39 45 45 48 55
China   1 2 4 3 6 8 18 14 24 21 32 45 65 66 98 89
Spain 5 7 7 5 12 14 15 14 17 19 30 34 37 51 60 49 54
Japan 12 9 17 12 5 23 17 26 27 45 32 29 33 33 26 47 35

Source: Bibliometric analysis conducted by OST.

Figure 5: Specialized Index (SI) of Publications in All 3 Priority Areas of DRS by Countries

Figure 5 long description
Country 2000-2016
China 0.41
Japan 0.61
Germany 0.94
France 1.13
Italy 1.15
Spain 1.15
United States 1.37
Canada 1.49
United Kingdom 1.65
Australia 1.84

Source: Bibliometric analysis conducted by OST.

Figure 6: Average of Relative Impact Factors (ARIF) of Publications in All 3 Priority Areas of DRS by Countries

Figure 6 long description
Country 2000-2016
Japan 0.94
Germany 0.96
Australia 1.04
Spain 1.05
China 1.06
France 1.13
Italy 1.15
Canada 1.15
United Kingdom 1.16
United States 1.31

Source: Bibliometric analysis conducted by OST.

Figure 7: Average of Relative Citations (ARC) of Publications in All 3 Priority Areas of DRS by Countries

Figure 7 long description
Country 2000-2016
Australia 1.35
Japan 1.37
China 1.45
Italy 1.48
Germany 1.49
Canada 1.54
Spain 1.59
United States 1.66
France 1.68
United Kingdom 1.70

Source: Bibliometric analysis conducted by OST.

Figure 8: CCNA Governance Structure (2015)

Figure 8 long description

CCNA has a well-defined governance structure which consists of three core components: Research Executive Committee, Strategic Advisory Council, and International Scientific Advisory Board. External oversight of the CCNA is provided by a CIHR CCNA Oversight Committee, which is comprised of three CIHR representatives (Director General and Scientific Director leads) and six partnership representatives involved in CCNA. The CCNA Oversight Committee raised concerns regarding the under-utilization of the International Scientific Advisory Board and the Strategic Advisory Committee. In the case of the Strategic Advisory Committee, it is reported that they have only met together once, although members are consulted on an ad hoc basis.

Source: Canadian Consortium for Neurodegeneration in Aging (CCNA) Response to Oversight Committee Report 5, 2015, CCNA. Legend: Biological Sample Access Committee (BSAC); COMPASS-ND Protocol Implementation Team (PIT); Conference Program Planning Committee (CPPC); Database Development Committee (DDC); Diagnostic Adjudication Committee (DAC); Ethical, Legal and Social Issues (ELSI); Knowledge Transfer and Exchange (KTE); Membership Committee (MC); Publication and Data Access Committee (PDAC); Research Executive Committee (REC); Training and Capacity Building (TCB); Women, Gender, Sex and Dementia (WGSD).

Figure 9: Partnership Contributions

Figure 9 long description
2010-2011 2011-2012 2012-2013 2013-2014 2014-2015 2015-2016 2016-2017 2017-2018 Total
International component $916,751 $2,107,596 $3,324,229 $2,347,406 $2,814,223 $3,790,790 $3,641,132 $3,581,727 $22,523,854
National component         $1,374,333 $2,712,496 $2,269,579 $1,764,580 $8,120,988

Source: CIHR EIS data.

Figure 10: Cost-Efficiency Table

2009-2010 2010-2011 2011-2012 2012-2013 2013-2014 2014-2015 2015-2016 2016-2017 2017-2018 Total
Non Salary Direct Administrative Costs $0 $60,162 $48,412 $26,374 $78,498 $43,072 $11,301 $15,288 $7,594 $290,701
Salary Direct Administrative Costs $411,635 $417,809 $222,787 $112,523 $243,712 $148,277 $303,822 $187,543 $192,942 $2,241,050
Total Direct Administrative Costs $411,635 $477,971 $271,199 $138,897 $322,210 $191,349 $315,123 $202,831 $200,536 $2,531,751
Total grants expenditures $80,000 $645,868 $1,444,451 $2,795,174 $3,519,832 $7,383,277 $7,354,361 $7,285,000 $7,231,279 $37,739,242
Total program expenditures $491,635 $1,123,839 $1,715,650 $2,934,071 $3,842,042 $7,574,626 $7,669,484 $7,487,831 $7,431,815 $40,270,993
DRS Direct Costs as % of Total Program Expenditures 83.70% 42.50% 15.8% 4.70% 8.40% 2.50% 4.10% 2.70% 2.70% 6.30%
CIHR Operating Costs as % of CIHR Total Expenditures 5.20% 5.50% 5.60% 5.70% 4.80% 4.80% 4.70% 4.60% 4.90% 5.10%

Figure 11: DRS Researchers Funding Trajectory

Figure 11 long description
National
  5 years before During and after DRS
Dementia $47,520,957 $70,445,124
Other funding $75,022,161 $53,385,724
Overall funding $122,543,118 $123,830,848
International
  5 years before During and after DRS
Dementia $17,064,504 $38,093,935
Other funding $45,746,035 $22,420,156
Overall funding $62,810,539 $60,514,091

Source: CIHR EIS data.

Figure 12: Collaborations Developed by CCNA Research Teams, by Theme

Figure 12 long description
Theme 1 - Prevention Theme 2 - Treatment Theme 3 - Quality of life
Sharing data/resources 5 12 12
Harmonizing outcome measures 3 7 3
New research inquiries 5 3 5
Joint funding proposal 4 3 5
Total 17 25 25

Source: Dementia Research Strategy (DRS) Document Review Report, 2018, CIHR.

Figure 13: Results of Acknowledgement Analysis of CCNA publications (as of October 2019)

  Number of Publications
Publications listed on CCNA website 123
Publications acknowledging CCNA or CIHR 109
Publications acknowledging CIHR supporting CCNA 40
Publications acknowledging CIHR and CCNA separately 26
Publications acknowledging only CCNA 24
Publications acknowledging only CIHR 19

Source: CCNA website; Web of Science.

Figure 14: Integration of CCNA Research into Other Academic Research

Figure 14 long description
Number of times cited Number of publications
(n=112)
Highly cited publications
(n=5)
5 and less 38  
Between 6 and 10 33  
Between 11 and 15 14  
Between 16 and 20 13  
Between 21 and 40 7 1
41 and more 7 4

Source: Bibliometric Analysis conducted by CIHR Evaluation Unit.

Appendix B: Methodology

This section presents additional details about the methodology used to conduct the evaluation of the DRS. The table below provides the evaluation matrix, which shows the issues and questions to be addressed by the evaluation, as well as the indicators and data sources used to inform findings for each.

Table 2: Evaluation Matrix

Indicator Document / Data Review Key Informant Interviews Bibliometric Analysis
Evaluation Question #1: To what extent is DRS meeting the need to support World class research on dementia? (Relevance)
Sub-Question #1.1: To what extent have DRS components been aligned with federal government and CIHR priorities?
1.1.1 Extent to which objectives/expected outcomes of DRS have been aligned with federal government and CIHR overall priorities Applicable Applicable Applicable
Sub-Question #1.2: To what extent have DRS components been aligned with federal roles and responsibilities?
1.2.1 Identified role for the federal government in supporting neurodegeneration in aging research Applicable Applicable  
Sub-Question #1.3: To what extent has DRS continued to address a demonstrated need?
1.3.1 Extent to which objectives/expected outcomes of DRS have been aligned with the needs of the Canadian and International Scientific landscape Applicable Applicable Applicable
1.3.2 Extent of duplication/overlap of the objectives of the DRS with other federal granting agencies Applicable Applicable  
Evaluation Question #2: To what extent have the design and implementation of DRS been effective and efficient? (Design & Delivery)
Sub-Question #2.1: To what extent has DRS been an appropriate approach to supporting research on neurodegeneration in aging?
2.1.1 Assessment of the effectiveness of the structure of the consortium model (i.e. Assessment of governance structure, type, number and focus of teams, themes and platforms, management of funding, coordination and communication processes) in meeting objectives (strengthening and synergizing Canadian innovative and collaborative research and becoming a Canadian research hub) Applicable Applicable  
2.1.2 To what extent the structure of DRS (i.e. having two components: national and international) facilitated achievement of objectives Applicable Applicable  
2.1.3 To what extent have DRS design and delivery enabled the establishment of sustainable and successful partnership with the stakeholder community (national and international) (e.g., peer review, matching funds, knowledge translation) Applicable Applicable  
Sub-Question #2.2: To what extent are the DRS components being delivered in a cost-efficient manner?
2.2.1 Ratio of operating expenditures to grant funds awarded (planned vs. actual) Applicable    
2.2.2 Leverage ratio of DRS funded grants and awards to partner funds (planned vs. actual) (For CCNA grant and for ICRSAD funded grants) Applicable    
2.2.3 Assessment of resource utilization by CCNA Applicable    
Sub-Question #2.3: What improvements, if any, can be made to design and delivery of DRS?
2.3.1 Evidence of potential improvements to the design/ implementation of DRS international component and CCNA (e.g., possible changes to the number and types of teams, platforms and cross-cutting themes, research, possible changes to the governance system, funding types and amount) Applicable Applicable  
2.3.2 Best practices/lessons learned from DRS components implemented to date Applicable Applicable  
2.3.3 Factors that have facilitated or inhibited the implementation of DRS components Applicable Applicable  
2.3.4 Assessment of alternative program designs and delivery models to DRS current structure and delivery that are more effective Applicable    
Evaluation Question #3: To what extent has DRS achieved expected objectives? (Performance)
Sub-Question #3.1: To what extent has DRS made progress toward the achievement of expected immediate outcomes?
3.1.1 Evidence outcomes from funded research have been disseminated and or integrated into other academic research Applicable Applicable  
3.1.2 Evidence of capacity building in the research enterprise (number of trainees under CCNA teams, number of trainees under ICRSAD grants and awards, number and value of training awards provided, researchers’ perspective of increased capacity) Applicable Applicable  
3.1.3 Evidence that CCNA researchers have adopted an innovative and collaborative research Applicable Applicable  
3.1.4 Evidence sex and gender research questions have been studied across CCNA teams and platforms Applicable Applicable  
Sub-Question #3.2: To what extent has the DRS made progress toward the achievement of expected intermediate outcomes?
3.2.1 Evidence findings from funded research are applied outside academia (e.g., translated into policies, made changes to health systems, etc.) Applicable Applicable Applicable
3.2.2 Evidence Canada is recognized as a hub for international research collaborations in neurodegeneration in aging Applicable Applicable  
3.2.3 Evidence that Canada is internationally competitive in the research for neurodegeneration in aging   Applicable  
3.2.4 Evidence of strengthening collaborative and innovative research in Canada due to CCNA Applicable Applicable  

(A)pplicable.

Source: Adapted from Dementia Research Strategy (DRS) Evaluation Design Report (EDR), 2017, CIHR.

As seen in the evaluation matrix, the evaluation of the DRS employed multiple lines of evidence, including a document and data review, key informant interviews, and bibliometric analysis. The following sub-sections outline key details about data collection for each line of evidence, followed by a description of how evaluation findings were triangulated and reported.

Document and Data Review

CIHR's Evaluation Unit reviewed DRS, CIHR, Government of Canada, and international documents. For example, the document review included relevant legislation, government documents such as departmental plans, CCNA progress and oversight reports, and related strategy and evaluation reports. The main sources of data included CIHR's EIS and RRS. Information provided context around the dementia research landscape and contributed to assessment of the initiative's progress and performance, including toward expected outcomes. In addition, the review was also used to assess annual progress reporting and to inform planning – for instance, to support CIHR decision-making in the design of CCNA's Phase 2 Funding Opportunity, to create targeted key informant interview guides, and to develop intermediate outcome indicators associated with strengthening collaborative and innovative research in Canada.

Key Informant Interviews

CIHR Evaluation Unit worked in collaborations Prairie Research Associates to conduct 44 key informant interviews with Program and Senior Management, DRS partners, and members of the research community using interview guides prepared by CIHR's Evaluation Unit. Interviews were 30 to 60 minutes long, conducted over the telephone, and digitally recorded and transcribed with participants' consent. The information gathered through the interviews was used to provide an in-depth understanding of views, perceptions, and opinions of the DRS and CCNA.

The following table outlines the number and type of respondents who participated in key informant interviews. Interviewees were selected for their implication in the international or national component.

Table 3: Number of Key Informants by Respondent Group

Respondent Group Number of Respondents (n)
CIHR Program Management 4
CIHR Senior Management 4
Scientific Directors 1
Partners National 9
International 4
Researchers National 8
International 10
Both 4
Total 44

Source: Dementia Research Strategy (DRS) Key Informant Interview Technical Report, 2019.

Bibliometric Analysis

CIHR contracted Observatoire des sciences et des technologies (OST) to conduct a bibliometric analysis of scientific papers in dementia overall as well as in three priority areas targeted by DRS (e.g., prevention and control, quality of life, and therapeutics) for Canada and nine other leading research countries, including Australia, China, France, Germany, Italy, Japan, Spain, UK, and USA. The analysis included papers indexed in World of Science (Clarivate) and PubMed (US National Library of Medicine) that were identified using relevant MeSH terms. In total, 13,818 relevant papers were identified in the three DRS priority areas and 73,744 papers were identified in dementia research overall. Information on the following was extracted for each:

Results were summarized quantitatively by country for each indicator and were used to inform findings about the relevance of dementia research to the international scientific community, and by extension, the DRS.

In addition, CIHR's RIU also conducted a supplementary analysis of the 123 publications listed on the CCNA website to estimate researcher compliance with the CIHR requirement to acknowledge CIHR as a source of support. Additionally, CIHR collected and identified all relevant DRS mandate related "Knowledge products", identified through Web of Science that were published between 2008-2010Footnote 53 and included author acknowledgement of CIHR, and ran a matching process with OIBA database which is maintained by CIHR's RIU. Results were used to inform findings about the DRS' performance with respect to achievement of expected intermediate outcomes.

Limitations and mitigations

The following describes the key limitations associated with the evaluation, which were mitigated through the tri-angulation of results across data sources as well as consideration of the relative strengths and weaknesses of each data source. The mitigation strategies employed throughout the evaluation help ensure that the evaluation results can be used with confidence to inform program decision making.

Table 4: Limitations and mitigation strategies

Limitations Mitigation Strategies
CCNA is the centerpiece of DRS
  • The national component of DRS had a larger budget, network-type structure and formal reporting requirements (e.g., annual reports) compared to the international component, which led to it being unintentionally overrepresented in documentation.
  • The Evaluation Unit sought additional information, such as supplementary documents as well as financial and grant data to bring to light all the initiatives of DRS.
Limitations of individual data sources
  • Each individual data source has its own strengths (e.g., large samples, readily available) and limitations (e.g., key informant interview bias, outdated documentation).
  • Findings were triangulated from multiple lines of evidence to enable cross-validation. In addition, the relative strengths and limitations of each data source were considered, such that greater priority was placed on data considered to be more reliable and/or more relevant to a given evaluation question.
Small sample sizes
  • The number of interviewees in some key informant subgroups were small, which may not have allowed for saturation of findings within groups.
  • Responses from very small groups were merged based on the nature and affiliation of respondents and findings were triangulated.
Identified data gaps
  • In some cases, there were unintended gaps in the data. For example, limited documentation was available for the some of the programs within the international component of DRS.
  • The Evaluation Unit sought additional information (e.g., areas for CIHR to conduct additional analysis or consult with stakeholders) to ensure there was sufficient evidence upon which to base findings.
Time lapse since some data collection activities
  • A considerable amount of time has passed since some of the data collection activities were conducted which could contribute to outdated findings.
  • Data were analyzed and reported with a view to identifying findings and recommendations that continue to be relevant. For instance, foundational documents published since the end of the study period (e.g., the National Dementia Strategy published in June 2019) were considered when formulating evaluation conclusions and recommendations.

Analysis & Reporting

Following data collection, CIHR Evaluation Unit worked in collaboration with Ference & Company Consulting Ltd. to synthesize data, triangulate findings, and develop the final evaluation report.

Findings from each line of evidence were organized by questions and indicators using a structured result matrix to ensure all findings were based on sufficient evidence, identify areas for any additional data collection or analysis to address evidence gaps, and ultimately serve as a clear roadmap as to how findings were derived. In developing conclusions related to each evaluation question and issue, findings were cross-validated across lines of evidence and the relative strength of each line of evidence was considered, such that greater priority was placed on data considered to be more reliable and/or more relevant to a given evaluation question.

Following data collection, CIHR Evaluation Unit worked in collaboration with Ference & Company Consulting Ltd. to synthesize the data, triangulate findings, and develop the final evaluation report which includes the evaluation findings and the conclusions and recommendations. CIHR's Evaluation Unit presented the preliminary findings and the recommendations to the Scientific Director of IA and CIHR President.

Date modified: