Evaluation of the Institute of Musculoskeletal Health and Arthritis (IMHA)

Report of IMHA Evaluation Panel

February 2017

At the Canadian Institutes of Health Research (CIHR), we know that research has the power to change lives. As Canada's health research investment agency, we collaborate with partners and researchers to support the discoveries and innovations that improve our health and strengthen our health care system.

Canadian Institutes of Health Research
160 Elgin Street, 9th Floor
Address Locator 4809A
Ottawa, Ontario K1A 0W9

This publication was produced by the Canadian Institutes of Health Research. The views expressed herein do not necessarily reflect those of the Canadian Institutes of Health Research.

IMHA Evaluation Panel: Dr. John Matyas (Chair), Dr. Claire Bombardier, Ms. Janet Yale, and Dr. Paul Allison

Acknowledgements

Thanks to all participants in this evaluation – survey respondents, interview participants – and to Goss Gilroy Inc. for data collection and analysis.

Additional thanks to the IMHA CIHR Evaluation Team: David Peckham, Michael Goodyer, Abigail Forson and Doaa Saddek, and Christopher Manuel

And special thanks to Dr. Hani El-Gabalawy, CIHR-IMHA Scientific Director and Ms. Nicole Szajcz-Keller, Assistant Director, IMHA.

For more information and to obtain copies, please contact Evaluation@cihr-irsc.gc.ca

Table of Contents

1. IMHA Evaluation Procedures and Methods
   1. Overview
   2. IMHA Evaluation Team
   3. Methodology
2. Overview of IMHA
   1. Mandate
   2. Financial Considerations
      • Total CIHR Spending on IMHA Mandate
      • IMHA Responsibility Centre Spending
3. Relevance and Performance
   1. Relevance
      • Canadian Context
      • Prevalence and Burden of Disease
      • Not-for-profit and Professional Organizations Aligned with IMHA’s Mandate
      • International Context
      • Relevance to CIHR’s Strategic Priorities
      • Future Relevance of IMHA
   2. Performance
      • Transformational Impacts
      • Catalyst/Convener
      • Operational Effectiveness
4. Conclusions and Recommendations
   1. Should IMHA be amended, merged or terminated?
   2. Should IMHA’s Mandate be Changed?
   3. Observations and Recommendations for CIHR
5. Other Considerations
   1. Observations for the Next Scientific Director
   2. Strategic Considerations
6. References
7. Appendices
   • Appendix 1: IMHA Evaluation Panel Members’ Affiliations and Conflict of Interest Declaration
   • Appendix 2: Overview of Data Sources and Methods
   • Appendix 3: Figures and Tables
     • Figure 1: Total CIHR Spending on IMHA Mandate and Spending on Each of IMHA Focus Research Areas, over Time
     • Figure 2: Value of CIHR Funded Grants & Awards within IMHA Mandate out of Total Value of CIHR Funded Grants & Awards over Time
     • Figure 3: The Value of Annual CIHR Spending on the Mandates of Each of the CIHR Institutes over Time
     • Figure 4: IMHA Mandate Expenditure by Funding Type over Time
     • Figure 5: The Percentage of CIHR Funded Grants and Awards within IMHA Mandate out of The Total Number of CIHR Funded Grants and Awards and The Number of CIHR Funded Grants and Awards in IMHA’s Focus Research Areas
     • Figure 6: IMHA Strategic Spending, over Time
     • Figure 7: Numbers of Direct and Indirect Trainees Supported under IMHA Mandate
     • Figure 8: Investment in Capacity Building Funding out of IMHA’s Strategic Investment over Time
     • Figure 9: Descriptive indicators related to publication sets*
     • Figure 10: Influence beyond academia by publication set*
     • Figure 11: Partners’ Contribution to Funding Opportunities under IMHA Mandate and Leverage Ratio of Partnership: Partners to CIHR Investment in IMHA Mandate

I. IMHA Evaluation Procedures and Methods

A. Overview

This evaluation of the Institute of Musculoskeletal Health and Arthritis (IMHA) is in response to the directional decision by CIHR’s Governing Council (GC), resulting from the Institute Modernization process, to support a regular assessment of CIHR Institutes performance and relevance as outlined in CIHR’s Act. Guided by the Evaluation Framework and Performance Measurement Strategy for CIHR Institutes approved by GC in November 2015, the evaluation assesses the relevance and performance of IMHA, and provides recommendations to inform GC decisions regarding the mandate of IMHA and the appointment of the next Scientific Director. As the current Scientific Director’s first term ends in July 2017 and he will not be seeking renewal, this report makes recommendations for recruiting and informing the incoming Scientific Director. Furthermore, given the significant changes CIHR is undergoing and the changing place of the Institutes in the overall organization, some recommendations are provided in relation to the interaction between IMHA and CIHR and the organization of CIHR to better support the mandate of IMHA and other Institutes.

B. IMHA Evaluation Team

This evaluation was conducted by an Institute Evaluation Panel comprised of experts in IMHA’s mandate areas with the extensive coordination and operational support of CIHR’s Evaluation Unit. An independent evaluation consultant, Goss Gillroy Inc., was engaged to lead primary data collection and synthesis to inform the panel’s deliberations. The names and affiliations of IMHA Evaluation Panel members are listed in Appendix 1.

C. Methodology

The evaluation involved multiple data collection methods and a wide range of data sources were developed and used, including both quantitative (e.g., financial, grants, citations, surveys) and qualitative data (interviews and discussions). The Panel reviewed various IMHA performance metrics (calculated or inferred from the data sources), carefully considered the input received from the stakeholder community, and made recommendations on two core questions: 1) Should IMHA continue as it currently is or be changed, and 2) should the mandate of the Institute be maintained as it currently is or be changed. The methods and data sources are outlined in Appendix 2 and key figures presented in Appendix 3.

It is noteworthy that the evaluation approach did not include the analysis of counterfactual data or interviews with unsuccessful grant applicants. This was not feasible because the assignment of grant applicants to individual, and often multiple, Institutes occurs only for funded applications, and hence the database of IMHA-mandate applicants does not include those who were unsuccessful. It might reasonably be expected that the responses of those who were unsuccessful could differ from award holders (acknowledging that most successful applicants have also been unsuccessful at some point in time). It is also noteworthy that the fluidity of Scientific Director leadership of IMHA and the organizational changes at CIHR over the past five years made the timing of this evaluation particularly challenging.

II. Overview of IMHA

A. Mandate

As one of the CIHR Institutes, the Institute of Musculoskeletal Health and Arthritis (IMHA) was created by the Governing Council (GC) in 2000. The stated mandate of IMHA is to support research and to enhance health in relation to musculoskeletal (MSK), skin and oral health research in Canada. “IMHA is the hub for strategic initiatives in MSK, skin, and oral health research in Canada. IMHA supports ethical and impactful research to enhance active living, mobility and oral health, and to address the wide range of conditions related to bone, joints, muscle, connective tissue, skin, and teeth.”^{Footnote 1}

B. Financial Considerations

To a large extent, evaluating an Institute involves assessing its investments in people, knowledge creation, and knowledge translation to determine how well an Institute has achieved its mandate. It is first necessary to distinguish between CIHR spending on research activities assigned to an Institute’s mandate areas from investigator-initiated and priority-driven grants and awards, from research activities funded out of Institute’s budget (i.e., its Responsibility Centre). It is important to note that the CIHR spending in an Institute’s mandate represents the vast majority of funding and is comprised of mostly investigator-initiated grants and awards assigned to an Institute’s mandate using a post-hoc process. As a result, the amounts, numbers, and success rates reflect more an achievement of the research community under the Institute’s mandate than the individual Institute’s direct research funding. Whereas Institute investments in strategic initiatives and capacity building may reasonably be expected to increase the quality and quantity of investigator-initiated applications in the open competitions, currently there are few tools for mapping such progressions. Hence, this report distinguishes between two different types of spending attributable to IMHA: “CIHR spending on IMHA mandate” and “IMHA spending out of Responsibility Centre (RC).” In evaluating the effectiveness of IMHA to fulfill its mandate, the panel focused primarily on IMHA’s spending out of its RC on grants and awards, and IMHA’s Institute Support Grant, summarized below.

Total CIHR Spending on IMHA Mandate

In this report, “total CIHR spending on IMHA mandate” refers to all grants and awards funded by CIHR in topics relevant to IMHA mandate research areas: bone, arthritis, muscle, rehabilitation, skin, and oral health. This spending includes investigator-initiated (open) and priority-driven (strategic) grants and awards, which could come from any of the 13 CIHR Institutes’ budgets or any other CIHR source. Since its inception in 2000-01 through to 2014-15, CIHR spending on IMHA mandate has risen from $25M to $103M (Figure 1). During this period, total CIHR spending on IMHA mandate was higher for MSK related areas (bone, arthritis, muscle, and rehabilitation) relative to skin and oral health. For example, in 2014-15, bone, arthritis, muscle, and rehabilitation research received $27.5M, $21M, $17.5M and $9M, respectively, while skin research received $10M and oral health research received $4M. As a percentage of total CIHR spending per year from 2000-01 to 2014-15, total annual CIHR spending on IMHA mandate rose slowly from 7% in 2000-01 to 11% in the 2014-15 (Figure 2). In comparison to other Institutes, total IMHA-mandate spending constitutes a relatively modest proportion of total CIHR research investments. Relative to other Institutes in 2015-2016, IMHA spending was less than a third of the Institute of Genetics (IG) or the Institute of Gender and Health (IGH), less than half of the Institutes of Infection and Immunity (III) or Neurosciences, Mental Health, and Addiction (INMHA), approximately the same as the Institute of Aging (IA), with only the Institute of Aboriginal Peoples’ (IAPH) Health having a smaller investment (Figure 3). The breakdown of IMHA-mandate spending by funding mechanism is illustrated in (Figure 4.

Of the total number of funded CIHR grants and awards, the percentage of those assigned to the IMHA mandate increased from 11% in 2000-01 to 19% in 2014-15 (Figure 5).^{Footnote 2}

IMHA Responsibility Centre Spending

From 2004-05 to 2014-15, each CIHR Institute (including IMHA) received an annual research budget of $8.6M. This report is based only on the budget model in effect during this interval. Beginning in 2015-16, due to the changes resulting from Institutes Modernization in 2014-15, half of the Institutes’ research budget ($55.9M per year of the $111.8M, or $4.3M per year per Institute) was invested in CIHR’s Roadmap Accelerator Fund (RAF) to support multi-Institute and multidisciplinary initiatives aligned with CIHR’s research priorities. The remaining half of the Institutes’ research continues to reside within the Institutes’ RC and remains at their individual discretion to direct towards Institute-specific initiatives and funding opportunities.

Strategic spending out of IMHA RC was $3M/year in 2002-03, rose to peak at $11.5M in 2008-09^{Footnote 3} and then steadily declined to $7.9M in 2014-15. In 2014-15, the ranking of IMHA strategic spending from highest to lowest amounts on IMHA focus areas was: bone ($3M), arthritis ($2M), muscle ($1.7M), skin ($0.6M), rehabilitation ($0.6M), and oral health ($0.3M). This ranking was fairly typical throughout the period 2002-15, with some fluctuations (Figure 6).

III. Relevance and Performance

A. Relevance

This section addresses the ongoing relevance of the Institute’s mandate and the appropriateness of the Institute’s focus within its mandate area.

Canadian Context

Canada has a long history of strength in some areas of IMHA’s mandate. For example, rheumatology developed strong roots in Canada in the 1930s, and later, with the support of The Arthritis Society, all Canadian Medical Schools established Rheumatological Disease Units. In the 1960s, the University of Toronto surgeon Robert Jackson popularized and developed arthroscopy thereby revolutionizing MSK diagnostics and surgery (while also training former IMHA SD Cy Frank). These and other world-class clinician-scientists inspired generations of rheumatologists, orthopaedists, MSK scientists, and bioengineers.

More recently, musculoskeletal health research received a boost during, the fifteen-year term of the Canadian Arthritis Network (CAN) (1998-2014). Between 1998 and 2014, CAN received a total of $55M from the National Centres of Excellence (NCE) program and $16M from private sector and non-profit organizations. Of the $55M from the NCE program, CIHR’s contributed a total of $39M to CAN between from 2000-01 to 2011-12. These CIHR contributions came in the form of “Special Applications” that flowed through to IMHA-mandate spending in CIHR grants and awards database. It is unclear if the plateau of total CIHR spending on IMHA mandate after 2012 (see Figure 3) is attributable to the timing of CAN sun-setting.

The Arthritis Society, the single national arthritis health charity that has specifically supported arthritis research for the past half century has also been an instrumental partner with both the Canadian Arthritis Network and IMHA. Indeed, the rich relationship among The Society, The Network, and IMHA has, in addition to partnering support of MSK research, given the arthritis community’s voice in priority setting and grant review exercises. The Arthritis Alliance of Canada, whose members include researchers, consumer groups, the Canadian Rheumatology Association and other health professional associations, industry and others, continues its ongoing activities that influence the Canadian arthritis research and health care agendas. These activities in MSK are a model of effective partnership relationships and consumer engagement, and it is hoped that similar relationships can be developed successfully with the skin and oral health communities.

Prevalence and Burden of Disease

The worldwide prevalence of diseases that directly fall into IMHA’s mandate is staggering. Although most of these diseases do not kill, they cause widespread pain and discomfort and result in functional problems, disabilities and handicaps, and cost Canada’s health care system and our population billions of dollars each year. In Canada, the estimated annual burden (direct and indirect costs) of these diseases is astonishing, approaching nearly $40B annually: MSK diseases are estimated at upwards of $22B^{Footnote 4}, dental care expenditures of $12.5B^{Footnote 5} and skin is estimated at $4B (Bickers et al. 2006).

These costs are compounded by both the prevalence of diseases of these organ systems, as well as the chronicity of many of these diseases. For example, the prevalence of musculoskeletal diseases exceeds 25% of the population (Wang et al. 2012) and musculoskeletal and skin diseases account for some 45% of occupational diseases^{Footnote 6}. Although not widely recognized, dental caries and periodontal diseases are the most common chronic diseases in Canada.^{Footnote 7} Moreover, eczema, psoriasis, and rosacea afflict some 5M Canadians, and acne, alopecia, hyperhidrosis affect more than 50% of Canadians.^{Footnote 8}

Indeed, this group of diseases are the most common reason for seeking primary medical care, the most common chronic diseases, and the most common cause of disability, a particular problem for those of working age. Thus, IMHA mandate areas are especially important economically and scientifically for Canada and Canadians. As with any problem of this magnitude, there is a strong incentive for devoting resources to research the causes, treatments, and management of these diseases, as well as prevention in relevant cases. Based on their widespread prevalence, large burden of disease, and limited effective therapies for many diseases, objective and subjective evidence reaffirm that CIHR and Canadian federal investment in musculoskeletal, skin, and oral health remains extremely important and relevant.

Not-for-profit and Professional Organizations Aligned with IMHA’s Mandate

It is noteworthy that both the Canadian Arthritis Network and The Arthritis Society have made substantial strategic contributions to both capacity building and knowledge creation in the MSK area over the lifetime of IMHA. For example, The Arthritis Society invests about $5M per year in arthritis research with a cumulative total investment of approximately $190M in research since its inception in 1948. The Canadian Arthritis Network funded more than 200 research projects that included more than 1,100 collaborators and partners from across Canada and internationally. Whereas The Arthritis Society continues to fund scholars and operating grants, the Canadian Arthritis Network, a Network Centre of Excellence funded by the Government of Canada, reached its mandatory sunset in 2012, resulting in the termination of this funding source of arthritis research and trainees. Similar support mechanisms and organizations are lacking for skin and oral health research.

The commitment of federal funding for CAN was driven by the relevance and economic importance of arthritis and musculoskeletal diseases to Canadians. While CAN made demonstrable contributions to both knowledge production and translation, and to capacity building, the lack of definitive outputs and outcomes renders the effect on arthritis research somewhat obscure.^{Footnote 9} However, the crescendo in total IMHA mandate spending (Figure 1) from 2009-2012, which is mostly attributable to increased success in the open competitions, may be a sign of CAN’s success. So too, the plateau in this success after 2012-13, may be a worrisome sign that the effect of CAN is dwindling after its sun-setting.

There are four prominent professional organizations that represent specialty medical and surgical practitioners who see patients with diseases falling under IMHA’s mandate, including the Canadian Rheumatology Association, the Canadian Orthopaedic Association, the Canadian Dental Association, and the Canadian Dermatological Association. The support of those and other professional groups for the work of IMHA varies. For instance, the Canadian Dental Association had a representative on the former IMHA Institute Advisory Board, and provided partner funds with IMHA for the Network for Canadian Oral Health Research.

It is especially noteworthy that direct and active involvement of consumers in setting priorities in MSK research has changed the way consumers, scientists, clinicians, not-for-profits, and policy makers view IMHA-mandate research. For example, consumer support for short- and long-term solutions to the problem of pain in chronic rheumatological diseases has been a strong driver of strategic funding initiatives by the arthritis research community including IMHA (e.g., the Canadian Pain Summit). The presence of a strong public/patient/clinician-based advocacy and support group for arthritis is a great asset for that research community, though something that the oral and skin community would benefit from developing further.

Thus, IMHA mandate areas are of interest and importance to Canadian citizens, medical professionals, the Government of Canada, and to Consumers.

International Context

The comparison of CIHR and IMHA investments in the IMHA mandate area with its peer organizations in other countries is complex and beyond the scope of this evaluation; however, comparisons with the US and UK can provide some insights. In the USA, the National Institutes of Health, CIHR’s peer organisation, and has a number of institutes covering a variety of research areas. Among its 21 institutes and 6 centres are the National Institute of Arthritis and Musculoskeletal and Skin Diseases (NIAMS) and the National Institute for Dental and Craniofacial Research (NIDCR), which cover the IMHA mandate areas and, in 2015, had budgets of $520M and $397M, respectively,^{Footnote 10} collectively representing 32% of the total NIH budget of $2,900M. Much like CIHR, at the NIH, there is a balance between centrally funded trans-institutional projects and institute-lead projects, although compared to CIHR and its Institutes, the balance at the NIH leans more to individual institutes leading funding initiatives. At the other end of the scale, the Medical Research Council in the UK has no formal institutes, recognizing instead six explicit “Science areas:” infections and immunity; molecular and cellular medicine; neurosciences and mental health; population and systems medicine; global health; and translational research. This categorization is based more on methodological approaches rather than disease groups or body systems, and these areas drive the organisation as a means to review and award applications. The “population and systems medicine” area has a remit and scope that covers 13 explicitly named systems or groups, including musculoskeletal research. Research related to skin and/or oral health is not mentioned explicitly anywhere, although a quick review of funded research reveals projects in both fields.

Relevance to CIHR’s Strategic Priorities

Findings from the interviews and surveys indicate that IMHA’s mandate has remained relevant, largely unchanged over the past five years, and aligns with CIHR’s strategic directions, priorities and initiatives (e.g., chronic conditions, rehabilitation, patient-oriented research).The IMHA strategic plan (2014-2018) outlines three strategic research priorities: Chronic Pain and Fatigue; Inflammation and Tissue Repair; and Disability, Mobility and Health.^{Footnote 11} Of these, Chronic Pain and Fatigue, most directly aligns with the key research priorities outlined in CIHR’s Strategic Plan 2014-15- 2018-19 Health Research Roadmap II.^{Footnote 12} Moreover, it is noteworthy that IMHA has established partnerships in eight CIHR major initiatives aligned with these priorities, with an annual average contribution of $1M). From 2011-12 to 2014-15, IMHA contributed $4.8M or 14% of its RC budget to CIHR major initiatives (ninth among the 13 Institutes), with percentage increase from 7% in 2011-12 to 27% in 2014-15. The largest IMHA contributions were under the Collaborative Health Research Projects CHRP program^{Footnote 13} ($1.1M), Inflammation in Chronic Disease Signature Initiative ($1M), which is co-led by IMHA, Personalize Medicine Signature Initiative ($870K), Community-Based Primary Health Care Signature Initiative ($730K) and Strategy for Patient-Oriented Research ($730K).

Future Relevance of IMHA

Two factors suggest that IMHA mandate areas are becoming increasingly relevant. Specifically, many diseases of the MSK, skin, and oral cavity are chronic diseases with a prevalence, disease burden, and morbidity that increases with age. Given the progressive increase in the population of the aging (many of whom hope to maintain an active lifestyle), and the increase in obesity (the primary risk factor for highly prevalent osteoarthritis), the relevance of research in IMHA-mandate areas is increasing dramatically.

Areas of health research that promise to influence IMHA-mandate research areas include precision medicine and gene-editing therapies. Advances in microfluidics and microanalytics greatly increase the speed and accuracy of biomarker identification and disease phenotyping, and a substantial database of patients and blood and joint fluid biomarkers has been already established (indeed Canada is leader in this area), particularly for rheumatology patients. Such advances in precision medicine are likely to reveal diagnostic features that will enable the identification of specific targets for both common and rare phenotypes. This is the first important step for developing scientifically based therapies for this large and diverse group of patients with high likelihood for collateral benefit for other chronic inflammatory diseases. Gene editing therapies are likely to transform medicine, and the MSK, skin, and oral cavity, by virtue of their accessibility in the periphery, are the most likely initial targets for such interventions.

Looking beyond specific treatments, the chronic and largely “incurable” nature of many of the conditions in the IMHA mandate also means that IMHA and other Institutes are highly relevant in driving CIHR’s work to develop alternative health care delivery strategies and systems in Canada. For example, three recent Canadian Academy of Health Sciences reports highlight relevant challenges and make recommendations with respect to chronic disease, scopes of practice among health professionals, and access to oral health care.^{Footnote 14}

Lastly, given the high costs of pharmaceuticals used to treat IMHA-mandate disorders, particularly the highly effective biologicals for chronic inflammatory diseases, research in this area is highly important for informing policy decisions on pharmacare. Similarly, given that Canada is one of very few countries with publicly-funded health care that lacks coverage of oral health, research on oral health services is highly important for informing the development of future policies on dental health care.

B. Performance

Panel interviews of select individuals, as well as key informant interviews and surveys of researchers and stakeholders indicate that IMHA has engaged a wide range of stakeholders (researchers, clinicians, consumers, etc.) through a wide variety of mechanisms (visioning exercises, meetings, workshops, etc.). IMHA has also engaged national and international partners in knowledge creation, sharing, and translation, in capacity building, and in policy development and implementation. Findings from the stakeholder survey identified few IMHA activities in commercializing research but show the SD to be knowledgeable, a good communicator, forward thinking, collaborative, and a strong advocate for patient engagement.

While quantifying the “treatment effect” of IMHA-sponsored strategic initiatives remains challenging, in part due to temporal shifts in outcomes and insensitivity of currently available methodology, the panel concluded that the accomplishments and performance of the current Scientific Director of IMHA in Transformational Impact is reasonable given the resources and timeframe in which he had to accomplish them. However, there were some difficulties in trying to evaluate ‘success’ in relation to the IMHA strategic plan given its lack of specificity and targeted outcomes and associated performance measurement data.

Transformational Impacts

Support to Innovative Research and Advancing Knowledge

Since its inception in 2001-02 to 2014-15 IMHA spent $104.5M on 199 different strategic initiatives, which include a wide variety of grants (Team, Emerging team, Catalyst, Operating, CHRP, Seed, Planning, etc.) and awards (chairs, new investigators, clinician-researchers, fellowships, graduate studentships, undergraduate scholarships, etc.) Under the current IMHA Scientific Director (April 2013 to present), RC spending was $8.5M in 2013-14 and $7.8M in 2014-15, with $1.2M (15%) and $2.1M (27%), respectively spent on Major Initiatives.

A bibliometric analysis by the CIHR Evaluation Team for IMHA-mandate related knowledge production (i.e., publications in PubMed and World of Science databases) mentioned support by CIHR for the years 2008-2015. The total number of citations for IMHA-mandate areas was n=9,954, with a focus area distribution of 8,221 citations for Musculoskeletal Diseases, 2,053 for Skin, and 279 for Oral health. These numbers likely underestimate the real number, particularly for some IMHA-mandate areas not included in the PubMed/Web of Science (WoS) databases (e.g., some engineering journals where functional studies might be published).

Contribution to Building the Capacity of the Health Research Enterprise

The number of trainees supported by CIHR is a proxy for the new capacity of the IMHA research community (acknowledging support from other sources). A sustained increase in the number of funded trainees indicates an expanding capacity of the IMHA-mandate research community.^{Footnote 15} Since its inception in 2001-02 to 2014-15, the number of direct trainees rose steadily from 1,250 to 2,250; similarly, during this period of time, the number of indirect trainees rose steadily from 4,500 to about 6,500, while, IMHA-mandate trainees as percentage of all CIHR trainees grew from about 6-8% to 12% (Figure 7).

Under the current Strategic Plan (2014-2018) developed by the current SD, IMHA adopted three research themes: Capacity Building, Innovation, and Translation. In these themes, the IMHA focused primarily on capacity building through training initiatives, and convening an annual young investigators forum and a research ambassador program. Capacity building is a long-term strategy for IMHA, with proportionately high numbers receiving CIHR support (Figure 8). The current SD has undertaken a number of capacity building activities. For example:

• held IMHA’s Young Investigator Forum, in October 2015, in conjunction with Arthritis Alliance of Canada (ACC);
• developed an Embedded Clinician Research Salary Award in conjunction with the Community-Based Primary Health Care Signature Initiative; and
• established two studentships for students working with established Canadian investigators on psoriatic disease-related research, in partnership with Canadian Association of Psoriasis Patients (CAPP).

Contribution to Achieving Broader Health, Economic and Social Impacts

A recognized gap in Institute performance measurement and evaluation, the Performance and Accountability Branch reviewed and analyzed data that partly assesses how CIHR-supported research has influenced decision-making beyond academia. A search was conducted of approximately 5,000 publically accessible electronic documents released between 2008 and 2015 – approximately 50% by Health Canada (HC) or the Public Health Agency of Canada (PHAC) -- to find instances when these documents were observably influenced by CIHR supported research.A summary of the methodology applied is provided in Figure 10.

An initial analysis of these data suggests that at least 252 downstream documents released between 2009 and 2015 (i.e., decision-making/policy making documents) were observably influenced by IMHA-related research that was supported by CIHR. Of these, 152 had a moderate to “strong influence.” ^{Footnote 16} The most common downstream documents included are guidelines (64), followed by reports (33) and health technology assessments (26). The most common publisher/author of these downstream documents was the Canadian Agency for Drugs and Technologies in Health (23), followed by the PHAC (21). The overall average time to influence for all influenced documents (the gap in years between publication of the article/knowledge product and its use in a downstream document) was 2.7 years. In addition, 235 patents have been observably influenced by IMHA-related publications, which were published in 2008 and 2009.

The following are two examples of the impact of IMHA-related publications.

The article “2010 Clinical Practice Guidelines for the Diagnosis and Management of Osteoporosis in Canada: Summary” was written by CIHR supported researchers and co-funded by Osteoporosis Canada. This guideline had a significant impact within academia, with over 330 citations in Web of Science as of November 2016, and has been used as a source of evidence by numerous entities and organizations in Canada (e.g., Public Health Agency of Canada, Canadian Physiotherapy Association, Alberta Health Services).

The article “A randomized trial of arthroscopic surgery for osteoarthritis of the knee” (2008) was solely funded by CIHR and the supported authors held a total of 22 CIHR grants and awards at time of publication that fall within the window of support of this publication. A follow-up study to this article was undertaken by other researchers in the United States “to determine if the results of clinical trials were associated with changes in practice patterns, and concluded that arthroscopic knee surgery declined in Florida by 47 percent between 2001 and 2010.” The US study specifically found that “rates also declined following publication of the results of Kirkley and colleagues' [CIHR] trial in 2008” and estimate that the total reduction in this surgery translates into national savings (in the US) of between $82 million and $138 million annually, indicating clearly that “clinical trials of widely used therapies can lead to cost–saving changes in practice patterns.” This CIHR supported research article was highlighted by CIHR in 2013 and has been used as a source of evidence by several health insurers in the United States including: AETNA, FirstCarolinaCare, Tufts Health Plan, and the Blue Cross of Idaho.

Catalyst/Convener

Partnering to Achieve CIHR and IMHA Objectives

IMHA has engaged in a number of notable partner relationships, aided by the current SD, including:

• recapping the Canadian Arthritis Network and IMHA accomplishments document: Celebrating the Impact of Health Research: Success Stories In Arthritis, Bone, Muscle, Musculoskeletal Rehabilitation, Oral Health, and Skin, that syntheses lessons learned and highlights accomplishments and impacts of research;
• maintaining ongoing partnerships with NSERC through the Collaborative Health Research Projects program (especially important for the functional assessment of MSK, skin, and oral tissues which require engineering approaches, particularly biomechanics);
• co-leading the Signature Initiative in Inflammation in Chronic Disease, including negotiating an international network partnership on personalized medicine approaches for treating inflammatory MSK diseases with the Dutch research agencies ZonMw and Reumafonds ($8M);
• providing four years of support for an oral health clinician-researcher on the Signature Initiative in Community-based Primary Health Care;
• co-leading with the Institute of Aboriginal Peoples’ Health (IAPH) and Institute of Population and Public Health (IPPH), IMHA is working to launch capacity-building funding opportunities as part of the oral health priority under the Pathways to Heath Equity for Aboriginal Peoples Signature Initiative;
• co-leading the Health and Productive Work Signature Initiative with IGH and IA, and assisted with the development and launch of Partnership Development Grant funding opportunity; and
• partnering with the James Lind Alliance to conduct a successful priority-setting exercise for adult fibromyalgia related to Chronic Pain and Fatigue.

The panel recognizes that the partnerships promoted by IMHA have helped establish and build networks, training opportunities, and furthered the granting capacity of the IMHA community. Moreover, IMHA has aligned itself well with the current CIHR research priorities, particularly in the areas of chronic conditions and Indigenous Health.

Partner Contributions

From its inception in 2000-01 to 2010-11, the total partners’ contribution to funding opportunities under IMHA mandate increased from $1.5M to almost $8M then diminished slightly to $6.3M in 2014-15. From 2001-02 to 2014-15, the Leverage Ratio of Partnership was bimodal, peaking in 2002-03 and again in 2009-10, which reflects changes in both partner contributions and CIHR investment in IMHA’s mandate (Figure 11).

Visibility and Profile

The primary assessment of IMHA visibility and profile draws on findings from key informant interviews and the survey of IMHA stakeholders. There is consensus among interviewees and survey respondents that IMHA is primarily visible on the website and by regular email communications with the stakeholder community. The stakeholder survey found that the vast majority of stakeholders (88%) consider themselves well informed and that this information and the visibility of IMHA is primarily related to information provided by the SD, periodic emails from the institute and the IMHA website.

Contribution of Scientific Leadership to Convener/Catalyst Role

Interviews and surveys revealed that the SD is a very well-respected, well-liked, trustworthy clinician-scientist who has credibility working with the IMHA scientific community, with The Arthritis Society, The Canadian Arthritis Network, the Arthritis Alliance of Canada, as well as many other not-for-profit and professional organizations (locally, nationally, and internationally). Three-quarters (74%) of stakeholders surveyed indicate that the current SD has excelled at ensuring community feedback, opinions, and concerns were taken into consideration when implementing Institute decisions, actions, and activities.

Operational Effectiveness

IMHA, as all other CIHR Institutes, receives $1M annually as an Institutional Support Grant (ISG) for operating and development expenditures. The unspent balance rolls over at the new fiscal year, hence, IMHA currently has accumulated a surplus of approximately $1M. IMHA has typically spent its full annual ISG installment of $1M or 50% of the total ISG funds available (~$2M) between 2011-12 and 2014-15, with a slight increase to 60% in 2015-16. This frugality may be valuable for the incoming SD for gathering advice (e.g., former Institute-specific Advisory Board), for establishing additional scientific support positions (e.g., Associate SD role), and for bridging to CIHR centrally.

A disadvantage of the high-level and broad-based strategic priorities and themes at IMHA is that it makes it difficult to define and evaluate success. Notwithstanding the very broad goals within each of these three strategic priorities, and stickhandling six distinct focus areas, IMHA was forced to choose some very specific projects that could be achieved within the relevant timeframe and tight budget constraints.

Based on interview and survey data, the operational effectiveness of IMHA in implementing its strategic plan is progressing as would be expected given a late start in 2014. Challenges encountered by the Institute include the redeployment of dedicated CIHR staff to the Institute (the loss of a “direct connection” to CIHR), the disbandment of the Institute-specific advisory boards, and the perception of weak host institutional support to IMHA’s SD and staff (e.g., Human Resources and Finances).

IV. Conclusions and Recommendations

A. Should IMHA be amended, merged or terminated?

The panel recognizes the challenges of the breadth of the IMHA mandate, though after considerable discussion, recommends keeping the IMHA mandate areas unchanged. It is noteworthy that—like IMHA— skin and MSK are bundled together in the (NIH) Institute for Arthritis and Musculoskeletal and Skin Diseases (NIAMS), yet, unlike CIHR, oral health is included in a separate NIH Institute for Dental and Craniofacial Research (NICDR). Nonetheless, given the size of CIHR and the amount of research activity in oral health, separating this mandate area, and either transferring this important area to another CIHR Institute or establishing a separate Institute seems unjustified at this time given the additional administrative overhead required. However, as noted above, future needs in oral health service research (in addition to expansion of basic and applied research), may be a driver for reconsidering a separate Institute for Oral Health in the future.

Given the history, the familiarity of the community with the current acronym and its acknowledged mandate areas of skin and oral health, changing the name of Institute for Musculoskeletal Health and Arthritis seems unwarranted. Based on panel interviews with select stakeholders, it was perceived that most of the community uses the acronym IMHA freely and have an implicit understanding that, in addition to MSK and arthritis, it includes skin and oral health research. Findings from key informant interviews indicate that overall the name was not a major concern for respondents; however, it was noted that the name was not representative of all components.

B. Should IMHA’s Mandate be Changed

The broad mandate of IMHA is to promote research in MSK, skin, and oral health, three organ systems composed in large part of connective tissue. Whereas this structural commonality literally binds these areas together, there are also fundamental differences that make them distinct. As there are fewer active Canadian researchers in skin and oral health, these mandate areas are smaller, and to a certain extent, orphaned compared to those associated with the MSK system. Nevertheless, IMHA historically has initiated and promoted strategic research in all its mandate areas.

The skin and oral health research communities are both small groups and data gathered for this review from members of both communities (and indeed members of the larger musculoskeletal groups) highlighted the point that both groups feel somewhat out of place at IMHA, while simultaneously agreeing that there is no obviously better Institute for them. Skin and oral health researchers recognize that there are clear links between their work and that of other members of the IMHA community and with no obvious alternative Institute to house their interests, they believe that, for now, they are better positioned to stay within the mandate of IMHA.

The panel believes the published IMHA Mandate statement is rather wordy, and might benefit from re-examination and rephrasing with an eye on succinctness, and how to know when success has been achieved. A specific example is the change, over time, in IMHA strategic priorities: A priority of the 2002-2005 strategic plan was “Mobility, Fitness and Exercise.” In subsequent strategic plans 2008-2013, and 2014-2018, Mobility was retained, and Fitness and Exercise were dropped. Mobility is explicitly stated in the IMHA mandate statement, as is “Active Living,” so it might be interpreted that Fitness and Exercise component of Active Living has been diminished. Moreover, the panel believes that IMHA would benefit from developing more precise aims, objectives, and goals that specify measurable targets and outcomes, and engaging the relevant stakeholders in their creation.

C. Observations and Recommendations for CIHR

In the fast-changing organization and programs of CIHR, and the disappearance of the Institute-specific advisory boards, the SD has become somewhat of a lone figure at the Institute. The panel recommends CIHR consider new mechanisms for strengthening lost connections between Institutes and 160 Elgin.

The persistent problem of small research group integration and relevancy begs to be addressed. The panel recommends that CIHR develop a mechanism for meaningful input and representation by small research groups to help build research capacity and activities in these small but critical areas. There are examples of successful strategies to build capacity elsewhere in CIHR (e.g., the Institute of Population and Public Health) and could be used to support other groups. To this end, a potential approach for creating a community voice for small research groups across CIHR was provided to CIHR management under separate cover. Nevertheless, the small size of these research groups coupled with the sense that they are somewhat out of place and sometimes an awkward fit, needs to be acknowledged.

With the current research budget of $4.3M per year, and the fact that many strategic initiatives span multiple budget years, the budget available to the SD for new initiatives can be rather limited. This can create the impression that IMHA and its Scientific Director has more discretionary budgetary resources than is actually the case. Given the amount of time needed by the panel to understand CIHR and IMHA finances, the panel recommends that CIHR make a special point to fully inform SD recruits of the fine details of how CIHR budgets work and what is available to the SD in the Institute’s research budget.

Whereas “tithing” the IMHA budget helps support large CIHR-supported initiatives, the shrinking budget available for IMHA-specific funding undoubtedly reduces IMHA’s ability to be nimble in responding to research opportunities and to take risks, both of which potentially threatens to dampen Institutional and community creativity.

Moreover, a budget cut to a small group will undoubtedly have a larger effect than on a larger group within an Institute, and there is a risk that smaller oases of important activities may dry up. It is imperative that major CIHR funding initiatives receive the same scrutiny for evaluating success as the Institutes themselves. Should these large cross-cutting initiatives be found to be effective, there still may be unintended consequences on dwindling Institute resources, particularly capacity building efforts.

Partnership funding for CIHR and Institute-specific initiatives has become a very important element of the work of CIHR over the past decade, and overall, this is an excellent evolution. That said, it is very clear that there are some fields of health research that have a good number of natural partners and those partners have considerable resources to invest in research, while other areas of health research have few, if any, natural partners and few or no resources to invest in research. In this sense, some partnerships can have the effect of increasing inequalities in research investment across CIHR. This is particularly relevant to IMHA as a whole, where partnership funding has declined significantly in recent years and there are no obvious replacement partners. It is also relevant to, again, the small research groups within IMHA, who have no partners with any level of significant resources to support their fields. This partnership investment, along with the very large differences in capacity and capacity building in certain areas across the CIHR mandate will contribute to enhanced research inequalities. If the current approach continues, the inequalities in research investment between the “have” areas of research and “have not” areas of research will result in certain groups going into terminal decline, and Canada effectively having no research activity in those areas.

V. Other Considerations

A. Observations for the Next Scientific Director

The broad mandate of IMHA, enfolding three organ systems and a catalog of body parts, poses significant challenges for the SD and Institute to work effectively with the various communities. The incoming SD must be aware of and credible to all communities to effectively represent their interests and promote their successes, individually and collectively. Hence, the panel recommends that the incoming SD should be a credible scientist to all mandate areas.

The disbandment of dedicated Institute Advisory Boards was mentioned repeatedly as a potential problem for the incoming SD. Furthermore, with the fast-changing organization and programs of CIHR and the disappearance of the Institute-specific advisory boards, the SD has become somewhat of a lone figure for his/her Institute. The panel recommends that the SD may wish to name an internal IMHA advisory group to provide feedback on tactical and practical decisions and planning, as well as to be a pulse of the community. The panel also recommends that the new SD of IMHA might consider creating an Associate SD position for scientific rather than support staff position(s). This Associate SD role would be to support the SD in his/her development of plans and overseeing elements of the institute’s leadership as assigned by the SD.

Similarly, the changing internal structure of CIHR provides less direct support for the SD, with different people being assigned one role for more than one Institute, thereby splitting their allegiance. Also, there is no longer the sense that there is a person with a strong support role in CIHR representing IMHA, i.e., IMHA’s “voice” in the administration of CIHR. This understanding and feeling, among IMHA mandate researchers, of the diminished role of the institutes and so the diminished voice of IMHA mandate researchers in the directions CIHR is taking is a strong part of the sense that CIHR has become a “top-down” rather than “bottom-up” or balanced input organization. Hence, the panel recommends that the new SD consider measures to strengthen the direct links between institute administrative support and that of CIHR with a view to enabling the Institutes to fulfill their specific mandate.

CIHR might wish to consider what type of individuals would be attracted to apply to become the SD. Decreasing discretionary budgets makes it less attractive for someone ambitious to undertake bold and large initiatives. Indeed, by limiting flexibility and opportunities to be creative may attract individuals with different ambitions and skills. Indeed, marketing the Institute’s interests and having skills to negotiate with other Institute Directors may be as important as scientific accomplishment.

The review panel members struggled to specify priorities for incoming SD, and how they might develop a strategic plan given the limited resources available to the Institute and the broad-based nature of the three strategic priorities. There was some sense that the new SD might initially focus on the clarity of purpose to the definition of ‘success’ at the outset of his/her mandate. That could include priority setting within the three research ‘themes’ in terms of whether capacity building, innovation, or knowledge translation is most important or cost-effective, and then to create some targeted initiatives to achieve specific goals within each of these themes. Evidence from this evaluation suggests that capacity building that leads to increased application pressure in investigator-initiated competitions may be the most effective strategy for growing the strength of the IMHA community.

B. Strategic Considerations

Given the high prevalence and burden of disease, it is somewhat disappointing that research in IMHA-mandate areas currently receive such a modest proportion of total CIHR spending. The current CIHR mandate is silent on strategic directions on disease prevalence or burden and how that might guide health research. While the current CIHR strategic plan, Roadmap II, specifically promotes “mobilizing research for transformation and impact,” and recognizing that even a small amount of progress on highly prevalent and costly diseases can have a powerful and transformational impact, it is unclear how Institutes in these mandate areas can efficiently and effectively mobilize. As is clear in the current CIHR model, the majority of research funding is assigned to Institutes through open grants and awards. Hence, a straightforward approach for an SD is to increase success in open competitions (meeting CIHR’s strategic goal to fund highly meritorious research), which necessarily requires increased numbers of highly qualified applications and applicants. However, the limited and diminishing funds available to IMHA to train and network their multiple communities act as the governor on building a critical capacity of IMHA-mandate researchers, and the time it will take to grow that community may be insufficient to benefit the well-described aging population bubble.

VI. References

Celebrating the Impact of Health Research: Success Stories In Arthritis, Bone, Muscle, Musculoskeletal Rehabilitation, Oral Health, and Skin

NIH budget overview FY 2015:
https://www.hhs.gov/about/budget/fy2015/budget-in-brief/nih/index.html

Canadian Academy of Health Sciences (report): Transforming Care for Canadians with Chronic Health Conditions: Put People First, Expect the Best; Manage for Results
http://cahs-acss.ca/transforming-care-for-canadians/

Canadian Academy of Health Sciences (report): Optimizing Scopes of Practice: New Models of Care for a New Health Care System
http://cahs-acss.ca/optimizing-scopes-of-practice-new-models-of-care-for-a-new-health-care-system/

Canadian Academy of Health Sciences (report): Improving access to oral health care for vulnerable people living in Canada
http://cahs-acss.ca/transforming-care-for-canadians/

Bickers, D. R., Lim, H.W., Margolis, D., Weinstock, M.A., Goodman, C., Faulkner, E., Gould, C., Gemmen, E., & Dall, T. (2004). American Academy of Dermatology Association; Society for Investigative Dermatology The burden of skin diseases: 2004, Journal of the American Academy of Dermatology, (55)3, 490–500.

Wang, H., Dwyer-Lindgren, L., Lofgren, K.T., Rajaratnam, J.K., Marcus, J.R.; Levin-Rector, A., Levitz, C.E.; Lopez, Alan D., Murray, C.J.L. (2010). Age-specific and sex-specific mortality in 187 countries, 1970–2010: A systematic analysis for the Global Burden of Disease Study 2010. The Lancet, 380(9859), 2071–2094.

VII. Appendices

Appendix 1: IMHA Evaluation Panel Members’ Affiliations and Conflict of Interest Declaration

Chair

• John Matyas, Associate Dean & Professor, Microscopic Anatomy Comparative Biology and Experimental Medicine, University of Calgary

Panel Members

• Claire Bombardier, Professor of Medicine, University of Toronto
• Janet Yale, President & CEO (Ex-Officio), Arthritis Society
• Paul Allison, Dean & Professor, Faculty of Dentistry, McGill University

At the outset of the IMHA Evaluation Panel Synthesis Workshop in February 2017, all Panel members were invited to declare any conflict of interest that could impair their ability to perform their duties as IMHA Evaluation Panel members in an objective and impartial manner.

Appendix 2: Overview of Data Sources and Methods

Data source	Description
Key informant interviews	Semi-structured telephone interviews (English and French), 45 – 60 minutes with representatives of organizations who have partnered with and/or are knowledgeable about IMHA, to gain informed perspectives on Institute relevance and performance Interviewees identified by Institute and CIHR Thematic analysis was conducted by evaluation question and indicator 25 interviews completed Category # Completed Institute staff, CIHR staff and IAB chair/members 6 Potential knowledge users: health system decision-makers practitioners patients 9 Institute research partners and stakeholders 8 International agencies 2 Total 25 Interviewee Focus Area (N=27) Note: Two interviewees were affiliated with two focus areas Long Description Interviewee Focus Area (N=27) Oral health MSK Rehab. Arthritis Pain Skin Bone Other 4 4 5 3 3 1 7
Researcher survey	Web-based survey of 2000-2015 grantees measuring the relevance of the mandate and contribution to knowledge, capacity, and larger impacts Response rate: 31%, 301 completed questionnaires out of 970 eligible respondents All data reviewed against key independent variables, proxies for affinity and cohort: % of IMHA funding within institute/CIHR funding (more or less than 80%) Year of most recent Grant/award funded out of the budget of a CIHR institute Number of years in research (0-15, 16+) Gender
Stakeholder survey	Web based survey of stakeholders measuring the relevance of the mandate and contribution to knowledge, capacity, and larger impacts Interviewees identified by Institute and CIHR Response rate: 21%, 208 completed the questionnaire out of 991 eligible respondents.
Secondary Data Analysis	Quantitative analysis for data from CIHR Electronic Information System (EIS) Financial data analysis from IMHA’s Institute Support Grant (ISG) Analysis of the influence of IMHA mandate-related knowledge products within and beyond academia, using data from Thomson Reuters’ Web of Science, CIHR’s administrative records and other studies undertaken by CIHR Performance and Evaluation Branch Documents review

Appendix 3: Figures and Tables

Figure 1: Total CIHR Spending on IMHA Mandate and Spending on Each of IMHA Focus Research Areas, over Time^{Footnote 17}

Figure 2: Value of CIHR Funded Grants & Awards within IMHA Mandate out of Total Value of CIHR Funded Grants & Awards over Time

Figure 3: The Value of Annual CIHR Spending on the Mandates of Each of the CIHR Institutes over Time

Figure 4: IMHA Mandate Expenditure by Funding Type over Time

Figure 5: The Percentage of CIHR Funded Grants and Awards within IMHA Mandate out of The Total Number of CIHR Funded Grants and Awards and The Number of CIHR Funded Grants and Awards in IMHA’s Focus Research Areas

Figure 6: IMHA Strategic Spending, over Time

Figure 7: Numbers of Direct and Indirect Trainees Supported under IMHA Mandate

Figure 8: Investment in Capacity Building Funding out of IMHA’s Strategic Investment over Time

Figure 11: Partners’ Contribution to Funding Opportunities under IMHA Mandate and Leverage Ratio of Partnership: Partners to CIHR Investment in IMHA Mandate

Date modified:

2017-08-15