Expert Review Team Report for Institute of Health Services and Policy Research

Submitted by: Professor Sally Redman
Chair, Expert Review Team
February 2011

Table of contents


Summary

The Institute of Health Services and Policy Research (IHSPR) has an extensive mandate - it is responsible both for developing health service and policy research within its Institute (institute role) and for stimulating research respecting health systems and services across CIHR as a whole (theme role). It must have a strong focus on translation and ensuring research is relevant and accessible to policy agencies.

There were considerable challenges facing the IHSPR at its establishment including limited capacity in health service and policy research (albeit with existing pockets of excellence), no prior history of funding through Medical Research Council and few established relationships between CIHR and policy agencies. In addition, the Institute was faced with the need to develop relationships with all of the provincial governments responsible for the delivery of health care.

The Institute has taken a strategic approach to achieving its mandate – for the first few years it focused on building capacity broadly in the health service and policy research community, increasing the community's capacity to conduct excellent research and forging partnerships to create more opportunities for researchers. More recently, it has developed knowledge translation (KT) strategies and refined its focus to a smaller number of priority areas. There appeared to be considerable support for the priorities among the researchers and stakeholders, particularly for the work in primary care.

Given the initial challenges, it will inevitably take some time before the impact of the Institute's work can be adequately assessed. Evaluation of the impact of the IHSPR will also require collection of more sophisticated information about the quality of research and its influence on policy and practice than was available to the ERT (see section 6).

However, the IHSPR has put in place a platform that seems likely to result in important changes in the future. It has:

The ERT team was impressed by the value placed by researchers and stakeholders on the Scientific Director and staff of the Institute.

A key measure of the impact of the Institute's capacity building is the proportion of open grants allocated to health service and policy research. This rose from 1% in 2000/01 to 6% in 2009/10 and is a significant achievement.

However, there are opportunities to strengthen the work of the Institute going forward. The ERT recommends that:

  1. The IHSPR with the support of CIHR as a whole place more emphasis on monitoring the quality and quantity of health service and policy research, the impact of this research on policy and practice and the evaluation of its own initiatives and programs.
  2. The IHSPR continue to refine its strategy to ensure the most effective portfolio of programs. This should include consideration of the number of current initiatives and the most appropriate balance of research focused and KT activities.
  3. CIHR work with the IHSPR to develop additional support for the Institute's theme role, including additional funding.
  4. As IHSPR moves beyond broad capacity building, there is a greater focus on the quality and policy and practice relevance of the research outputs. Particular attention should be paid to areas where there is an opportunity for both international leadership and relevance to Canadian policy and practice including:
    1. Further development of the work to make secondary data more available to researchers. Canada has excellent resources and an existing skill base; these data are of fundamental importance to informing health services and policy
    2. The development of approaches to integrate rigorous research into the evaluation of government policies and programs. Canada has a unique opportunity to exploit the natural experiments arising from variations across the provinces and an existing skill base in implementation research, including economics.

1. Institute mandate

The mandate of the IHSPR is to:

'champion and support excellent health services and policy research and knowledge translation to identify, understand and address health system needs and challenges and to contribute to health system accessibility, responsiveness, effectiveness, efficiency and sustainability.'

We note that this is a very ambitious mandate, and that one would not expect immediate or short-term impacts on health system accessibility, responsiveness, effectiveness, efficiency and sustainability in Canada as a result of the activities of this Institute.

2. Status of this area of research in Canada

Health services and policy research is an emerging field internationally and lacks the status of more established research fields. The mandate of the CIHR has a special focus on health services and policy research, as it is required to ensure translation of knowledge from research into 'more effective health services and products and a strengthened health care system'. Accordingly the CIHR Act defines Research respecting health systems and health services as one of its four themes.

Significant progress has been made over the past ten years in building the status of health services and policy research in Canada, although more work remains to be done. There are several indicators of the status of research in this area.

In terms of international standing of Canadian health service and policy research, the ERT noted that Canada is internationally regarded as having research strengths in health services and policy research including in systematic reviews (through the Cochrane Collaboration), the use of secondary data sources, knowledge transfer and implementation science.

Although there has been considerable investment in building capacity in health services and policy research, it was difficult to judge the extent to which this had resulted in an increase in scientific excellence and international standing. The bibliometric data provided by CIHR included analysis of only one area, access to appropriate care, and was unable to distinguish whether papers were the outcome of CIHR funding; this limited analysis provided some evidence of international standing, indicating that Canada was ranked fourth in the world based on average of relative citations and that some of the researchers working in this area were evidently highly productive and had excellent citation rates. However, the overall standing of Canadian health services and policy research could not be judged from the bibliographic information (see also section 6).

In terms of the status of the research among policy makers and practitioners, the IHSPR report provided several examples of the impact of particular pieces of research on policy including those focused on rising cancer drug costs and geographical variation in access to coverage, the quality of care provided in long term care facilities, improving patient safety in Canada and the effectiveness of public report cards for improving the quality of cardiac care. The policy stakeholders perceived that health services and policy research had the potential to be very useful in their work and identified specific pieces of research that had been useful. However, they also felt that more research of direct relevance to policy makers should be supported.

In terms of the status of health services and policy research within the CIHR, growing value is being placed on this research although there is room for further improvement. In recent years, a greater number of funded researchers are identifying an affiliation with IHSPR and a greater proportion of competitive funds are going to this area. However, this theme continues to receive the smallest amount of funding of all four themes. There has been relatively limited success in partnering with other institutes, particularly those outside the cross cutting themes, although some interesting partnerships are beginning to emerge such as that with the Institute of Genetics to support work on genetics and health services and policy research.

3. Transformative Impacts of the Institute

The IHSPR has a broad mandate and is beginning from a relatively low base in terms of both capacity and partnerships; it will therefore take some time before the transformative impact of the Institute can be adequately assessed. However, it is evident that the IHSPR has put in place a platform that seems likely to result in transformative impacts in the future including:

4. Outcomes

Strategies: IHSPR has been very active in implementing a large number of varied programs designed to build capacity and increase the impact of research. Capacity building strategies include establishment of team grants, researcher support (through the Capacity for Applied Developmental Research and Evaluation (CADRE) and Applied Chairs programs) and Strategic Training Initiatives in Health Research (STIHR). Other programs seek to develop research infrastructure or stimulate research in areas of priority – examples include the Drug Safety and Effectiveness Network and the new primary care initiative. The KT programs under the umbrella of Evidence on Tap include Best Brains, Café Scientifiques and Expedited Knowledge Synthesis.

Outputs: There is also some evidence of outputs from this work. In terms of capacity building, over a ten year period there was an 8 fold increase in the number of researchers identifying a primary affiliation with IHSPR. The CADRE program alone supported 83 post docs, 12 mid-career chairs, five regional training centres and 13 career reorientation awards and in total supported more than 1250 researchers. The 33 STIHRs over the past ten years have attracted 2,400 trainees. The high numbers participating in the conferences indicates the growing development of a community of health service and policy researchers. It would be helpful in the future to collect data on next destinations and ultimate job profiles of these recipients of training funds, both to assess the value of these training programs and also their contribution to policy, practice, and academia.

There is evidence that the KT activities are valued; over 300 policy makers have participated in the ten Best Brains Exchanges.

Outcomes: The ERT found it difficult to judge the outcomes from the work of the IHSPR. It is still early in the development of the Institute and the data provided were insufficient for this purpose. However:

  1. The increase in the share of funds from the CIHR open competition from 1 -6% indicate that the work of the IHSPR is contributing to the development of significant capacity in health services and policy research and
  2. A number of examples of research impacting on policy and practice were cited in the IHSPR report and during the interviews.

5. Achieving the Institute mandate

The Institute's mandate is extensive and encompasses both (i) championing and supporting excellent health services and policy research and (ii) knowledge translation to identify, understand and address health system needs and challenges and to contribute to health system accessibility, responsiveness, effectiveness, efficiency and sustainability. The Institute has both institute and theme responsibilities.

It is too early to fully evaluate the extent to which the Institute is delivering on its mandate. The Institute has:

The ERT also noted that the Institute has continued to modify its approach to delivering on its mandate from broad programs in the first five years to an increasingly targeted approach focused on a smaller number of selected priority areas – access to care, pharmaceutical policy, community based primary care, and use of secondary data. This is an appropriate approach and likely to make best use of the limited funds available.

An adequate assessment of the extent to which the broad and complex mandate of the IHSPR is being achieved will likely require purpose built evaluation strategies. The Institute has recognised this need and should be encouraged to develop stringent approaches including independent assessment and rigorous methodologies. Over time, these should include assessment of the impact on health systems and health (see also section 6).

6. ERT observations and recommendations

Additional observations

In addition to the comments above, the ERT noted that:

Measuring impact

Theme and Institute
The IHSPR is required both to undertake the work of a traditional institute within CIHR and to lead the development of the Research respecting health systems and health services (theme 3) across CIHR. While a good start has been made in integrating health services and policy research within other institutes, much more could be done. CIHR could consider providing additional strategic funds to IHSPR to assist it in achieving its dual roles. Additional strategies to impact on other institutes should be considered – for example, it might be of value to include someone with health service or policy research skills on the Institute Advisory Boards of the other institutes to provide a prompt and expertise for considering these research opportunities, and to ensure representation of health service or policy research skills on open grant funding committees.

Complex and shifting environment
The ERT noted that the IHSPR is working in a complex environment with many agencies and some overlap in roles and responsibilities. Some of the stakeholders saw that Health Canada should seek to better coordinate agencies in this area. It is apparent that the changed role of the Canadian Health Services Research Foundation and the closure of the social science funding agency will have some implications for IHSPR. Likewise some opportunities to engage more closely with policy agencies may be created through the renegotiation of the Health Accords.

IHSPR appeared aware of the challenges described above, and in our view needs high-level support from the CIHR corporately if it is adequately to meet them.

Focus and balance of activities
The IHSPR has clearly worked very hard to establish strategies and programs to address its broad remit. While a large portfolio of programs may be inevitable at the outset, there is a danger that a small staff (four plus a half time Scientific Director) may be spread too thinly. There is evidence that the Institute is aware of this challenge and has begun to refine its focus.

However, some of the stakeholders expressed a concern that the IHSPR may be too thinly spread and should increase its focus on its core activity of building strong and relevant health services research. These stakeholders felt that the Institute could do more to ensure that the research is as relevant to policy makers as possible; for example, it would be of considerable value to stimulate research that makes the evaluation of government policies and programs more rigorous.

However, other stakeholders felt that the Evidence on Tap programs were very valuable and the Scientific Director reported that these programs have demonstrated the potential value of health services and policy research and resulted in partnerships and additional funding.

Given the limited resources, it may be of value to undertake a strategic review of the IHSPR portfolio to ensure that the best balance is achieved.

Competitive funding and peer review
A key outcome for the IHSPR is to increase the amount of competitive funding from CIHR open grant schemes for health services and policy research. This will be determined by the volume of applications and the numbers of review committees. Careful attention will be required during the upcoming collapse of the number of committees to ensure that the capacity of health service and policy researchers to successfully compete for the open funding is not diminished.

Staff
The ERT was impressed by the very positive comments about the staff of the IHSPR during the interviews. The previous scientific director was clearly respected and trusted and had excellent skills innovation, facilitation and partnership development, and there was considerable goodwill towards the incoming director. The staff was regarded as able and hard working. However we note that the activities and performance of the Institute (as with the other institutes) is heavily dependent on the skills, vision and energies of a single part-time research leader. (We note for example that all the Best Brain sessions had been facilitated by the outgoing scientific director). This dependence on one person may pose a risk to the organisation, both because of the excessive demands that may be put on that one person and because of the changes in strategy and direction that may occur in the handover from one scientific director to another. The excessive demands are particularly relevant to the two institutes which have a dual mandate of responsibility for their Institute and for representing their pillars across the other 12 institutes and the funding committees. A related issue is what the career incentives are for scientific directors, and whether research leaders might be deterred from becoming or remaining scientific directors because of the perceived lack of benefit for their research careers.

Areas for future focus
The areas nominated by the IHSPR as priorities for the future are clearly important. Particular attention should be paid to areas where there is an opportunity for both international leadership and relevance to Canadian policy and practice including:

Recommendations

The ERT recommends that:

  1. The IHSPR with the support of CIHR as a whole place more emphasis on monitoring the quality and quantity of health service and policy research, the impact of this research on policy and practice and the evaluation of its own initiatives and programs.
  2. The IHSPR continue to refine its strategy to ensure the most effective portfolio of programs. This should include consideration of the number of current initiatives and the most appropriate balance of research focused and KT activities.
  3. CIHR work with the IHSPR to develop additional support for the Institute's theme, including additional funding.
  4. As IHSPR moves beyond broad capacity building, there is a greater focus on the quality and policy and practice relevance of the research outputs. Particular attention should be paid to areas where there is an opportunity for both international leadership and relevance to Canadian policy and practice including:
    1. Further development of the work to make secondary data more available to researchers. Canada has excellent resources and an existing skill base; these data are of fundamental importance to informing health services and policy
    2. The development of approaches to integrate rigorous research into the evaluation of government policies and programs. Canada has a unique opportunity to exploit the natural experiments arising from variations across the provinces and an existing skill base in implementation research, including economics.

Appendix 1 - Expert Review Team

Chair - Professor Sally Redman
CEO Sax Institute
Sydney, NSW Australia

Expert Reviewer – Professor Sally Macintyre
Professor, Division of Community Based Sciences, Faculty of Medicine, University of Glasgow
Honorary Director MRC/CSO Social & Public Health Sciences Unit, UK

International Review Panel – Dr. Chris Murray
Director, Institute for Health Metrics and Evaluation
Professor of Global Health, University of Washington
Seattle WA USA

Appendix 2 - Key Informants

Session 1 – Review of Institute

  1. Dr. Colleen Flood, IHSPR Scientific Director (former)
  2. Dr. Robyn Tamblyn, IHSPR Scientific Director (current)
    Professor, Departments of Medicine, and Epidemiology and Biostatistics
    Faculty of Medicine
    McGill University
  3. Jean Louis Denis, Chair – Institute Advisory Board
    Director, Institut de recherche en santé publique de l'Université de Montréal
    Professor, Department of Health Administration
    Université de Montréal
  4. Dr. Anne Sales
    Associate Professor, Faculty of Nursing
    University of Alberta

Session 2 – Consultation with researchers

  1. Dr. Pat Martens
    Director, Manitoba Centre for Health Policy
    Professor, Faculty of Medicine
    University of Manitoba
  2. Dr. Paula Goering
    Director, Health Systems Research and Consulting Unit
    Centre for Addiction and Mental Health
    Professor, Department of Psychiatry
    University of Toronto
  3. Dr. Bill Hogg
    Director, CT Lamont Primary Health Care Research Center, Élisabeth Bruyère Research Institute
    Professor and Director of Research, Department of Family Medicine
    University of Ottawa

Session 3 – Roundtable with stakeholders

  1. Ms. Pauline Rousseau
    Executive Director, Strategic Planning Branch
    Saskatchewan Health
  2. Ms. Lillian Bayne
    President, Lillian Bayne & Associates
    Former President, Canadian Association for Health Services and Policy Research, Former Assistant Deputy Minister of Health in British Columbia
  3. Ms. Alison Paprica
    Acting Director, Health System Planning and Research Branch
    Ontario Ministry of Health and Long-Term Care
  4. Mr. Dave Clements
    Director, Corporate Planning and Accountability
    Canadian Institute for Health Information
  5. Dr. Ruth Wilson
    Consulting Director, Health Policy, College of Family Physicians of Canada
    Professor, Department of Family Medicine
    Queen's University
Date modified: