Case 12: Consulting Ontario citizens to inform the evaluation of health technologies: The citizens' reference panel on health technologies

Health Quality Ontario (HQO)
Julia Abelson, PhD, McMaster University, F. Wagner, MHSc, University of Toronto, L. Levin, MD, Ministry of Health and Long-Term Care, Head, Medical Advisory Secretariat, Y. Bombard, PhD, Yale University and Memorial Sloan Kettering Cancer Center, F-P. Gauvin, PhD, National Collaborating Centre for Healthy Public Policy, D. Simeonov, MSc, Institute for Clinical Evaluative Sciences, S. Boesveld, MSc, Doctoral student, McMaster University

Introduction

In December 2008, a 14-member Citizens' Reference Panel on Health Technologies (CRPHT) was established to inform the work of the Medical Advisory Secretariat (MAS) and the Ontario Health Technology Advisory Committee (OHTAC), which produces evidence-based recommendations regarding the use of health technologies throughout Ontario's health system. One of the panel's specific goals was to elicit the societal and ethical values that should guide OHTAC in developing its recommendations. The creation of the CRPHT represents an early experimentation with a deliberative engagement method in order to incorporate public values and citizen perspectives into the expert-driven health technology assessment processes.

Methods

Participant recruitment

A civic lottery methodology was used to recruit the panel.¹ Thirty-five hundred Ontario residents were mailed an invitation letter, information sheet and postage-paid response form. From this sample, 163 expressions of interest in panel membership were received, from which 14 panel members — selected by geographic region and stratified by gender and age, with one participant coming from each of Ontario's 14 Local Health Integration Network regions — were selected blindly from the respondent pool. All travel expenses associated with meeting attendance were covered and each panel member received a $65 honorarium after attending each meeting.

Deliberative discussions

The 14-member panel was convened for five one-day meetings at McMaster University in Hamilton, Ontario, between February 2009 and May 2010. At each meeting, panel members participated in structured dialogues covering two broad topic areas:

1. the core values that should guide the evaluation of health technologies — when and by whom should this be done?; and
2. reviews of selected health technologies (e.g., screening for colorectal and breast cancer, serological testing for celiac disease, and personalized medicine) to provide ethics and social values input to OHTAC deliberations.

The technologies selected for discussion at each meeting were jointly agreed upon by the research team, Medical Advisory Secretariat staff and OHTAC members.

Background material about each deliberation topic — including research Summary and media articles, a workbook with topic summaries, and a set of discussion questions — was circulated one week before each meeting. Meetings began with an introduction to each topic, a brief Summary presentation by the meeting facilitator or invited guest expert, and a question and answer session. Following the presentation, large and small groups held either externally- or self-facilitated discussions of questions that had been circulated prior to the meeting. Representatives from MAS and OHTAC attended two of the five meetings to describe the MAS-OHTAC process, to explain the rationale for the creation of the Citizens' Panel, and to answer questions from panel members.

Each of the five CRPHT meetings generated two types of outputs to inform MAS-OHTAC deliberations regarding the evaluation of selected technologies:

1. Qualitative reflections on a set of pre-circulated questions designed to elicit societal and ethical values related to the technology under review.
2. Pre- and post-meeting survey results of panel member rankings of the societal and ethical values central to the technology under review.

Examples of the "moral" questions reflected on by panel members for each technology included (but were not restricted to) considering how (or does) the technology:

• demonstrate respect for religious, social or cultural convictions?
• affect the distribution of health care?
• challenge patient autonomy?
• ensure that human dignity is respected?²

Despite slight variances across technologies, "universal access," "choice," and "quality care" were identified as core values that should be considered in both the evaluation of health technologies and the ensuing recommendations.³

A thematic Summary of panel discussions was prepared in either report or presentation format following each panel meeting. It was then shared with OHTAC members for discussion at monthly OHTAC meetings.

Outcomes and impact

While tangible policy impacts are rarely observed in time-limited studies of this kind, there were some impacts observed over the course of this two-year project. Since it used social values and ethics data to inform its work, the CRPHT provided new insights in these areas to MAS and OHTAC. Panel input was also used explicitly in OHTAC deliberations for three of the five technologies reviewed.

Based on direct reports back to the CRPHT about how participant input was used, most panel members felt that their input was valued by OHTAC. Attendance of senior MAS/OHTAC officials at two CRPHT meetings, combined with the open invitation for CRPHT members to attend an OHTAC meeting so that they could provide their input on the value of the panel model, instilled further confidence among participants about the panel's perceived value to the two organizations. Other panel members believed the CRPHT's work had minimal influence but felt it was an crucial step to increasing the awareness of MAS and OHTAC about the importance of incorporating public values into these types of decisions.

Lessons learned

Panel members rated the citizens' panel structure and organization very highly in all post-meeting evaluations. The adequacy of time provided for appropriately detailed discussion around certain topics was viewed slightly less favourably than other panel features.

A debriefing session was held at the final CRPHT meeting, followed by telephone interviews to allow panel members to reflect more deeply on their experiences (summarized in Table 1). Panel members unanimously called for the panel to continue and felt that it was just beginning to hit its stride.

Several specific suggestions were made to improve the panel's structure, processes and impact:

1. Increase the frequency of meetings, and consider alternative (and additional) communication vehicles.

A shorter gap between meetings (no more than 2 months) was recommended to make it easier for panel members to re-immerse themselves in the material and to improve how the CRPHT's functioned. Face-to-face meetings could continue to be used as the dominant interaction medium, but additional media such as the internet, teleconferences and video conferencing also should be explored as a way of complementing face-to-face meetings.

2. Ensure adequate attendance at each meeting, and expand the size and reach of the panel.

Panel members expressed concerns about ensuring both a critical mass of participants and balanced representation across age and gender (minimum) at each meeting. Ultimately, this yielded several suggestions:

• Alternates for each of the 14 panel members should be selected, or the panel's size should be expanded.
• The work and reach of the panel should be complemented by surveys, mini-panels or focus groups on selected topics.

3. Improve the clarity of roles and activities for CRPHT.

A clearly articulated purpose for these types of panels is essential, and it needs to be re-affirmed and revised periodically. This ensures that the panel reflects the thoughts of panel members and sponsors about how best to use it, but it also clarifies the panel's role relative to the sponsors, ensuring that it contributes meaningfully to their deliberations.

4. Clarify and strengthen the supporting infrastructure of CRPHT and the interface between CRPHT and sponsoring organizations.

An adequate infrastructure for the panel was viewed as critical to sustaining it over time. More specifically, a number of roles need to be formalized and nurtured to establish an effective and efficient interface, including:

• content expertise to inform the preparation of materials for CRPHT meetings and relevant technology-specific discussions; and
• a strong liaison functionto ensure that potential topics for CRPHT meetings are identified in a timely manner and that input from the citizens' panel is effectively integrated into MAS and OHTAC deliberations.

Date modified:

2013-11-18