Deliberative Priority Setting – a CIHR KT module

by Sandy Campbell

June 2010

Author correspondence can be directed to: sandy.campbell@gmail.com. I am grateful to Jonathan Lomas, Lydia Kapiriri and Pierre Ongolo-Zolo for either commentary or discussion on previous drafts of this Module.

[ PDF (1.29 MB) ]

Section One: An Introduction
Section Two: Priority-Setting Processes
Section Three: Deliberative Processes
Section Four: Priority Setting and the Deliberating Stakeholder
Section Five: References
Notes

Section One: An Introduction

1. Knowledge Translation

In theory, researchers and policy-makers both work to the same end of addressing societal ills and solving collective problems. In practice, researchers and policy-makers are often pitted on opposite sides of a great rift – the "know-do" gap, where what a society knows and what it does misalign.¹ Working towards scientific truth, researchers often see policy-makers as unable to understand or apply their work. Working towards consensus, policy-makers often see researchers as just another group trying to influence their decisions (Choi et al 2005).² Only rarely do their vision and needs converge.

Since its star turn at the 2004 Ministerial Summit in Mexico, the concept of knowledge translation (KT) has emerged as a leading approach in narrowing this rift between research and policy.³ Often misunderstood as a technique to transfer research findings directly to policy – of putting research into the "right language" to ensure its uptake, as a coin animates a pinball machine – KT is more properly imagined as a meeting and a marrying of the research and policy processes.⁴ This definition of KT gets to the very fundamentals of the research and the policy processes, treating both as non-linear, richly complex, continually evolving, and often opaque to the outsider, with roots that reach far deeper than any particular product (a research paper; a policy statement) or a stand-alone event (Lomas 1997).

In its efforts to blend these processes, the ultimate goal of KT is for each to influence the other. The greater the degree of mutual understanding – of researchers understanding policy and of policy-makers understanding research – the better the chances of research informing policy and of policy informing research.⁵ Above all else, KT envisions a constantly turning cycle of policy-informed research leading to evidence-informed policy which, in turn, creates more policy-informed research. And it does this through strong relationships and open dialogue, through trust, respect and the achievement of shared goals (Ginsburg et al 2007, Innvaer et al 2002, Jackson-Bowers et al 2006, Dobbins et al 2007, Golden-Biddle et al 2003). KT is a fundamentally social process.

2. Priority Setting

Priority setting represents the first movement of this KT cycle. By identifying policy needs and research options, priority setting processes unite policy-makers and researchers before the research begins. A singular KT tool, priority-setting processes ask: what are the research needs of policy-makers? And how can those needs inform and shape the research agenda?

There are two major types of priority-setting processes: priority setting for research (determining, weighting and ranking specific research topics and/or research questions) and priority setting for service delivery (determining, weighting and ranking the interventions a health care institution offers – also called "rationing" or intervention priority setting). There has been much more work and experience in priority setting for service delivery, with the literature strongly reflecting this imbalance. Notably, there is a strong cross-over between these two types of processes, but they are notably distinct and to date there has been very little attention in the literature to exploring these synergies. While we have included here much of the thinking, logic and examples from the priority-setting for service delivery literature, our focus is primarily on research priority setting – at the national level (e.g. setting priorities for a national research agenda), and at a theme- or issue-based level (e.g. prioritizing and ranking different researchable issues or research questions) – with a further focus on low- and middle-income countries (LMICs). This Module does not fill the literature gap linking together these two major types of priority-setting processes, but does attempt to make some models of priority setting for service delivery (e.g. Programme Budgeting and Marginal Analysis) relevant to priority setting for research.

Whatever the type, though, priority setting is by its nature a philosophical and economic act. It is philosophical in the questions it poses – how does a society decide which topics are urgent, relevant and under-researched? – and economic in the guidance it can offer investors – how does a society align its financial resources with its knowledge needs? Ultimately, however, priority setting is a political act. Even the term itself – priority setting – carries with it a rigid political finality: one has assessed a society's current weaknesses, anticipated its future needs, and decided what needs to be done, in what order and to what effect. For most societies, this kind of decision-making is a very high-level process that an elected representative (a Minister of Health, for instance) may see as their own distinct purview.⁶ No matter how inclusive or representative a priority-setting process may determine itself to be, the mechanism of where and how a society's needs are identified and how its resources are allocated is a definite political act. To that end, the very term of "priority setting" may need to be re-imagined. Throughout this Module, we argue that what is important is less the actual setting of priorities (situating or fixing them) and more the deliberating around those priorities (where an inclusive, representative group gather to discuss collective needs). It is the process and not the product that ultimately matters (WHO 2003). Dialogue may not solve a problem or set actual priorities, but it will build the social relationships, trust and interactions critical to KT and any health system.⁷ Identifying priorities and discussing them is the heart of the matter. Though we use "priority setting" throughout this Module in recognition of the accepted terminology, a better term capturing this spirit of deliberation and interaction may be "priority identification".

In its ideal interpretive form, priority setting selects the right people to brainstorm on the right issues to determine what a society's, a system's, or an institution's priorities are.⁸ It is a transparent, fair, legitimate and accountable process designed to guide decisions, a rational means to determine how resources are invested to address societal needs and to steer researchers towards topics of "national interest and priority" (Lenaway et al 2006).⁹ Within such political and philosophical dynamics, there are a number of different priority-setting approaches, exercises and processes. Each is typically distinguished by level (global, national, institutional), comprehensiveness (research priorities for the health sector; research on specific issues; determining precise research questions), the balance between technical and interpretive approaches, and stakeholder involvement (Ranson and Bennett 2009). Each is a complex, value-driven process.

By far the most cited framework in providing and assessing the ethical complexities of priority setting is that of Daniels and Sabin (1998, 2000, 2002, 2008). Their "Accountability for Reasonableness" (AFR) framework for priority setting describes a "fair process" that allows stakeholders to determine the legitimacy, fairness and context-specificity of values and priorities that ought to combine and ultimately inform decisions. AFR establishes a moral underpinning that emphasizes the principles of democratic deliberation (Bruni et al 2007; Kapiriri and Martin, 2007). It prescribes four necessary conditions for any priority-setting process:

it must be relevant to the local context as determined by accepted criteria
its eventual decisions – and the reasons behind them – must be publicized;
it must include appeal mechanisms for challenging, revising, and reversing decisions;
its leaders must be able to enforce the above three conditions.¹⁰

AFR rotates around the principles of social justice, and as such describes the idealized "value core" of priority setting processes (i.e. fair, just, deliberative, legitimate, transparent, accountable, enforceable). However, as Sibbald et al (2009) explain, because priority-setting processes depend upon the adjudication between those values, they set the stage for conflict as stakeholders will necessarily and fundamentally disagree about the importance of different values. And precisely because of this constant, context-driven adaptability, there can be no normative approaches (Menon, Stafinski and Martin 2007) nor agreement on the required elements for a successful process – despite the desire of some authors to find a replicable algorithm that can rank and weight priorities in any given situation (Rudan et al 2007a, Baltussen and Niessen 2006).¹¹ While some priority-setting processes will involve a relatively simple judgement on whether an issue or argument advances equality of opportunity for a given population (Hasman and Holm 2005), in many others the process will generate more questions than answers.

Given the context-specificity of values, arriving at criteria for a specific priority-setting exercise is a crucial step (Working Group 2000; COHRED 2006). There are many ways to determine criteria, from considering how any given issue correlates to a laundry list of philosophical concepts to responding in realpolitik fashion to the desires of those with power and influence (Rudan et al 2007a; Baltussen and Niessen 2006). In general though, there seem to be a set of criteria that are increasingly useful and used, either for determining a research agenda or for ranking competing issues. This criteria includes attention to:

magnitude, relevance and urgency – will the research be needed within the next 3-5 years? (El-Jardali 2010; Working Group 2000). Will the issue at hand still be a priority in five years' time? (Alliance for Health Policy and Systems Research 2009).
applicability, deliverability, affordability, sustainability (Rudan et al 2007a)¹²
maximum potential to reduce disease burden equitably (Rudan et al 2007a; Baltussen and Niessen 2006)
originality – is the issue unresearched or is there an existing knowledge base? (El-Jardali 2010, Working Group 2000).
research capacity and feasibility: do the skills exist to do the research? Is it feasible financially, technically, socio-culturally and ethically? (Working Group 2000; COHRED 2006; Rudan et al 2008).
policy relevance (El-Jardali 2010); and
expected impact of the research – will its impact be greater than its relative cost? (COHRED 2006).¹³

3. Priority Setting and the "Right People"

No matter the criteria, the priorities arrived at depend upon those consulted (Commission on Health Research for Development 1990). Slum dwellers will have differing priorities than physicians (Sitthi-amorn 1989 cited in Lomas et al 2003), and researchers typically have different ideas than policy-makers. But what of the needs and preferences of other stakeholders, including funding agencies, health care practitioners, "street-level bureaucrats" within the health system, civil society and/or the private sector?¹⁴ After all, for many authors, it is the very involvement of an inclusive set of stakeholders that confirms priority setting's attractiveness as a vehicle to entrench and enrich democracy (Charles and DeMaio, 1993; Abelson et al, 2003; Bruni et al 2007; Bruni et al 2008; Mshana et al 2007; Martin, Abelson and Singer 2002). The involvement of multiple stakeholders confers "legitimacy and fairness" (Bruni et al 2008), evidence of a society openly debating and advancing the social-justice values necessary for a strong health system, while also allowing decision-makers to gauge whether their perceptions of constituent needs are in fact accurate (Smith et al 2009).

Balancing the different voices within a priority-setting process is a certain challenge. Most of the literature on this focuses on the role of the wider public (Working Group, 2000), studying whether the public's voice has been co-opted by the "professionals" (Abelson et al 2003), muted by louder wheels squeaking for the grease (Teng et al 2007), or ignored by those with real policy-making power (Wait and Nolte, 2006, Rudan et al 2007a). The priority-setting literature on service delivery is much more concerned with public involvement as the public is in fact the primary receiver of those services and thus deserves a hand in how those services are prioritized and ultimately delivered. When it comes to research priority-setting, however, the public may be more effectively represented by civil society organizations or global entities (e.g. WHO, UNICEF) with the requisite expertise and experience in research and the research community who can engage meaningfully in the often technical nature of the deliberations. This is not to downgrade or disregard the public voice but to recognize the inherent specializations within health research and the training required to identify and analyze knowledge gaps, needs and priorities. Such issue specialization, however, demands even closer attention to balance – since, as Kapiriri et al (2007) observe, "the values and criteria important to scientists and technical experts may vary remarkably from those of other relevant stakeholders".

4. Priority Setting as a Deliberative Dialogue

From a theoretical perspective, the concept of the deliberative dialogue offers a variety of ways to involve multiple stakeholders in any decision-making process and to balance their different opinions, values, needs and criteria. We define a "deliberative dialogue" here as a process of collective and procedural discussion where an inclusive and representative set of stakeholders consider facts from multiple perspectives, converse with one another to think critically about options, and through reasoned argument refine and enlarge their perspectives, opinions and understandings.¹⁵ A deliberative dialogue creates joint meaning and shared understanding (Franco 2006), a unique and effective "problem-structuring tool" (McDonald, Bammer and Deane 2009) that represents the soul of democracy itself (Dryzek 2000). Unlike any other mechanism, a deliberative dialogue adds both scientific and social credibility to the decision-making process, as it unites and empowers those who will be affected by the eventual decision (CHSRF 2006). A deliberative dialogue can open new windows of understanding, particularly for those whose views are not yet fully formed or informed (London 2005).

A deliberative dialogue is not intended to solve problems (Lavis 2009a) but rather to create open discussion on the nature of the problem itself, working to reveal the true complexity of the problem (Abelson et al 2003), or to frame it within its wider, systemic framework. In short, it is not in itself an end goal but rather a point within a much longer process of decision-making – much as priority setting is simply one point in the much longer process of creating and applying user-informed knowledge.

The literature on multi-stakeholder deliberative dialogues suggests careful consideration of how each stakeholder is selected, and how each stakeholder actually functions in the dialogue. Of these two elements, there is little evidence about what works in selecting participants (Abelson et al 2007). Rowe and Frewer's (2005) typology of public engagement mechanisms makes rough divisions between dialogues that have no control over who is selected to participate (e.g. public hearings, study circles, hotlines, open spaces, town meetings – where participants self-select) and those that do have some degree of control over who participates (e.g. opinion polls, consultation documents, citizen panels/juries, consensus conferences, task forces, planning cells).¹⁶ Beyond descriptions and case studies of these dialogue types, however, there is simply little evidence for what works in multi-stakeholder processes, including the impact the deliberative processes have on those stakeholders, on decision-makers and on decision-making; or or "how these processes are shaped and constructed by the different contexts within which they are implemented" (Abelson et al 2007).¹⁷

When it comes to the function of stakeholders in a deliberative dialogue, there is a similar evidence gap. If all relevant stakeholders participate in a deliberative dialogue, will they all have the same ability to contribute? Given the fundamentally technical core of a priority-setting exercise, how does a deliberative dialogue handle differing levels of expertise? Clearly, there is much to understand and evaluate, particularly in assessing the links between priority-setting processes and decision-making (Smith et al 2009), and between stakeholder involvement and the types of priorities identified (Ranson and Bennett 2009). Helpfully, Abelson et al (2003) suggest four components to apply in evaluating any deliberative process, which we might certainly advance (along with AFR) for evaluating any priority-setting process. These include representation (what demographic, geography or political affiliation did participants represent?), procedures (were the procedures governing the dialogue open, reasonable, understood, responsive, legitimate and fair?), information (what information was selected, presented and interpreted?), and outcomes (was multiple stakeholder input incorporated into the decision-making process?).

In Section Three of this Module, we discuss new ways of identifying and selecting the "right" stakeholders; in Sections Two and Three of this Module, we discuss means to involve these stakeholders in the dialogue, observing the different functions each might play.

5. Priority-Setting Processes at the National Level

Of central concern in this Module is priority setting at the national level, particularly as a function of a national health research system in low- and middle-income countries (LMICs). While the literature here is relatively scarce, there is nonetheless a strong global enthusiasm for creating priority-setting processes in LMICs as part of an overarching governance/management system for health research, partly in recognition of the high value of research in addressing LMIC issues, and of the need to target that research as accurately as possible (Ghaffar et al 2004, Nuyens 2007). Unfortunately, the literature reveals very little about actual LMIC priority-setting processes, focusing instead on the many challenges to implementing any sort of priority-setting process. Such challenges include: the general dearth of available information; disconnections between who does and who should set priorities, and the values that do and should set priorities (Kapiriri and Martin 2007); how national priority setting could inform sub-national levels and how those sub-national levels might adapt national priorities to their own needs (Nuyens 2007); how the process itself may affect identified priorities (Makundi, Kapiriri and Norheim 2007); how to move priority-setting from the realm of researchers to a broader composition of stakeholders (Kapiriri et al 2007, Rudan et al 2007a, Rudan et al 2007b, Tomlinson et al 2007); how to push the discussion past disease-specific issues to the systemic (el-Jardali, 2010, Walker et al 2009); and how to move beyond the simple cost-effective analyses that have long dominated LMIC priority-setting initiatives (Mshana et al 2007). All of these challenges are of course punctuated by governance institutions often too weak to make lasting or enforceable decisions – or to consider appeals of its decisions – and often subject to the very real and considerable pressure of bilateral and multilateral funders to align with their own perception of priority.

"Paradoxically, it is the international donor community which has helped governments avoid making difficult resource-allocation decisions. Donor agencies have traditionally financed and supported separate projects based on their own priorities. These priorities may be expressed in terms of particular geographic areas, particular diseases, particular levels of health care (e.g. primary care and public health, rather than hospital services), particular types of interventions (e.g. immunization or contraceptive supplies), particular management systems (e.g. drug supply), or particular parts of the population (e.g. pregnant women, children under five). This has to some extent undermined aid-receiving governments' ability to develop coherent sectoral policies and allocate resources accordingly".

– WHO 2003

On top of the above LMIC concerns, incorporating the views and voices of multiple stakeholders to make priority-setting processes "fair and legitimate" is an additional challenge. The Working Group (2000) suggests "sequential patterns of consultations, analyses and priority setting at the community and district levels, and aggregation and synthesis of these outputs at the provincial and national levels" going through several iterations to achieve consensus. However, despite the rubric of many LMIC health reforms to advance a participatory, multi-stakeholder process in health policy, this type of participation depends upon a genuine willingness of decision-makers to cede some of their power (Wait and Nolte, 2006), which is often better phrased on paper than seen in action. As Nuyens (2007) observes, "the strategies and methods necessary to achieve a participatory process have not been developed or implemented systematically. There is no body of knowledge about how and when various stakeholders should be involved or what their functions and responsibilities should be….priority setting has often been handled as an isolated, one-off event that has little impact on, or relation to, other essential components used to construct a health research system."

In applying the AFR framework to Uganda, Norway and Canada, Kapiriri, Norheim and Martin (2007) find no obvious mechanisms for achieving publicity, relevance or revisions of priority-setting processes, with no apparent leadership for ensuring a fair process. Likewise, Sabik and Lie's 2008 study of national (Northern) processes in the Netherlands, Sweden, New Zealand, Oregon, the UK, Denmark and Norway reveal that the act of priority setting had a negligible effect on the actual issues that had been prioritized. They, as others, conclude that priority-setting processes at the national level – in countries of any income level – require strong decision-making abilities to determine, oversee and enforce what is (or is not) eventually implemented (Sabik and Lie 2008, Peacock et al 2006, Rudan et al 2008), and also to ensure that the right stakeholders are at the table.

Despite the lack of evidence or evaluations at the national level, there are some strong recommendations in the literature for the way forward. These range from countries creating advisory panels of key stakeholders who are first trained in the principles of priority setting and can then assist various bodies in its execution (Peacock et al 2006) to finding a priority-setting algorithm or model that can be safely adapted or replicated country-by-country (Rudan et al 2007a). Perhaps the most important contribution to the field at this time though is outright documentation of LMIC experiences in setting priorities, with particular attention to the actual criteria developed to assess respective research priorities. Martin and Singer (2002) advance this as a describe-evaluate-improve strategy: describe priority setting in its context; evaluate it using an ethical framework like AFR; and use the evaluation to improve the act of priority setting. To this we would add a fourth component: disseminate. As it will be some time before the formal evidence base documents LMIC experience, we nonetheless need to learn from the many priority-setting projects and experiments underway – to adapt, apply and ultimately refine these crucial deliberative acts.

6. An overview of the Module

In Section Two, this Module considers individual priority-setting processes and exercises in greater detail. While each process is not necessarily an interpretive priority-setting process for research, we will examine each for its relevance to (and lessons learned for) this type of process. This includes an overview of the "ideal" interpretive priority-setting process; participatory priority-setting methods; Listening for Direction (LfD); the Combined Approach Matrix (CAM); and Program Budgeting and Marginal Analysis (PBMA). The Module reviews the literature on each, presents their basic functions, and discusses their logical flow, strengths and limitations.

Section Three describes and analyzes key deliberative dialogue concepts, from ways of identifying and analyzing relevant stakeholders (including different types of stakeholder analysis tools) to means of eliciting multi-stakeholder input (e.g. Concept Mapping, the Delphi Technique and the Nominal Group Technique) to the types of deliberative dialogue mechanisms with high relevance to priority setting (e.g. workings of a Technical Working Group and a Larger Reference Group).

Section Four features some observations and conclusions on how to better fuse deliberative techniques with any priority-setting process. This includes remarks on implementing priority-setting processes in low- and middle-income country settings; areas for future research; how institutions and systems (eg national health research systems) can increasingly inform their priority-setting work with deliberative techniques; and some key evaluation considerations.

Section Two: Priority-Setting Processes

1. Introduction

There are a number of different priority-setting processes, with each typically distinguished by level (global, national, institutional), comprehensiveness (research priorities for the health sector; research on specific interventions), the balance between technical and interpretive approaches, and stakeholder involvement (Ranson and Bennett 2009). In Section Two, this Module shifts into very specific consideration of five different priority-setting approaches. This includes an overview of an "ideal" interpretive priority-setting process; participatory priority-setting methods; the Canadian Health Services Research Foundation's Listening for Direction (LfD) model; the Global Forum for Health Research's Combined Approach Matrix (CAM); and Program Budgeting and Marginal Analysis (PBMA).¹⁸ In this section, we'll review the literature on each, provide an overview of their logical flow with flowcharts and graphs, and provide a brief discussion of their strengths and limitations.

2. Ideal Interpretive Priority Setting

In its ideal interpretive form, priority setting selects the right people to brainstorm on the right issues to determine what a society's, a system's, or an institution's research priorities are.¹⁹ While there is little consensus on what marks a "successful" priority-setting process (Sibbald et al 2009), we can nonetheless identify, from a theoretical perspective, an iterative list of seven ideal elements within a priority-setting process for research that satisfy Daniel's (2000) Accountability for Reasonableness conditions – one that is fair, legitimate, relevant, public, subject to appeal, and enforceable. Figure 1 below outlines the seven steps of this ideal interpretive priority-setting process.

Figure 1: The Seven Steps of the Ideal Interpretive Priority-Setting Process²⁰

2.1 Planning the Process

This step is crucial in anticipating the needs, barriers and challenges for all of the remaining steps. At this planning stage, the process needs to identify leadership, with particular attention to the central individuals and institutions. This may coalesce into a "Technical Working Group" (TWG) (variously called a Senior Advisory Team, a Council of Elders, etc.) of individuals representing major institutions. The TWG will need to determine the timing of the process, its logistics, budget, dissemination plan, etc., without unduly influencing or corrupting the many details.²¹ Members of the TWG may need to be identified using some of the stakeholder analysis techniques in step two below (and in Section Three).

2.2 Stakeholder identification, analysis and engagement

As discussed in the Introduction, few priority-setting processes pay sufficient attention to how their stakeholders are actually selected: how do the right people get around the table? How can a priority-setting process ensure a balanced representation of different institutions, interests, and levels?. As discussed in much greater detail in Section Three, a stakeholder analysis is a critical tool in determining who should sit at the table. It can help the leaders of a priority-setting process identify:

the importance, influence and interests of stakeholders in relation to the research agenda or particular research theme/issue. This could include identification of both primary and secondary stakeholders.
the layers and levels within complex stakeholder institutions – e.g. entities like a Ministry of Health, a community, or the World Health Organization can all be treated as a single or as a multiple actor depending upon the nature of the issue at hand. One institution may in fact turn out to be several different stakeholders.
the "level" of stakeholder to involve – local, regional, national, and/or international.
potential conflicts, conflicts of interest, or risks that could jeopardize the process;
opportunities and relationships that can be built on during the process;
groups that should be encouraged to participate in different stages of the process; and
appropriate strategies and approaches for stakeholder engagement (WWF, 2005).

Performed by the Technical Working Group (TWG) (or a consultant hired for the purpose), a stakeholder analysis may well identify members of the TWG itself, and potential members of a Larger Reference Group, with each acting as a filter and adding layers of context and legitimacy to the process. The TWG may also complete or commission a situation analysis as a way of further understanding the actors relevant to a domain, theme or question; the actual information needs of the process; and/or the relevant aspects of the overall health research system.

2.3 Knowledge management

The flow of knowledge throughout the process is also of critical importance. Mechanisms to guarantee a circular flow must be in place from the outset to ensure that all stakeholders have the same information, with provisions made to ensure that the less technically skilled participants can interact with the more technically advanced input. Before the process begins, a range of relevant information must be made available to stakeholders (e.g. systematic reviews, peer-reviewed and grey literature, national health research policy documents). The availability of knowledge – and the knowledge selected to be available – has a direct influence on the priorities chosen, and thus there must be a balance of the "rigorous with the spontaneous" (Working Group 2000).

Clearly, any priority-setting process requires the development of a knowledge management strategy to satisfy the above concerns and needs. Such a strategy should coalesce around three broad areas:²²

what knowledge and information needs do the stakeholders in the process have? Do they all have a similar educational background? Can they all read the same language? Do they all have similar access to knowledge resources?
how will knowledge influence the priority-setting process? Where will it need to be introduced in the process? How will the leaders ensure that information is available to stakeholders during and after the process? How will it be disseminated to ensure the transparency and accountability of the process?
how should people, processes and technologies – in terms of specific tools and practices – be used? How will people share and access knowledge and information?

2.4 Interpretive workshops

This may be one workshop among a select group of stakeholders (i.e. the TWG), or may be several; it may also be a sequence of workshops among different groups or levels of stakeholders. In any case, an interpretive workshop sees stakeholders gathering to discuss criteria for assessing priorities; to establish the means to weight or rank the criteria; to apply the criteria to identified priorities (Working Group 2000); or to determine broader health research options (Rudan et al 2007b). These workshops are open, deliberative and iterative – conflicting priorities may trigger some consensus-building techniques, or the leaders may decide the nature of the conflict to be an important element that need not be resolved. Workshops may flow from:

reactions to the pre-circulated material (which may include a comprehensive list of priority research issues or questions)
a brainstorm on the big, pressing or contentious research issues or questions
a discussion of criteria that should be used to rank the research issues or exact questions
agreement on a formal set of criteria, and means of weighting that criteria (i.e. some may be more/less important than others)
applying that criteria to research issues or questions to arrive at a scored or ranked list of issues or questions
a discussion of the types of research that could best illuminate the issues or questions.

Note the three possible levels of deliberation. The interpretive workshops may discuss particular research issues (e.g. malnutrition), a specific research question (e.g. the role of Vitamin A supplementation coverage in three regions in Tanzania in reducing nutritional deficiencies) or a broader health research option (e.g. health system barriers and opportunities for increasing Vitamin A supplementation coverage in Tanzania).

2.5 Translation of issues into research questions

While some interpretive workshops may discuss precise research questions, it's more likely that they will identify policy concerns, research issues and/or research options that may then (context depending) be translated into actual questions. The TWG should lead this process, and may then wish to convene another interpretive workshop to discuss the list of questions and their respective priority.

2.6 Publication and validation

The final list of ranked issues, questions and/or options must now be made available to a larger group of stakeholders (a Larger Reference Group (LRG)), ensuring that the list aligns with the values and needs of all concerned, with clear definitions of the expected outcome of its work – from broad lists of priority health (system) problems or issues to a detailed list of priority research questions. The TWG should provide each LRG member with all necessary information sufficiently ahead of time. The TWG will then lead the LRG meeting itself, which might be initiated virtually but will conclude either with a complete list of validated priorities or with a set of criteria ranked for a specific context (i.e. rather than determining precise priorities, the LRG validates which elements (e.g. equity, maternal-child health) is of the greatest concern to them). Ballots and voting may be the easiest process for arriving at some sort of consensus on criteria ranking or priority listing.

2.7 Revision or appeal mechanism

Following publication and larger discussion of the list of priorities, there needs to be a mechanism in place to address disagreements comprehensively and constructively. Such a revision process would "1) improve the quality of decisions by providing opportunities for new information to be brought forward, errors to be corrected, and failures in due process to be remedied; and 2) operationalize the key ethical concept of responsiveness" (Sibbald et al 2009).

Figure 2 below provides a visual representation of some of the barriers and facilitators at different stages of this process, adapted here for the ideal interpretive model.

Figure 2: Facilitators and Barriers to an Ideal Interpretive Priority-Setting Process²³

3. Participatory Priority-Setting Methods

Recognizing the dearth of viable and tested priority-setting mechanisms for the national level, particularly in low- and middle-income countries, recent research and experience have focused on a new set of approaches – most are variants on the above ideal interpretive process. Involving a range of stakeholders, these approaches identify existing policy concerns and research priorities; assess the extent to which current research addresses these concerns and priorities; develop a list of core priorities for future research; and then widely disseminate those priorities for discussion and revision. Such approaches have led to context-sensitive, evidence- and stakeholder-informed priorities, and critically revitalized the act of priority setting at national, regional and even global levels with particular relevance for health systems research and for strengthening national health research systems.

In this section, we'll explore specific examples of participatory priority-setting models and lessons from the regional level (sub-section 3.1); on a particular global health research issue (sub-section 3.2); at a country level (sub-section 3.3); and for a national-level institute (sub-section 3.4). It is important to note that each of these models has a different number of "steps" in moving from the start to the finish of the process. This reflects the ability of each exercise to tailor the process to its specific needs – in some processes, for instance, stakeholders may be more difficult to identify or to convene, and may need to be treated as discrete groups.

3.1 Participatory Priority-Setting at the Regional Level: Latin America, East Asia, South-East Asia and the Middle East/North Africa

The WHO's Alliance for Health Policy and Systems Research has pioneered this method of priority setting in three health-research themes – the role of the non-state sector; financing; and human resources for health – involving twenty-four LMICs in four regions (Latin America, East Africa, South-East Asia, and the Middle East/North Africa).

The process featured key-informant interviews, literature reviews and thematic papers all done by four different LMIC organizations. With this regional identification and synthesis of existing policy concerns and research topics, a validation and ranking consultation workshop of regional experts created criteria for sorting those concerns and priorities, and ranked them according to the following questions: a) can the research question be answered? b) how large is the impact on social welfare likely to be? (intended to include both health and equity impacts); and c) is there a lack of research on this topic? They then discussed in some detail the kinds of research that could best address the four resulting research questions that ranked highest. Dissemination of the results and products of the validation workshops ensured that the "loop closed" with policy-makers receiving priority lists in answer to their initial concerns.²⁴

From a meta level, Figure 3 below demonstrates the four key steps in this approach. Figure 4 shows detail on a specific regional process.

Figure 3: The Four "big-picture" Steps in Participatory Priority-Setting Methods at the Regional Level

Figure 4: The Regional Priority-Setting Process in the Middle East and North Africa²⁵

3.2 Child Health and Nutrition Research Initiative

One particularly innovative participatory priority-setting process was led by the Child Health and Nutrition Research Initiative (CHNRI) (Rudan et al 2008; Tomlinson et al 2007; Rudan et al 2007b) and used for the first time at a country level in South Africa in 2006. As a participatory priority-setting process, it made two major contributions to the field: by concentrating less on the creation of new knowledge and more on the "health research option" and its "vision of implementation" (Rudan et al 2007b); and by focusing on ways of incorporating societal values and principles through an LRG. The CHNRI method is a hybrid of technical and interpretive approaches, principally by ensuring that technical experts provide opinions and input anonymously, with the end ranking of each research option done in an open, systematic fashion for transparency, replication and validation (Rudan et al 2008).

As Rudan et al (2008) comment, the CHNRI priority-setting methodology results in seven strong outcomes: a better understanding of the context; an agreement on expectations and acceptable risks; a definition of main criteria for priority setting; a listing of the many different research investment options; a transparent assessment of each option; adjusting each options according to social values; and combining this adjustment with "predicted cost, expected profits and risk preferences to decide on the optimal investment strategy" (Rudan et al 2008). It is systematic, it limits the influence of technical experts, has a final quantitative output (research priority score), can evaluate and rank different types of research and, critically, through the involvement of the LRG "incorporates an efficient means of considering the voice of stakeholders and the wider public, who are given the power to place thresholds and weights upon intermediate scores (which are based on the collective opinion of technical experts) and in this way considerably shape the final outcome" (Rudan et al 2007b).

In 2006, this methodology was tested for the first time at a country level, in South Africa (Tomlinson et al 2007). Figure 5 shows the steps employed during that exercise.

Figure 5: Setting Priorities in Child Health Research Investments for South Africa²⁶

3.3 Mongolia

In Mongolia, a currently unfolding priority-setting process intends to guide future investments in health research with particular attention to the intersections of health and mining. The process is in the midst of generating an agreed-upon, multi-stakeholder research agenda through expert discussion, analysis and the creation and application of criteria to weight and rank research questions. This agenda will be actively disseminated, with a mix of public and private funds supporting investigation into the top-ranked issues and questions. Importantly, it will involve and address the needs of Mongolian policy-makers, researchers, and communities. Figure 6 provides a graphic demonstration of this process.

Figure 6: Setting Priorities in Mongolia: Mining and the Health System

Beyond the creation of scientific knowledge on pressing priority areas, expected "by-products" of the process, include:

new ownership and participation in the health research system. The expert group convened for the process could take on other important (and often unfilled) roles within the health research system, including coordination, monitoring and becoming an ad hoc health research committee providing expertise and insight to the policy process.
Identification of priorities could reveal in detail the blockages in coordinating research and networking among principal actors, leading to new perceptions, understandings, projects and initiatives (e.g. creation of a national coordinating body; creation of a national network of health research and researchers; creation of new databasing and knowledge management strategies).
Beyond identifying knowledge gaps in Mongolia, the process could also reveal the types of "best practice" and "lessons learned" from other contexts that could directly inform Mongolian perspectives (e.g. the interplay among mining, corporate social responsibility and health systems in other middle-income countries – South Africa, Chile, Colombia).
Discussing research as an input into the policy process – and as an input into solving socio-economic problems – reinforces the perception of research as a valuable resource.

3.4 The National Institute for Occupational Safety and Health

Slightly different than the above models, the National Institute for Occupational Safety and Health (NIOSH) in the United States developed their own participatory priority-setting process (Rosenstock et al 1998). This was a consensus-building process revolving around input from working groups and open town meetings to arrive at final research priorities. With the working groups – one comprised of researchers, one of other stakeholders, one of health professionals, and one of internal NIOSH staff – identifying initial priority areas, an iterative process whittled this list down to 23 topics, using criteria on the seriousness of hazard, numbers of works exposed, potentials for reducing risk, etc. Public town meetings then contributed to and further refined this "master list" of priorities for a National Occupational Research Agenda (NORA), with comments and criticisms leading to the development of a public process for shaping the research agenda. Figure 7 shows how the steps link one to the next.

Figure 7: Determining a National Occupational Research Agenda²⁷

4. Listening for Direction

Pioneered by the Canadian Health Services Research Foundation (CHSRF), Listening for Direction (LfD) is a strong interpretive priority-setting process that does not deviate greatly from the ideal described above. It creates a situation in which the right stakeholders can present their opinions on the big issues. "Based on the principle of linkage and exchange between research funders and researchers on the one hand and the research's potential users on the other," LfD moves from environmental scan to regional workshops to national workshops "with invited participants, focused on pressing issues that decision-makers expected to encounter in the medium-term (3-5 years)" (Smith et al 2009). It ultimately wants to create and promote the production of timely, relevant evidence to inform healthcare decision-making: "both sides need to listen to each other in order to arrive at research priorities that are relevant to decision-makers, feasible within the research community, and eventually incorporated into practice" (Lomas et al 2003). Figure 8 below provides the graphic progression among the steps.

Figure 8: The Six Steps of Listening for Direction²⁸

"The LfD model builds on the respective areas of expertise of research funders, researchers and research-users. All groups participate in the process with varying degrees of intensity throughout its various phases. Representatives from the partner organizations (primarily research funders whose roles include knowledge transfer and research use) oversee the consultation process, listen to the discussions and, with the aid of expert researchers, use the concerns identified by decision-makers and other research-users to derive research questions. During the consultation workshops, research-users identify priority issues that would benefit from being informed by research. Researchers 'translate' those priority issues and concerns into research questions that are likely to provide evidence to help address those issues…

The CHSRF has tested this model in three different rounds to date. Critically, Listening for Direction II distinguished two sorts of researchable questions from the identified priorities: syntheses of existing evidence – where there was felt to be a sufficient body of knowledge which could be synthesized within 6 to 24 months; and primary research – where new evidence was required which could be produced over the next 2 to 5 years…

For the 2007 LfD process, the objectives were to identify health-system priorities for the short term (one-to-two years) and the longer term (3 to 10 years). These results would be analyzed by researchers who would identify questions suitable for syntheses of existing research, and questions that required original and perhaps longer-term research."

– Law, Flood and Gagnon 2008

As described in Smith et al (2009), LfD was used to bring together a range of "listening" stakeholders in the Canadian province of British Columbia. The organizers followed the below seven steps:

key individuals from each of BC's six health authorities were recruited as research partners.
a priority-setting fora invited these research partners, select decision-maker partners (or their substitutes), and individuals from different sectors and job positions (e.g. finance managers, vice presidents, a CEO).
various priority-setting fora discussed potential research questions.
each fora produced a set of notes capturing the discussion and the context.
a summary of these notes – including potential questions – was circulated to decision-makers to ensure their ideas and opinions were adequately captured.
the study's lead author analyzed these notes "in order to group potential research questions and general comments into a set of overarching key themes; a second author provided additional analysis and review of the findings, while all authors provided comments" (Smith et al 2009).
Following these fora, researchers had face-to-face meetings with decision-makers to discuss these findings and chart out possible areas for partnership.

5. The Combined Approach Matrix

The Combined Approach Matrix (CAM) was developed by the Global Forum for Health Research in 2004. Its objective is to incorporate both economic and institutional aspects into a single tool – a combined approach – for priority setting. The advantage of this tool is that it helps "organize, summarize and present all available information on one disease, risk factor, group or condition, and facilitate comparisons between the likely cost-effectiveness of different types of interventions at different levels. The information may be partial, and possibly even sketchy in some cases, but it will improve progressively, and even limited information is sometimes sufficient to indicate promising avenues for research. The CAM seeks, above all, to summarize the evidence base for particular diseases or risk factors, thus highlighting gaps that require further research and action" (Ghaffar et al 2004). Figure 9 below shows the type of table critical to executing the CAM.

Figure 9: The Combined Approach Matrix²⁹

The Matrix helps to organize and summarize information about a particular disease and the interventions available to combat it. As shown in the above chart, the Matrix categorizes this information according to five "economic dimensions" (e.g. disease burden and cost-effectiveness of interventions) and four "institutional dimensions" representing the various levels at which interventions can be implemented.

"Information gathered in a priority-setting exercise conducted at country, regional and global levels could be introduced into the CAM as a common framework to organize and present the collected information (as a basis to identify gaps in health research and health research priorities). As we can see in filling in the above matrix, the CAM:

brings together in a systematic framework all information (current knowledge) related to a particular disease or risk factor
identifies gaps in knowledge and future challenges
relates the five-step process in priority setting (economic axis) with the actors and factors (institutional axis) determining the health status of a population
permits the identification of 'common factors' by looking across the diseases of risk factors
is applicable to priority setting in the field of:
- national, regional or global problems
- both diseases and risk factors
permits the linkage of priorities in the field of health and health research
enables the rapid identification of the effect of a change in one of the 'boxes' of the matrix on the others
permits taking into account the large number of factors outside the health sector that have an important impact on people's health." (Ghaffar et al 2004).

Figure 10 shows the steps distilled into graphic form.

Figure 10: The Seven Steps of the Combined Approach Matrix³⁰

As various actors note, though, the CAM has some shortcomings, from a non-inclusivity to a disease-driven orientation. "Although it is an extremely helpful tool for gathering and organizing information needed for priority setting, it does not in itself represent an algorithm for making the decisions on the priorities by ranking or separating the competing investment options. Therefore, in the absence of reliable information, which is usually very scarce for developing countries, most of the decisions will still be based on discussions and agreements within the panels of experts" (Rudan et al 2007a). "It is a predominantly disease-driven approach to priority setting that does not serve the health system level well. Many health systems research questions stand to provide benefits for multiple different diseases. Linking health systems research questions to specific diseases, rather than seeing them in total, results in these topics being systematically de-prioritized and contributes to the fragmentation of health systems research" (Ranson and Bennett 2009).

6. Programme Budgeting and Marginal Analysis (PBMA)

PBMA is most often used at the organizational level (typically by service-delivery institutions) to decide among competing services, or to determine a feasible basket of services. "The basic principle is that to do more of some things we have to take resources from elsewhere, by either doing the same things at less cost or reallocating resources from other areas of care. This requires accurate measurement of the costs and benefits of healthcare programmes" (Peacock et al 2006). While PBMA differs from the above priority-setting tools in that its focus is on service delivery, it has been effectively incorporated into priority setting for research – most notably by Tomlinson et al (2007) in South Africa, where PBMA was used to help cost out respective research options. Indeed, this function – of assessing a "true" cost of a question or option – may be the strongest contribution PBMA can make to priority setting for research.

As Mitton and Donaldson (2004) and Mitton et al (2003) explain, PBMA embraces two key economic underpinnings. "The first is that of opportunity cost, which carries with it the understanding that in investing resources in one way, some opportunity for benefit, through investing those resources elsewhere, has been lost. One of the keys in setting priorities, then, is to measure or weigh out the costs and benefits of doing one thing vis-à-vis another… The other principle is that of the margin, which is about shifting or changing the resource mix. If the budget increases, one could reasonably ask how best the additional resources should be spent. Conversely, if the budget decreases, one would likely want to take resources from areas which are producing the least benefit. Lastly, if the budget was neither increasing nor decreasing, at least not continuously, the questions remains as to whether resources should be re-allocated (with some areas cut back so that others can expand) so as to improve benefit to the population being served. The concept of the margin is crucial to this development of an economic approach to priority setting" (Mitton and Donaldson 2004). Figure 11 below sets out the seven different steps of the process.

Figure 11: The Seven Steps of PBMA³¹

Section Three: Deliberative Processes

1. Introduction

A critical gap in the priority-setting literature lies at the outset of the process – with the stakeholder. What is a stakeholder exactly? How are stakeholders identified and analyzed? How can one be deemed relevant to a priority-setting process and another irrelevant? And what are some tools for analyzing the relationships and dynamics that exist among stakeholders?

Added to these questions are issues around elicitation. Clearly, not all mechanisms for eliciting stakeholder input – from, for instance, a structured stakeholder interview to a chance meeting in the corridors – are equal. In the interests of fairness and legitimacy, every priority-setting process must not only identify the right stakeholders, it must also grant those stakeholders the time and space to speak, or the loudest – the squeakiest wheels – may well dominate any discussion. And building on this, deliberative processes then shape, weigh and synthesize the opinions and concerns of each stakeholder to ensure an objective balance – between the "rigorous and the spontaneous," the technical and the general, and among stakeholders themselves.

To that end, this section is divided into three. Here we explore a range of practical tools for:

identifying and analyzing stakeholders;
eliciting stakeholder input; and
convening deliberative dialogues.

Identification to Elicitation to Deliberation

2. Stakeholder Identification and Analysis

Identifying the "right" stakeholders for any priority-setting process is much more than simply creating a list of individuals. While the identification process may start with this type of brainstorming, the greater the process can identify and analyze not only the relevant institutions, networks, and individuals but also the underlying assumptions, the forces for and against change, and the critical dynamics among stakeholders, the better any priority-setting process will prepare itself for understanding and grappling with complexity. Some processes may be relatively straightforward, but most involve a tangled multiplicity of stakeholders – e.g. in setting priorities for research in health financing, which actors need to be involved? and how do they need to be involved, and when?

In any priority-setting process, the leaders need to understand:

the different stakeholders in any given research or policy environment;
the power, involvement and interests of those stakeholders;
the degree of any stakeholder's support for a particular position, ideology or political perspective;
the dynamics that exist between various stakeholders; and
underlying policy networks and influencers.

2.1 Analyzing the Stakeholder

What is a "stakeholder"? And how do we distinguish the key variables of power, involvement and interest among different stakeholders?

Stakeholders are defined here as any individual, group or institution with a vested interest in the prioritization process. Their "vested interest" includes something they stand to win or lose, or the rights or ownership they may possess (WWF, 2005; Bourne and Walker 2006). As Bryson (2004) concludes, a stakeholder gains relevance "if they have information that cannot be gained otherwise, or if their participation is necessary to assure successful implementation".

A stakeholder analysis is, then, a multi-step tool to identify and analyze relevant stakeholders, the dynamics among them and between them and their environment. It typically maps the nature of stakeholders' interests; the extent to which stakeholders' interests converge or overlap; the stakeholders' importance or influence with regard to the issue at hand; the dynamics at play among various stakeholders; and how a shifting context can modify any of the above variables.

As the WWF (2005) concisely states, "the goal of a stakeholder analysis is to develop a strategic view of the human and institutional landscape, and the relationships between the different stakeholders and the issues they care about most". It is neither possible nor desirable to involve or satisfy all possible stakeholders (Bryson 2004). Instead, a stakeholder analysis typically identifies and focuses on the key stakeholders – typically called the primary and secondary stakeholders. Primary stakeholders have a direct interest in the issue at hand; secondary stakeholders may have a lesser interest or may be influenced indirectly – for instance by the changed actions of the primary stakeholder. This reflects the cascade effect that may exist among stakeholders: their dynamics are not static, and their relationships are subject to constant shift and change.

Critically, the very act of determining who is primary – and who is not – turns a stakeholder analysis into a political act with certain ethical consequences (Bryson 2004) and reinforces the fundamentally political nature of the overarching priority-setting process. To mitigate this, stakeholder analyses are often done by teams of individuals, which can reduce or neutralize biases and assumptions (Varvasovszky and Brugha 2000).

In summary, a stakeholder analysis can help the leaders of a priority-setting process identify:

the importance, influence and interests of both primary and secondary stakeholders;
the layers and levels within complex institutions – e.g. entities like a Ministry of Health, a community, or the World Health Organization can all be treated as a single or as a multiple actor depending upon a closer analysis of their positions and needs;
the "level" of stakeholder to involve – local, regional, national, and/or international.
potential conflicts, conflicts of interest, or risks that could jeopardize the process;
opportunities and relationships that can be built on during the process;
groups that should be encouraged to participate in different stages of the process; and
appropriate strategies and approaches for stakeholder engagement (WWF, 2005).

2.2 Stakeholder Analysis: Components, Approaches and Tools

There are various different ways of conducting a stakeholder analysis, as evidenced in the literature. Below is an amalgam of these approaches, distilled here into nine separate components. Some are sequential steps; some are stand-alone tools. Some depend upon facilitated group work, and some rely upon scientifically rigourous data collection.

2.2.1: Planning the process
2.2.2: Identifying (and possibly training) a working or expert group
2.2.3: Developing a plan and timeline
2.2.4: Identifying the issue or problem (types or priorities)
2.2.5: Research tools for identifying stakeholders
2.2.6: Facilitating a brainstorming session
2.2.7: Power vs Interest Grid
2.2.8: Stakeholder Influence Mapping
2.2.9: Stakeholder-issue interrelationship diagrams

2.2.1 Planning the process

This first step helps to define the overarching purpose of the stakeholder analysis, while identifying the potential users of the information and creating a plan for using that information. A leader should be identified and then assume ownership of the entire process. This leader should ensure that a consensus exists on the purpose of the process, and on who will use the information and how (Schmeer 2005).

2.2.2 Identify (and train) a working/expert group

The leader should assemble an inclusive working or expert group (e.g. a Senior Advisory Team, a Task Force, a Technical Working Group) to oversee the analysis. As Schmeer (2005) notes, this group should in effect be a team representing different organizations and interests to avoid biases and mitigate power and personality dynamics, with ideally at least one individual considered neutral and/or independent. A training session could go over the entire priority-setting process and discuss the specific role of stakeholder analysis within that process, as well as specific tools and techniques the group's members will oversee and themselves use.

Note that for a priority-setting process, there may already be a Technical Working Group in place; it would obviously take ownership and leadership of the analysis.

2.2.3 Develop a plan and timeline

The working/expert group should create a timeline by listing the specific steps in the process. There should be ample time allocated to meetings, interviews, presentations, and so on. (Schmeer 2005).

2.2.4 Identify and discuss the issue at hand

Every issue – from the national to the thematic – has its own layers and components. A stakeholder analysis must capture and understand these, as they may further identify a set of relevant stakeholders that might be hidden at first glance. (Varvasovszky and Brugha 2000).

2.2.5 Research tools for identifying and analyzing stakeholders

As Varvasovszky and Brugha (2000) outline, there are several different methods for designing a comprehensive list of stakeholders. Choosing the right method depends entirely upon the issue at hand – some will require relatively simple brainstorming techniques, while those of great complexity will require a much more thorough and scientific approach to ensure the right stakeholders are identified and not omitted. Brainstorming may begin with some familiar research tools to get a better handle on the issue itself, which in turn reveals various stakeholders and essential variables, as Varvasovszky and Brugha (2000) detail: "Face-to-face interviews using checklists, semi-structured interviews and structured – often self-administered – questionnaires can all be used to collect data from primary sources. Usually these are individual respondents, though groups of stakeholders may also be interviewed, e.g. through focus group or informal group discussions. Secondary sources include published and unpublished documents, reports, policy statements, internal regulations of organizations, etc. Interviews provide opportunities to access additional secondary sources, e.g. internal documents not obtained in the initial literature search. Semi-structured interviews can help structure data collection while keeping the focus sufficiently broad to allow for hidden or emerging themes. When analyzing complex issues, especially for policy analysis, qualitative approaches are essential so as to preclude premature focusing on a limited number of aspects of the issue, to the neglect of others which may emerge during the process of data collection and analysis".

Following the data collected with these research tools, the working/expert group would make some strong evidence-informed decisions about the stakeholders relevant to their issue, setting the stage for some deeper analysis of their dynamics and relationships.

2.2.6 Facilitated Brainstorming Session

Bryson (2004) describes this type of group-based facilitated approach for identifying and analyzing stakeholders. In this brainstorming session, the working/expert group assembles, potentially adding to its numbers with some invited members, for a facilitated session or sessions to fully brainstorm out all possible stakeholders. At such a meeting, the facilitator would:

Use a flipchart, white board or computer software (eg "mind mapping" or word processing) to identify each stakeholder.³²
For each stakeholder, create a separate sheet/document and place their name at the top of the page
Create a narrow column down the right side of each page/sheet and leave the column blank
For each stakeholder, in the area to the left of the narrow column, list the criteria the stakeholder would use to judge the project, policy or policy reform.
Use coloured dots to indicate a stakeholder judgement of good (green), fair (yellow) or poor (red).
Identify and record what can be done quickly to satisfy each stakeholder.
Identify and record longer-term issues with individual stakeholders and with stakeholders as a group.

Additional steps that might be included here:

Specify how each stakeholder influences the issue at hand
Decide what the issue needs from each stakeholder.
Rank the stakeholder according to their importance to the issue, with considerations of the stakeholder's power, legitimacy and influence. This can help to create the categories of primary and secondary stakeholders (with the possibility of adding a third category of "special" or "key" stakeholders potentially not captured in the primary or secondary categories, or to highlight particularly important members of the primary category).

2.2.7 Complete a Power vs. Interest Grid

These grids (as described in Bryson 2004) array stakeholders on a matrix where the axes are the stakeholder's interest in the issue, and the stakeholder's power to affect it. This results in four different "types" of stakeholder: players who have both an interest and significant power; subjects who have an interest but little power; context setters who have power but little direct interest; and the crowd which consists of stakeholders with little interest or power. Figure 1 shows a completed grid.

Figure 1: Power vs. Interest Grid³³

Power vs. Interest grids can indicate which stakeholders must by all accounts be involved in addressing the issue at hand. "They also help highlight coalitions to be encouraged or discouraged, what behaviour should be fostered and whose 'buy-in' should be sought or who should be 'co-opted'. Finally, they provide some information on how to convince stakeholders to change their views" (Bryson 2004).

2.2.8 Stakeholder Influence Mapping

Upon completing a Power vs. Interest grid, the next step visualizes relationships and influence between and among the stakeholders. With a facilitator, the working/expert group can:

chart lines of influence from one stakeholder to another
highlight two-way influences (with every attempt to identify the primary direction of any influence)
discuss which influence relationships exist, their relative importance and primary directions (Bryson 2004).

The International Institute for Environment and Development (IIED) (2005) adds a different dimension to stakeholder influence mapping, preferring instead a pyramid that arranges different stakeholders. The closer a stakeholder is to the issue at the top of the pyramid, the more influence they have. Moreover, this type of diagramming allows the working/expert group to visualize the relative size of stakeholders (in terms of numbers or of power) as well as the connections among stakeholders. Figure 2 below demonstrates this.

Figure 2: Stakeholder Influence Mapping³⁴

IIED (2005) suggests following nine steps to preparing such a triangle, which are very similar to those for the Power vs Interest Grid. Figure 3 shows a hypothetical example of a triangle relevant to setting national health research priorities.

Figure 3: Stakeholder Influence Mapping for a National Health Research Agenda

2.2.9 Stakeholder-issue interrelationship diagrams

Expanding on stakeholder influence mapping are diagrams that link particular stakeholders to particular issues; this can reveal connections to other stakeholders who also have a link with the same issue. As Bryson (2004) notes, "the resulting diagrams help provide some important structuring to the problem area, in which a number of actual or potential areas for cooperation – or conflict – may become apparent. Group work should start with a Power vs. Interest grid and/or a stakeholder influence diagram, brainstorm the issues at hand, and then how stakeholders array around the issues."³⁵

Figure 4: Stakeholder-issue Interrelationship Diagram³⁶

3. Eliciting Stakeholder Input

With a better awareness of who the primary and secondary actors in any given priority-setting process may be, we can turn now to mechanisms for eliciting their input. Recognizing that each stakeholder will have different levels of knowledge, different interest in the priority-setting process, and different means of expression, exactly how their insights and concerns are elicited is of high importance. Moreover, as stakeholder elicitation efforts are both costly and time-consuming, the question of whom to involve and how – particularly as research issues and questions become more technical in nature – is particularly relevant.

This section highlights three tools particularly useful for eliciting stakeholder input in the lead up to a deliberative dialogue. These tools can provide the raw data or information for the dialogue itself (especially in contributing to pre-circulated materials); shape the contents and structure of the dialogue; provide a platform for those who cannot participate in the dialogue (due for instance to financial or geographic constraints); provide anonymity on sensitive issues; and indicate how various stakeholders might best participate in the dialogue.

3.1 Concept Mapping

Sometimes referred to as "structured conceptualization," concept mapping is a participatory tool that combines organized brainstorming with statistical analysis to create a visual aide that can be used to kick-start, plan, inform or even evaluate deliberation (Novak and Cañas 2008, National Cancer Institute 2007). The tool begins with a focus question or focus prompt that isolates the problem or concern a concept map can address. As Novak and Cañas (2008) observe, "every concept map responds to a focus question, and a good focus question can lead to a much richer concept map". Organizers can send this focus question to a range of stakeholders for their own private deliberation; or can convene a facilitated dialogue to discuss the focus question. Either way, this can be a strong tool to elicit stakeholder input and then synthesize that input visually.

Following the initial elicitation – i.e. stakeholder responses to the focus question – organizers group those responses into 15-25 concepts. "These concepts could be listed, and then from this list a rank ordered list should be established from the most general, most inclusive concept, for this particular problem or situation at the top of the list, to the most specific, least general concept at the bottom of the list. Although this rank order may be only approximate, it helps to begin the process of map construction. We refer to the list of concepts as a parking lot, since we will move these concepts into the concept map as we determine where they fit in. Some concepts may remain in the parking lot as the map is completed if the map-maker sees no good connection for these with other concepts in the map" (Novak and Cañas 2008).

A preliminary map should reveal linkages between the concepts, and suggest ways that ideas might be combined or further contrasted. This map can be the visual aid brought to the deliberation, or it may, for instance, be posted online for further revision and concept re-positioning, with a "final" map ultimately informing the priority-setting work.

Figure 5: Concept Mapping Flowchart

Below is a case study from the National Cancer Institute (2007) outlining how concept mapping provided critical stakeholder input to the focus prompt:

Case Study: Concept Mapping in Tobacco Control

A monograph by the National Cancer Institute (2007) used concept mapping to generate stakeholder input to the focus prompt: one thing that should be done to accelerate the adoption of cancer-control research discoveries by health service delivery programs is… "Approximately 55 people contributed (by email) more than 200 answers to the focus prompt, which were subsequently synthesized by the steering committee into 98 unique ideas. The data were aggregated and analyzed with a sequence of multivariate analyses that included multidimensional scaling and hierarchical cluster analysis. The resulting map grouped the 98 ideas into 12 conceptual categories. The participants also were asked to identify clusters of clusters that seemed to belong together and provide a label for each such region of the map. Participants identified four major regions: 1) policy, consisting of policy issues that would enable more integration of research and practice, as opposed to policy that results from such efforts; 2) research; 3) practice; and 4) partnerships and support" (National Cancer Institute 2007).

Figure 6: Concept Mapping³⁸

Used in a priority-setting context, the above concept map in Figure 6 could provide a good starting point for a deliberative dialogue. Particularly useful features of a concept map for priority setting include:

anonymous, scientific elicitation of stakeholder input that equalizes each voice and eliminates power differentials among the stakeholders;
can be done at a distance, which can both lower meeting costs and secure the input of a wider range of individuals (e.g. the time pressed or those in distant, rural communities);
thematic grouping of stakeholder input, including connections among the themes and relative size of themes, depicts stakeholder interest and thus sets some thematic parameters for the dialogue;
short, visual summary of issues to inform and influence deliberative dialogue (an ideal piece of pre-circulated material for the dialogue).

3.2 Delphi Technique

This is a well-used methodology for the systematic elicitation and synthesis of opinions and judgements on a topic, issue or theme through a flow of sequential questionnaires, information summaries, and feedback (Delbecq et al 1975). While typically used to integrate the judgements and feedback of a small expert group, the Delphi Technique can be used in advance of a deliberative dialogue to better understand issues and to begin the preliminary synthesis of ideas, concerns and priorities (McDonald, Bammer and Deane 2009). As with Concept Mapping, it is often done anonymously – potentially a critical factor in its success.

"The Delphi Method [or Delphi Technique] is a systematic, interactive forecasting method which relies on a panel of experts. The experts answer questionnaires in two or more rounds. After each round, a facilitator provides an anonymous summary of the experts' forecasts from the previous round as well as the reasons they provided for their judgments. Thus, experts are encouraged to revise their earlier answers in light of the replies of other members of their panel. It is believed that during this process the range of the answers will decrease and the group will converge towards the "correct" answer. Finally, the process is stopped after a pre-defined stop criterion (e.g. number of rounds, achievement of consensus, stability of results) and the mean or median scores of the final rounds determine the results."

– Wikipedia³⁹

The Delphi Technique typically involves three groups:

organizers who design the overarching questions and approach, summarizing responses and preparing subsequent questions;
experts or respondents who offer judgements and answers to the organizers' questions; and
decision-makers who may act upon receiving a consensus statement or recommendations produced by the technique.

Figure 4: The Delphi Technique⁴⁰

As an early phase of a priority-setting process, the Delphi Technique can be particularly useful at:

eliciting input from a range of different stakeholders anonymously and repeatedly – this may help to amplify and strengthen the voices of those who might not ordinarily participate due to power differentials and dynamics;
incorporating "virtual participants": as with Concept Mapping, can be done at a distance, both lowering meeting costs and securing the input of a wider range of individuals (e.g. the time pressed or those in distant, rural communities)
refining that input through rounds of feedback, evaluation and revision – particularly helpful for highly complex or technical research issues/options/questions
providing a strong starting point for a deliberative dialogue, with some of the issues or complexities already resolved.

3.3 Nominal Group Technique

This is a tool typically used to pool knowledge and arrive at a ranked list of ideas, concerns and/or priorities that is accepted and considered legitimate by the group (McDonald, Bammer and Deane 2009). It is highly structured and thus best in a facilitated situation; unlike the Delphi Technique or Concept Mapping, it is not an anonymous, at-a-distance tool, which can be both a strength and a weakness. It can be applied "in two phases: knowledge exploration (a search for major conceptual frameworks and broad insights) and solution exploration (the refinement of broad insights by specifying components which should be included in the solution program)" (McDonald, Bammer and Deane 2009).

"The nominal group technique (NGT) is a decision making method for use among groups of many sizes, who want to make their decision quickly, as by a vote, but want everyone's opinions taken into account (as opposed to traditional voting, where only the largest group is considered). The method of tallying is the difference. First, every member of the group gives their view of the solution, with a short explanation. Then, duplicate solutions are eliminated from the list of all solutions, and the members proceed to rank the solutions, 1st, 2nd, 3rd, 4th, and so on. The numbers each solution receives are totaled, and the solution with the lowest (i.e. most favored) total ranking is selected as the final decision. There are variations on how this technique is used. For example, it can identify strengths versus areas in need of development, rather than be used as a decision-making voting alternative. Also, options do not always have to be ranked, but may be evaluated more subjectively."

– Wikipedia⁴¹

Like the Delphi technique, the Nominal Group Technique (NGT) gives each participant an equal voice, thus minimizing or eliminating the potential power differentials among stakeholders. Additionally, the face-to-face nature of NGT provides a range of opportunities to better understand the opinions and judgements of others.

Figure 5: The Nominal Group Technique⁴²

As a tool within the priority-setting process, the Nominal Group Technique is particularly adept at:

arriving at consensus on highly complex issues
amplifying participation and creativity, while also expanding perceptions of key points in defined areas
determining and dissecting initial priorities (McDonald, Bammer and Deane 2009).

4. Convening Deliberative Dialogues

There are many ways to convene a deliberative dialogue. For some priority-setting situations, the best way may also be the simplest: a neutral chair presiding over a meeting of the "right" stakeholders (identified through a stakeholder analysis), each having read the pre-circulated material (with some of these materials informed by any of the above elicitation techniques) and arriving at a list of weighted, ranked priorities for broader dissemination, review and revision. Depending on context and the stakeholders involved, however, certain dialogue mechanisms may prove superior to balancing and synthesizing different voices, and ultimately creating a comprehensive list of priorities to further influence decision-making.

Any multi-stakeholder deliberative dialogue should embrace the following principles:⁴³

the process should encourage a multiplicity of voices, with listening as critical as speaking, and the conversation more "horizontal" than "vertical" (i.e. among equals);
a range of views should be heard and discussed before any decision or resolution is made;
the discussion should emphasize analysis and reasoned argument. "The powerful work that occurs in dialogue – identifying the connections between personal and public concerns, creating mutual understanding, and building relationships based on trust – is necessary for solving complex public problems" (McCoy and Scully 2002).

Multi-stakeholder deliberative dialogue principles.

In reviewing the ideal, interpretive priority-setting process outlined in Section 2 of this Module, we find (at minimum) three different areas where a deliberative dialogue might be convened: at the beginning of the process (step 1), where a small group comes together to plan the process through its various steps; during the interpretive workshops, where the small group drafts an initial list of priorities (step 4); and during the larger publicization and validation workshop, where the initial list of priorities is submitted to a larger reference group for validation, dialogue and revision (step 6).

As the Technical Working Group will use different deliberative dialogue techniques than the Larger Reference Group in a priority-setting process, we've divided the discussion below by group, with several deliberative techniques suited to each.

4.1 The Technical Working Group – Formation, leadership, ownership

At the heart of the ideal interpretive priority-setting processes is a small group of leaders – sometimes called a Senior Advisory Team, Advisory Committee or Group of Elders – charged with overseeing the entire process. But how do these groups form and how might they effectively lead an actual priority-setting process?

4.1.1 Visualization in Participatory Planning (VIPP)

Visualization in Participatory Planning (VIPP) is one such tool extremely useful in creating and bonding disparate groups and bringing out the best of each individual member. VIPP embraces these group dynamics through the use of diagrams, cards and photographs to express main ideas, with the "less talkative" participants finding a means of expression, and the group arriving at an effective and genuine consensus. VIPP depends upon an expert facilitator to guide the group through its structuring.

It derives from educational processes in both Germany and Latin America, but was originally developed as its own approach by UNICEF in Bangladesh (McKee et al 2009). It is a unique methodology for group processes – putting people at the centre "of solving development and social problems and coming to new, collective visions of the future. It is unique in that it provides a creative combination of different participatory approaches which emphasize visualization techniques" (McKee et al 2009). It builds in turn upon Bruce Wayne Tuckman's 1965 work describing how groups come together and weather conflict to become a cohesive whole able to make broad, consensual decisions. As Smith (2005) explains of these group dynamics, "it is clear, for example, that people tend to want to know something about the other members; have to develop a degree of interdependence in order that the group or team may achieve its tasks and be satisfying to its members; and has to learn at some level to deal with conflict if it is to survive." Illustrating this, Tuckman's five steps or stages of group development include:

Forming. Here, groups "concern themselves with orientation accomplished primarily through testing" (Tuckman, 1965). This orientation and testing allows group members to understand "interpersonal and task behaviours" while also establishing relationships with other members, and becoming increasingly comfortable with the group's leader.
Storming. In this stage, groups are often "characterized by conflict and polarization around interpersonal issues" (Tuckman, 1965). There can be some resistance to the group at this stage as personalities battle to determine the course and activities of the group.
Norming. Here, groups tend to overcome the earlier conflict and develop an "in-group feeling and cohesiveness" with new roles, standards, and a warmer atmospherefor expressing opinions.
Performing. At this stage, with roles established yet flexible and the group's work understood and clear, the structure of the group "can now become supportive of task performance" (Tuckman, 1965) and the central purpose of the group can be meaningfully and lastingly fulfilled.
Adjourning. While not part of Tuckman's earlier identification of the stages involved, this last stage necessarily concerns the dissolution of the group, with central tasks complete, and a roadmap for the future drawn and awaiting implementation.

VIPP photographs by Cole Dodge (2009), from a Technical Working Group meeting in Zambia (discussing the formation of the National Health Research Council of Zambia). More on this project online.

VIPP photograph by Cole Dodge (2009), from a Technical Working Group meeting in Zambia

4.1.2 The Expert Witness Panel

Once the TWG has achieved some coherence – with clarity of purpose, of teamwork, and of the scope of the problem or issue at hand – there are a number of participatory techniques that can be employed to add critical elements to the group's brainstorming. The Expert Witness panel is one such participatory tool highly successful in incorporating the voices of individuals who have particular insights but who, because of time constraints or institutional conflict, cannot or should not serve on the Technical Working Group. To convene an Expert Witness panel, the TWG brainstorms on experts of relevance to their discussions; invites a select number of experts to come to a TWG meeting at specific times; the experts then sit individually before the TWG, with two TWG members at a time sitting in front of the expert and asking specific questions (to inform the TWG's wider thinking); once a TWG member has asked his/her question, s/he returns to the wider group and is replaced by another TWG member. Following all expert witness interviews, the TWG summarizes and synthesizes their input and uses these insights to assist its decision-making processes.

For priority-setting processes, the Expert Witness panel can be of great assistance as a prelude to the formal interpretive workshops, getting advice on criteria to weigh priorities, on further priorities to consider, on the overall process itself, and a general tool to expand the TWG's approach and thinking. The panel has further utility in disseminating its work to a select audience (i.e. the take-away messages each expert will return to his/her institution), which will expand the collective ownership of the process as those whose voice is incorporated into the process are typically more likely to accept its outcomes.

The Chatham House Rule

This technique can be added to any deliberative dialogue, assuring those present that nothing will be attributed, thus opening up discussion to areas that some may deem sensitive, or may be hesitant to comment on for fear of misrepresenting or exposing an institution. "The Chatham House Rule is a principle that governs the confidentiality of the source of information received at a meeting. Since its refinement in 2002, the rule states: 'when a meeting, or part thereof, is held under the Chatham House Rule, participants are free to use the information received, but neither the identity nor the affiliation of the speaker(s), nor that of any other participant, may be revealed'. The rule allows people to speak as individuals and to express views that may not be those of their organisations, and therefore, encourages free discussion. Speakers are free to voice their own opinions, without concern for their personal reputation or their official duties and affiliations. The Chatham House Rule resolves a boundary problem faced by many communities of practice, in that it permits acknowledgment of the community or conversation, while protecting the freedom of interaction that is necessary for the community to carry out its conversations. Meetings, or parts of meetings, either may be held on the record, or, under the Chatham House Rule. In the latter case, all participants are understood to have agreed that it would be conducive to free discussion that they should be subject to the rule for the relevant part of the meeting. The success of the rule may depend upon it being considered morally binding, particularly in circumstances where a failure to comply with the rule may not result in sanction. Sometimes the rule is half-jokingly summarized as, 'You may be quoted, but you cannot be fired.'"

– Wikipedia⁴⁴

For more on deliberative techniques at interpretive workshops, see Section Two's discussion of the ideal, interpretive priority-setting process.

4.2 Larger Reference Group – wide multi-stakeholder deliberation

Taking initial priorities to a wider pool of stakeholders to ensure alignment with their values and needs is a critical step in the interpretive process. This "Larger Reference Group" (LRG) brings with it a host of components critical to deliberative dialogues. They are representative in ways that the TWG likely cannot be due to their larger size; can include wider members of the public and thus further democratic ideals; create shared meaning and open new windows of understanding; are critical to disseminating both the list of priorities and the process of priority-setting itself; and can add both scientific and social credibility to wider decision-making processes.

As Kapiriri et al 2007 observe, "the actual size and composition of the [LRG] would depend on the research options to be considered and the available resources including time. Stakeholders can be represented by a reasonably small or very large reference group, depending on the context. For example, the group may comprise of only 10 persons (e.g. main individual donors to a private charity who want to participate in decision-making on how their money is invested) or more than a thousand people (e.g. in the case of addressing research to avert global child mortality, where everyone from members of [the] public in developing countries, public and private donor foundations, to the UN, World Health Organization (WHO) and UNICEF officials represent the potential stakeholders)."

Larger Reference Groups in Priority Setting – two examples

"To ensure that the assessment of the research priorities is combined with a view of the wider society, the relative weights for each criterion were measured from 30 stakeholders' representatives from the larger reference group. LRG members included academics from three Cape Town universities, members of the public, government representatives, clinical psychologists, and other professionals. In choosing LRGs, we used a convenience sample attempting to secure a diverse mix of researchers, clinicians, professionals, academics, and members of the public and allowed them to express their opinions through an interview and as a quantitative score. LRG members were ?rst told the elements of the process, then asked for feedback, and eventually asked to rank those ?ve criteria from the most important within the South African context (rank 1) to the least important (rank 5)."

– Tomlinson et al 2007.

"To ensure involvement of the wider society in directing research investment priorities, we collected opinions from a larger reference group, comprising 43 stakeholders' representatives… The reference group comprised nine psychiatrists, four psychologists, two social workers, three government employees, six non-governmental organization representatives, six researchers, six users of mental health services and seven members of the public. We contacted members of the group by e-mail and asked them to express their opinions through an electronic questionnaire that described the elements of the process. We asked them to rank the five criteria used for setting priorities."

– Tomlinson et al 2009.

The Nominal Group Technique and Delphi Model are two strong deliberative tools that might assist the functions of the LRG, given that the core function of the LRG is to rank – be it criteria or priorities. They can be initiated virtually and concluded at a physical meeting.

4.3 Other Deliberative Dialogue Models

There are a variety of other models for convening a deliberative dialogue. Some may not be ideally suited to priority setting, but may accord well with other discussions within any national health research system. Such models are discussed in detail in McDonald, Bammer and Deane (2009), and include:

Consensus Conferences. "This is a highly structured event designed to involve the non-expert, non-partisan citizens in deliberating on important (and typically complex) social, technological, planning and/or policy issues, and through doing so to integrate judgements. Their objective is to 'bridge the gap between the general public, experts and politicians'… Its chief characteristics include: a representative sample of participants in small numbers (12-25); a skilled facilitator; participants provided with written evidence before they meet; participants decide who to call in as an expert witness; interactive; preparatory weekend; deliberative meeting of 2-4 days; recommendations published in a formal report; either the recommendations are implemented or sufficient grounds must be provided publicly to explain why they will not be implemented" (McDonald, Bammer and Deane 2009).
Consensus Development Panels. "The purpose of these panels is to provide guidance in areas of medical and broader health practice, particularly in areas in which controversy exists and a body of scientific evidence is available that can be scoped, explored, assessed and synthesized to produce a consensus statement on the issue. The consensus development panel process developed and used regularly by the National Institutes of Health in the USA is a highly structured approach for integrating scientific research evidence emanating from different disciplines. At the core of the process is judging the evidence and reaching a consensus position that is then communicated to the health professions and the public in non-technical language" (McDonald, Bammer and Deane 2009).
Town Meetings. "To date the most successful Town Meeting format is a day-and-a-half meeting that includes a scientific symposium and an open forum for community members. The full meeting is open to community involvement, although the most active participants during the scientific symposium are researchers, health care providers, and public health officials who want to hear about the latest research findings in this particular field. The open forum is designed specifically to encourage and facilitate comments and questions from residents. To this end, the forum is set at a time and location that allows for the greatest level of community participation, and panelists give brief comments on research, policy and public health before the floor is opened to the public. The expert panel addresses the comments and questions raised. Sometimes the host organization will have a local or federal politician, a popular local television news anchor, or a communications expert facilitate the meeting. This strategy often results in a more interesting meeting and attracts a larger audience" (Olden 2003).
Future Search Conferences. "These are large-group planning conferences, using face-to-face dialogue to develop plans, including the identification of action steps. Future search conferences are particularly useful in situations of rapid change (for eg changes in knowledge, society, technology, the environment) when various stakeholders can be expected to make different judgements about the implications of change for the future. They begin with a focus on visions and use these to guide the proposals for action. The implementation methods vary. Some proponents of this technique argue for limiting the number of future search conference participants to about 60-80, all meeting in one room and in active dialogue, on the grounds that more than this number means that productive dialogue is not feasible. Others are comfortable with far more participants. In these instances, participants are broken up into smaller groups. The tasks of the conferences are expressed as 'The future of X is Y…' Conditions for success include:
- getting the 'whole system' into the room; invite a significant cross-section of all parties with a stake in the outcomes of the conference
- exploring the 'white elephant' before seeking to fix any part; get everyone talking about the same world; explore the global context before focusing on local issues
- emphasizing common ground and future focuses, while treating problems and conflicts as information, not action items
- encouraging self-management and responsibility for action by participants before, during and after the conference. (McDonald, Bammer, Deane 2009).
Open Space Technology (OST). "This tool can assist groups of people to identify and explore issues, identify opportunities for change and identify and set priorities among action steps to achieve desired goals. The approach is based on the theories of complexity, self-organization and open systems… The key elements of the setting are one or more circles of chairs, with circles being seen as the 'fundamental geometry of human communication'. The room has to be large enough to have a number of small groups sit in circles, and to change the configuration of circles as the process unfolds. The facilitator explains the process: whoever comes is the right people; whatever happens is the only thing that could have; whenever it starts is the right time; when it's over, it's over… The facilitator then asks people to come to the centre of the circle, introduce themselves and tell everyone else of one aspect of the session's theme about which they are passionate. The person records this aspect on a large sheet of paper along with their name and a suggested time and place for discussing the issue, before attaching the sheet to the wall. In this way, a first-draft agenda evolves… When all the issues for discussion have been posted, participants sign up for the groups with which they wish to be involved. At this stage, topics can be combined. Once this step is completed, the facilitator announces that he is departing, inviting the groups to get to work…OST is effective in situations where a diverse group of people must deal with complex and potentially conflicting material in innovative and productive ways. It is particularly powerful when nobody knows the answer and the ongoing participation of a number of people is required to deal with the questions. The process requires participants to shed their power roles in organizational hierarchies and interact as equals… The underlying philosophy is that synthesis will occur through small-group discussions among people self-selected to address a topic about which they are passionate. Integration is taken to be an emergent property of the group process" (McDonald, Bammer and Deane 2009).
Citizens' Jury. Connected to the idea of an Expert Witness panel is the citizens' jury. Here, a randomly selected group (18-24 individuals) of lay citizens convene to interview witnesses and collectively deliberate on an issue. They are "representative – selected by a recognized sampling method; informed – witnesses present to the jury a variety of facts, information and opinions on the matter under consideration, and are questioned by the jury; impartial – those organizing the process select witnesses whose evidence is carefully balanced to ensure fair treatment to all sides of the issue; and deliberative – the jury deliberates in a variety of formats and is given sufficient time to ensure that all of the jurors' opinions are considered" (McDonald, Bammer and Deane 2009). A citizens' jury typically concludes with the jury issuing recommendations for the way forward. Several authors (e.g. Price (2000), Lenaghan et al (1996)) have explored the use of citizens' juries in priority setting for service delivery.

Section Four: Priority Setting and the Deliberating Stakeholder

How can deliberative techniques improve research priority-setting processes, particularly in low- and middle-income countries? What steps can institutions and systems take to increasingly infuse their priority-setting processes with deliberation?

To re-emphasize the central point throughout this Module, the concept of the deliberative dialogue offers a variety of ways to determine and involve the "right" stakeholder in priority setting – not only as those who are consulted, but as those who play an active role in the process. Much as Rudan et al (2007b) identify the "health research option" as an essential addition to priority-setting theory, we advance here the concept of the "deliberating stakeholder" as a missing or glossed-over component of priority-setting processes. Stakeholder values and criteria drive the entire priority-setting process; any process that seeks to gauge and address a society's knowledge needs must then focus on who is involved and how they participate in the process. Failure to do so will leave this critical function to the technical experts and scientists, who often have significantly different values and criteria than other relevant stakeholders (Kapiriri et al 2007). Priority setting must begin with a comprehensive identification and analysis of relevant stakeholders, understanding their values, interests and dynamics; it must then implement platforms to elicit their input (especially virtual or distance platforms); and by appreciating overarching context, values and stakeholder dynamics, then balance and synthesize their input through appropriate deliberative techniques.

This three-pronged approach – identification, elicitation, and deliberation – is critical to the success of any interpretive priority-setting process. As with other KT techniques, such an approach recognizes and values the social nature of priority setting, one that is based on relationships and trust. The better any priority-setting process can be cast in this light – as an iterative, deliberative, and fundamentally social mechanism – the greater chance a society's priorities can align with its values, and its resources with its needs.

Experimenting, Piloting, Evaluating

That said, there is still a great deal unknown about priority setting. We need to experiment and pilot, we need to understand what techniques work, for whom, and under what circumstances – particularly in assessing the links between deliberative techniques and identified priorities, between stakeholder involvement and the types of priorities identified (Ranson and Bennett 2009), and between priority-setting processes and decision-making itself (Smith et al 2009). We already have good evaluative frameworks for priority setting (e.g. the fair, just, deliberative, legitimate, transparent, accountable and enforceable elements of Daniels' Accountability for Reasonableness) and for deliberative processes (e.g. around representation, procedures, information and outcomes as suggested in Abelson et al (2003)). Given the need to understand, assess and document the complex dynamics within priority-setting processes, it is imperative that innovative and strong evaluations capture this experimentation and piloting in priority-setting.

Incorporating deliberative dialogue techniques into priority-setting processes

Not every priority-setting process needs to rotate around a costly series of meetings bringing in stakeholders from across a country. There are tremendous opportunities for refining and advancing virtual tools – such as Concept Mapping and the Delphi Method – to elicit multi-stakeholder input and kick-start deliberations among a wider, more inclusive pool of stakeholders. If priority setting to date has been an expert-driven process, it is the increasing use of these kinds of deliberative tools that will bring out new perspectives and different ideas. In many cases, this stage of stakeholder value and opinion elicitation is too often ignored in priority setting, but there has never been a better time to learn how other contexts have used such tools, and to apply them in priority setting, at a national level, a regional level, a provincial/state level, even an organizational level.

Applying this type of innovation is even more crucial for the interpretive workshops that lie at the heart of most priority-setting processes. The literature brims with interpretive workshops that mirror scientific conferences – events that depend upon an equality of scientific understanding, on an acceptance of scientific rationale and method. If the leaders of a national health research system are serious in bringing more stakeholders to the priority-setting table, it must combine deliberative approaches and mechanisms to level the playing field, blending the rigourous with the spontaneous, the quantitative with the qualitative. This does not mean that every interpretive workshop should be akin to a Town Meeting, but that among the medley of deliberative techniques discussed in this Module various elements might come together to identify, elicit and synthesize the valuable contributions from across the stakeholder spectrum.

Emphasis on process

Ultimately, it is the process and not the product that ultimately matters (WHO 2003). Dialogue may not solve a problem or set actual priorities, but it will build the social relationships, trust and interactions critical to KT and any health system. If a priority-setting process fails to arrive at a ranked list of priorities but does strengthen a system now able to diagnose and discuss its weaknesses and opportunities, then it will have more than achieved its objectives. After all, a deliberative dialogue is not intended to solve problems (Lavis 2009a) but rather to create open discussion on the nature of the problem itself, working to reveal the true complexity of the problem (Abelson et al 2003), or to frame it within its wider, systemic framework. The point is not to solve: it is to deliberate.

And it is to evolve. We've shifted our vocabulary from "the Third World" to "the developing world" to "the South"; we've incorporated gender as a fundamental research lens to perceive a society's often hidden power relationships. Along these lines, we may need to recast research "priority setting" as "informing research priorities" to soften the rigidity and finality of the term. We must recognize that the political actors of any given setting may not yet be ready to cede or share their priority-setting powers (no matter who participates in a Technical Working Group), and that this is a battle likely not worth fighting: it is the process, not the product that matters. After all, any Minister can set a priority, but much more valuable are those situations where an inclusive range of voices can discuss a society's values, needs and concerns.

Re-imagining knowledge generation and application

Lastly – and absolutely critically – we must start to re-imagine how knowledge is generated and applied. For far too long, the research agenda has been controlled by researchers and research funders, and the implementation agenda by local, national and global policy-makers. To achieve any progress in bridging the knowledge-to-action divide, these two camps must converge, and must do so by involving a range of other concerned stakeholders. If there is to be any real progress in KT, in strengthening national health research systems, we must increasingly treat knowledge as a multi-faceted public good: demanded, shaped and used by the many stakeholders it has always claimed to serve.

Section Five: References

Abelson J et al. Examining the role of context in the implementation of a deliberative public participation experiment: Results from a Canadian comparative survey. Social Science &Medicine. 64. 2007.

Abelson J et al. Deliberations about deliberative methods: issues in the design and evaluation of public participation processes. Social Science &Medicine. 57, 2003.

Alliance for Health Policy and Systems Research. Priority setting for health policy and systems research. Briefing Note: Alliance for Health Policy and Systems Research, 2009.

Baltussen R, Niessen L. Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Effectiveness and Resource Allocation. 4:14, 2006.

Bourne L, Walker DHT. Using a visualising tool to study stakeholder influence – two Australian examples. The Project Management Journal, 37 (1), 2006.

Bruni RA et al. Public involvement in the priority setting activities of a wait time management initiative: a qualitative case study. BMC Health Services Research. 7:186, 2007.

Bruni RA, Laupacis A, Martin DK. Public engagement in setting priorities in health care. Canadian Medical Association Journal. 179:1, 2008.

Bryson JM. What to do when stakeholders matter: stakeholder identification and analysis techniques. Public Management Review, 6 (1), 2004.

Byskov J et al. Accountable priority setting for trust in health systems – the need for research into a new approach for strengthening sustainable health action in developing countries. Health Research Policy and Systems. 7:23, 2009.

Canadian Health Services Research Foundation (CHSRF). Weighing up the Evidence: making evidence-informed guidance accurate, achievable, and acceptable. A summary of the workshop held on September 29, 2005. 2006. Accessed November 3 2008.

Charles C, DeMaio S. Lay participation in health care decision making: a conceptual framework. Journal of Health Politics, Policy and Law. 18:4, 1993.

Choi, B et al. Can scientists and policy makers work together?. Journal of Epidemiology and Community Health. Vol 59: pp 632-637, 2005.

COHRED. "Priority Setting for Health Research: Toward a management process for low and middle income countries". Accessed June 1 2010

Commission on Health Research for Development. Health Research: essential link to equity in development. New York, Oxford University Press, 1990

Daniels N, Sabin JE. Limits to health care: fair procedures, democratic deliberation, and the legitimacy problem for insurers. Philosophy and Public Affairs 26:303-350, 1997

Daniels N, Sabin JE. Accountability for reasonableness: an update. BMJ 337, 2008.

Daniels N, Sabin JE. Setting Limits Fairly: Can We Learn to Share Medical Resources? Oxford University Press, 2002.

Daniels N. Accountability for reasonableness: establishing a fair process for priority setting is easier than agreeing on principles. BMJ, Vol 321 25-11-00, 2000.

Daniels N. The ethics of accountability in managed care reform. Health Affairs, Vol 17, Issue 5, 50-64

de Savigny D, Adam T, Eds. Systems thinking for health systems strengthening. Alliance for Health Policy and Systems Research, WHO, 2009.

Delbecq AL, Van de Ven AH, Gustafson DH. Group techniques for program planning. Glenview, IL: Scott, Foresman, and Co. Association.

Dobbins M et al. Information transfer: what do decision makers want and need from researchers? Implementation Science. 2:20, 2007

Dryzek J. Deliberative Democracy and Beyond. Oxford: Oxford University Press, 2000.

El-Jardali F et al. Eliciting policymakers' and stakeholders' opinions to help shape health system research priorities in the Middle East and North Africa region. Health Policy and Planning. 25, 2010.

Fishkin J. Town Halls by Invitation. The New York Times, August 16, 2009.

Franco LA. Forms of conversation and problem structuring methods: a conceptual development. Journal of the Operational Research Society. 57: 2006.

Ghaffar A, de Francisco A, Matlin S, Eds. The Combined Approach Matrix: a priority-setting tool for health research. The Global Forum for Health Research, 2004.

Gibson JL, Martin DK, Singer PA. Setting priorities in health care organizations: criteria, processes, and parameters of success. BMC Health Services Research. 4:25, 2004.

Ginsburg L et al. Revisiting interaction in knowledge translation. Implementation Science. 2: 34, 2007.

Golden-Biddle K. Toward a communicative perspective of collaborating in research: the case of the researcher-decision-maker partnership. Journal of Health Services Research & Policy. Vol 8: suppl 2, 2003 .

González-Pier E et al. Priority setting for health interventions in Mexico's system of social protection in health. The Lancet. 368, 2006.

Graham I et al. Lost in Knowledge Translation: Time for a Map? The Journal of Continuing Education in the Health Professions, Volume 26, pp. 13–24. 2006

Greenhalgh T, Russell J. "Reframing Evidence Synthesis as Rhetorical Action in the Policy Making Drama". Healthcare Policy. Vol 1:1. 2005

Gregory J, Hartz-Karp J, Watson R. Using deliberative techniques to engage the community in policy development. Australia and New Zealand Health Policy. 5:16, 2008.

Hamid M et al. EVIPNet: Translating the spirit of Mexico. The Lancet, 2005.

Hasman A, Holm S. Accountability for reasonableness: opening the black box of process. Health Care Analysis. 13:4, 2005.

Hovland I. Successful Communication: A toolkit for researchers and civil society organizations. Overseas Development Institute, 2005.

Innvaer S et al. Health policy-makers' perceptions of their use of evidence: a systematic review. Journal of Health Services Research & Policy. 7:4, 2002.

International Institute for Environment and Development (IIED). Stakeholder influence mapping. March 2005. Accessed April 7 2010.

Jackson-Bowers E et al. "Focus on Knowledge Brokering." Primary Health Care Research & Information Service. December 2006.

Kapiriri L et al. Setting Priorities in Global Child Health Research Investments: Addressing Values of Stakeholders. Croatian Medical Journal. 48(5): 2007.

Kapiriri L, Martin DK. A Strategy to Improve priority setting in developing countries. Health Care Analysis 15, 2007.

Kapiriri L, Norheim OF, Martin DK. Priority setting at the micro-, meso- and macro-levels in Canada, Norway and Uganda. Health Policy 82, 2007.

Lavis JN et al. SUPPORT Tools for evidence-informed health Policymaking (STP) 3: Setting priorities for supporting evidence-informed policymaking. Health Research Policy and Systems. 2009a.

Lavis JN et al. SUPPORT Tools for evidence-informed health Policymaking (STP) 14: Organizing and using policy dialogues to support evidence-informed policymaking. Health Research Policy and Systems. 2009b.

Law S, Flood C, Gagnon D (on behalf of Listening for Direction partners). Listening for Direction III: Final Report. Accessed January 27 2010.

Lenahan J, New B, Mitchell E. Setting priorities: is there a role for citizens' juries? BMJ. 312, 1996.

Lenaway et al. Public health systems research: setting a national agenda. American Journal of Public Health. 96:3, 2006.

Lomas J et al. On being a good listener: setting priorities for applied health services research. The Milbank Quarterly. 81:3, 2003.

Lomas J. Improving research dissemination and uptake in the health sector: Beyond the sound of one hand clapping. McMaster University Centre for Health Economics and Policy Analysis - Policy Commentary C97-1, 1997.

London S. "The power of deliberative dialogue," in Kingston R, Ed., Public Thought and Foreign Policy. Dayton, Ohio: Kettering Foundation Press.

Makundi E et al. Priority setting on malaria interventions in Tanzania: strategies and challenges to mitigate against the intolerable burden. American Journal of Tropical Medicine and Hygiene. 77(Suppl. 6), 2007

Makundi E, Kapiriri L, Norheim OF. Combining evidence and values in priority setting: testing the balance sheet method in a low-income country. BMC Health Services Research. 7:152, 2007.

Martin D, Abelson J, Singer P. Participation in health care priority-setting through the eyes of the participants. Journal of Health Services Research & Policy. 7:4, 2002.

McCoy M, Scully P. "Deliberative Dialogue to Expand Civic Engagement: What kind of talk does democracy need?". National Civic Review. 91:2, 2002.

McDonald D, Bammer G, Deane P. Research Integration Using Dialogue Methods. The Australian National University E Press, 2009.

McKee N et al. Visualisation in Participatory Programmes (VIPP): Taking stock of its diffusion and impact. Journal of Communication for Development and Social Change: A Global Journal, Hampton Press. Creskill, N.J. Vol. 2, No. 4 2009 (in press).

Menon D, Stafinski T, Martin DK. Priority-setting for healthcare: who, how, and is it fair? Health Policy. 84, 2007.

Mitton C et al. The evolution of PBMA: towards a macro-level priority setting framework for health regions. Health Care Management Science. 6, 2003.

Mitton C, Donaldson C. Health care priority setting: principles, practice and challenges. Cost Effectiveness and Resource Allocation. 2:3, 2004.

Mshana S et al. What do district health planners in Tanzania think about improving priority setting using 'Accountability for Reasonableness'? BMC Health Services Research. 7:180, 2007.

National Cancer Institute Greater than the sum: Systems thinking in tobacco control. Tobacco Control Monograph No 18. Bethesda MD: US Department of Health and Human Services, National Institutes of Health, National Cancer institute. NIH Pub. No. 06-6085, April 2007.

Novak JD and Cañas AJ. The Theory Underlying Concept Maps and How to Construct Them. Technical Report IHMC CmapTools 2006-01 Rev 01-2008, Florida Institute for Human and Machine Cognition, 2008. Accessed May 15 2008.

Nuyens Y. Setting priorities for health research: lessons from low- and middle-income countries. Bulletin of the World Health Organization. 85:4, 2007.

Olden K. Strategies for setting a national research agenda that is responsive to community needs. Environmental Health Perspectives. 111: 2003

Peacock S et al. Using economics to set pragmatic and ethical priorities. BMJ. Vol 332 25 February 2006.

Peacock S et al. Overcoming barriers to priority setting using interdisciplinary methods. Health Policy. 92, 2009.

Price D. Choices without reasons: citizens' juries and policy evaluation. Journal of Medical Ethics. 26, 2000.

Ranson MK, Bennett SC. Priority setting and health policy and systems research. Health Research Policy and Systems. 7:27, 2009.

Rosenstock L, Olenec C, Wagner GR. The National Occupational Research Agenda: a model of broad stakeholder input into priority setting. American Journal of Public Health. 88:3, 1998.

Rowe G, Frewer LJ. A typology of public engagement mechanisms. Science, Technology, & Human Values. 30:2. 2005.

Rudan I et al. Setting priorities in global child health research investments: universal challenges and conceptual framework. Croatian Medical Journal. 49, 2008.

Rudan I et al. Setting priorities in global child health research investments: assessment of principles and practice. Croatian Medical Journal. 48, 2007a.

Rudan I et al. Childhood pneumonia and diarrhoea: setting our priorities right. The Lancet Infectious Diseases. 7: 2007b.

Sabik LM, Lie RK. Priority setting in health care: Lessons from the experiences of eight countries. International Journal for Equity in Health. 7:4, 2008.

Schmeer K. Stakeholder Analysis Guidelines. Accessed April 7 2010.

Sibbald SL et al. Priority setting: what constitutes success? A conceptual framework for successful priority setting. BMC Health Services Research. 9:43, 2009.

Smith E et al. Developing a service-user framework to shape priorities for nursing and midwifery research. Journal of Research in Nursing. 10:1, 2005.

Smith MK. Bruce W. Tuckman – forming, storming, norming and performing in groups. The Encyclopaedia of Informal Education. 2005. Accessed March 25, 2008.

Smith N et al. Identifying research priorities for health care priority setting: a collaborative effort between managers and researchers. BMC Health Services Research. 9:165, 2009.

Start D, Hovland I. Tools for Policy Impact: A Handbook for Researchers, RAPID Toolkit, London: ODI, 2004.

Teng F, Mitton C, MacKenzie J. Priority setting in the provincial health services authority: survey of key decision makers. BMC Health Services Research. 7:84, 2007.

The Working Group on Priority Setting. Priority setting for health research: lessons from developing countries. Health Policy and Planning. 15:2, 2000.

Tomlinson M et al. Setting priorities for global mental health research. Bulletin of the World Health Organization. 87: 2009.

Tomlinson M et al. Setting priorities in child health research investments for South Africa. PLoS Medicine. 4:8, 2007.

Tuckman B. Developmental sequence in small groups. Psychological Bulletin, 63: 1965. The article was reprinted in Group Facilitation: A Research and Applications Journal – Number 3, Spring 2001.

Varvasovszky Z and Brugha R. How to do (or not to do)… a stakeholder analysis. Health Policy and Planning, 15 (3), 2000.

Wait S, Nolte E. Public involvement policies in health: exploring their conceptual basis. Health Economics, Policy and Law. 1, 2006

Walker D et al. Alliance HPSR Technical Paper: Establishing non-state sector research priorities in developing countries using a participatory methodology. 2009. Accessed January 27 2010.

World Health Organization (WHO) World Report on Knowledge for Better Health. WHO, Geneva, 2004.

World Health Organization (WHO) WHA: Priority setting in the health sector. Ministerial round table at the 1999 WHA. Bulletin of Medicus Mundi Switzerland. 91: 2003.

World Wildlife Foundation (WWF) Cross-Cutting Tool: Stakeholder analysis. October 2005. Accessed April 7 2010.

Notes

See WHO's (2004) World Report on Knowledge for Better Health for a description of the phenomenon; also see the special issue of the Bulletin of the World Health Organization (2004), "Bridging the Know-Do Gap in Global Health" for various articles on the subject.
Also see Greenhalgh and Russell (2005), where policy-making is described as "the messy unfolding of collective action, achieved mostly through dialogue, argument, influence and conflict". Lomas (1997) equates policy-making with sausage-making: one doesn't want to see how either is actually made.
In Mexico, governments were urged to support "evidence-based public health and health care delivery systems, and evidence-based health related policies". For details see Hamid et al 2005.
see Graham et al (2006) for a thorough analysis of the leading definitions and terms in the KT lexicon. CIHR's more comprehensive definition of KT is: "the exchange, synthesis and ethically-sound application of knowledge – within a complex system of interactions among researchers and users – to accelerate the capture of the benefits of research…"
This is not to discount or ignore other research-users (the media, private sector, the non-state sector, healthcare practitioners, etc.) – who are essential elements and actors within the reseach-and-policy dynamic - but rather to highlight the central dynamic of research and policy.
For more on this dynamic, see Makundi et al (2007). Also see Makundi, Kapiriri and Norheim (2007) for a discussion of other political dynamics, including the role and influence of dominant interest groups, funders, and different levels of government, all competing against the backdrop of "dysfunctional financing systems".
A particular debt of gratitude is owed Dr. Pierre Ongolo-Zolo here, who in personal communication voiced well-placed concern over the highly political nature of the term "priority setting".
Recognizing the sharp differences between technical (centred on quantifiable – typically epidemiologic – data) and interpretive (creating consensus among stakeholders) priority-setting approaches, our focus here is exclusively on interpretive priority setting. For useful examinations of technical priority setting, see for instance the Combined Approach Matrix (Ghaffar et al 2004), discussed in Section Two. As Ranson and Bennett (2009) observe, interpretive priority setting can and should involve technical elements, in many ways a blend of quantitative and qualitative approaches, though clearly favouring the latter.
For other definitions of priority setting, see the Alliance for Health Policy and Systems Research (2009), who define it as "a programme to generate consensus about a core set of research issues that urgently require attention in order to facilitate policy development". Rudan et al (2008) note that "some authors define priority setting as 'who gets what at whose expense'" but define it as "a 'science' intending to serve the needs of a community or a society at a specific point in time, within given policy, context, time limit, and financial constraints. It is value-driven and there are many interested stakeholders who will necessarily promote a diverse set of opinions and values".
See Byskov et al (2009) for a discussion of AFR with specific application to the REACT project in Africa (REsponse to ACcountable priority setting for Trust in health systems), with details on its application in Tanzania in Mshana et al (2007); Daniels (2000) for a broad yet concise overview of AFR; and Peacock et al (2009) for a general description of AFR. Also see Hasman and Holm (2005) for a sound critique of AFR, highlighting how AFR fails to describe the actual process of priority-setting, with process being a "black box". See Kapiriri, Norheim and Martin (2007) for how AFR can guide evaluations of priority setting. Indeed, this latter element may be AFR's primary contribution to priority setting – adaptable criteria for evaluating any given priority setting process. For instance, in priority-setting process X, how did the decided-upon priorities reflect the needs of the local context? How were the criteria contextualized? How did the process publicize its decisions? How did the process create appeal mechanisms? And how did the process ensure compliance?
Ghaffar et al (2004) echo the implausibility of finding an algorithm that automatically generates priorities "if the evidence base is somehow fed into the process".
Also see Rudan et al (2008) for their excellent list of questions to consider in assessing competing priorities – e.g. answerability ("how likely is it that the objectives will be met given the current state of science and the size of the gap in knowledge?"), equity ("how likely is it that the proposed research will benefit those who are most vulnerable"?), and existing research capacity ("how likely is it that the objectives will be met given existing research capacity?"
For specific examples of criteria arrived at within priority-setting exercises, see for instance Rosenstock, Olenec and Wagner (1998) for steps taken in determining the National Occupational Research Agenda in the USA ("seriousness of hazard; number of workers exposed; potential for risk reduction; sufficiency of existing research; expected trend in importance of topic; and probability that research will make a difference"). Also see the discussion in González-Pier et al (2006) dividing criteria into quantifiable (cost-effectiveness analyses, available resources, implementation issues) and qualitative considerations (including "an ethical assessment addressing equity implications in population groups and a discussion of social acceptability").
See Kapiriri et al (2007) for a good discussion of the stakeholders relevant to and with an interest in the prioritization process.
This definition has been informed by Gregory, Hartz-Karp and Watson (2008); McCoy and Scully (2002); London (2005); and Lavis et al (2009b). While CHSRF (2006) defines deliberative dialogues as being "a mechanism for eliciting colloquial evidence while making it subsidiary to the science," we pass no judgements here on the hierarchy of evidence.
see also McDonald, Bammer and Deane (2009), Start and Hovland (2004), National Cancer Institute (2007) for examples of various dialogue types. Section Three of this module will explore several of these in much greater detail. Also see Martin, Abelson and Singer (2002) for a list of methods to involve multiple stakeholders in priority setting – from mass approaches (hotlines, surveys) to targeted approaches (focus groups) to proxy approaches (key informant interviews) to "informed involvement" (citizens' juries).
The lack of strong evaluations of priority setting contributes further to this evidence gap – see Abelson et al (2003) and Smith et al (2009) for a fuller discussion.
This last method is of particular relevance for priority setting for service delivery. We include it here as it may be incorporated into research priority-setting processes, and its logical flows may provide useful examples or structure for more interpretive processes.
The ideal process outlined here is informed in particular by Sibbald et al (2009); COHRED (2006); the Alliance for Health Policy and Systems Research (2009); Rudan et al (2007a) and Rudan et al (2007b); and author experience in developing a priority-setting process in Mongolia.
This and all other diagrams, unless noted otherwise, were created by the author.
This may also include the development of a proposal to receive funding for the process.
For more on developing a KM strategy, see the Research Matters Knowledge Translation Toolkit (2008).
Informed by Figure 1 in Peacock et al 2009.
For more, see Alliance for Health Policy and Systems Research (2009); El-Jardali et al (2010); Walker et al (2009); and Ranson and Bennett (2009).
Adapted from el-Jardali et al 2010. Permission not required.
Source: adapted from Tomlinson et al 2007.
Informed by Figure 1 from Rosenstock, Olenec and Wagner (1998).
informed by Lomas et al 2003.
Adapted from Ghaffar et al 2004.
Informed by Ghaffar et al 2004.
Distilled from Mitton and Donaldson (2004) and Mitton et al (2003).
See online for a list of good mind mapping software tools - an essential brainstorming tool
Informed by Figure 3 from Bryson (2004).
Informed by IIED 2005.
See also the Policy Networking Tool as outlined in Crosby (1992) and further detailed in the Research Matters Knowledge Translation Toolkit (2008).
Informed by Bryson (2004).
Available online. Accessed April 12 2010.
Reproduced from National Cancer Institute 2007. Permission not required.
Available online. Accessed April 28 2010.
Adapted from McDonald, Bammer and Deane (2009).
Available online. Accessed April 28 2010.
Informed by McDonald, Bammer and Deane 2009.
This list is adapted from one provided by McCoy and Scully, 2002.
Available online. Accessed April 28 2010.

Date modified:: 2011-05-12

Deliberative Priority Setting – a CIHR KT module

Table of Contents

Section One: An Introduction

1. Knowledge Translation

2. Priority Setting

3. Priority Setting and the "Right People"

4. Priority Setting as a Deliberative Dialogue

5. Priority-Setting Processes at the National Level

6. An overview of the Module

Section Two: Priority-Setting Processes

1. Introduction

2. Ideal Interpretive Priority Setting

2.1 Planning the Process

2.2 Stakeholder identification, analysis and engagement

2.3 Knowledge management

2.4 Interpretive workshops

2.5 Translation of issues into research questions

2.6 Publication and validation

2.7 Revision or appeal mechanism

3. Participatory Priority-Setting Methods

3.1 Participatory Priority-Setting at the Regional Level: Latin America, East Asia, South-East Asia and the Middle East/North Africa

3.2 Child Health and Nutrition Research Initiative

3.3 Mongolia

3.4 The National Institute for Occupational Safety and Health

4. Listening for Direction

5. The Combined Approach Matrix

6. Programme Budgeting and Marginal Analysis (PBMA)

Section Three: Deliberative Processes

1. Introduction

2. Stakeholder Identification and Analysis

2.1 Analyzing the Stakeholder

2.2 Stakeholder Analysis: Components, Approaches and Tools

2.2.1 Planning the process

2.2.2 Identify (and train) a working/expert group

2.2.3 Develop a plan and timeline

2.2.4 Identify and discuss the issue at hand

2.2.5 Research tools for identifying and analyzing stakeholders

2.2.6 Facilitated Brainstorming Session

2.2.7 Complete a Power vs. Interest Grid

2.2.8 Stakeholder Influence Mapping

2.2.9 Stakeholder-issue interrelationship diagrams

3. Eliciting Stakeholder Input

3.1 Concept Mapping

3.2 Delphi Technique

3.3 Nominal Group Technique

4. Convening Deliberative Dialogues

4.1 The Technical Working Group – Formation, leadership, ownership

4.1.1 Visualization in Participatory Planning (VIPP)

4.1.2 The Expert Witness Panel

4.2 Larger Reference Group – wide multi-stakeholder deliberation

4.3 Other Deliberative Dialogue Models

Section Four: Priority Setting and the Deliberating Stakeholder

Experimenting, Piloting, Evaluating

Incorporating deliberative dialogue techniques into priority-setting processes

Emphasis on process

Re-imagining knowledge generation and application

Section Five: References

Notes