Stimulating Methodological and Theoretical Innovation in Population Health Intervention Research - Workshop Summary Report

Toronto ON
December 1, 2010

Table of Contents


Introduction

Population health intervention research (PHIR) uses scientific methods to produce knowledge on policy and program interventions that have the potential to affect health at the population level. Evidence from PHIR influences decision making and is vital to addressing the population health problems facing nations today.

To discuss some of the key issues surrounding PHIR and help stimulate its quality, quantity, and use, the Canadian Institutes of Health Research–Institute of Population and Public Health (CIHR-IPPH) and the Population Health Intervention Research Initiative for Canada (PHIRIC) held an inaugural symposium, Accelerating Population Health Intervention Research to Promote Health Equity, in Toronto on November 29 and 30, 2010.

The symposium raised a number of methodological issues related to PHIR, including the advantages and disadvantages of different study designs (e.g., randomized control trials, natural experiments, and case studies). In order to delve more fully into these issues and help foster advances in theories and methods related to PHIR, CIHR-IPPH and PHIRIC hosted a workshop immediately following the symposium.

Approximately 60 researchers from Canada, the US, the UK, France, and Australia met in Toronto on December 1, 2010, for the workshop, entitled Stimulating Methodological and Theoretical Innovation in PHIR. Participants were encouraged to come to an understanding of the range and mix of methods needed to address important questions and to contribute to the development of a consensus statement on the meaning of PHIR.

The workshop was organized with the guidance of an expert Planning Committee and sponsored by CIHR-IPPH, the Public Health Agency of Canada (PHAC), and the CIHR Institute of Nutrition, Metabolism, and Diabetes.

Objectives

The objectives of the workshop were to:

  • create a forum to contrast, debate, and discuss the different theoretical and methodological underpinnings of PHIR in order to advance the science of population health interventions; and,
  • identify priority areas for theoretical and methodological development in PHIR.

Processes and Methods

The workshop was organized around three chaired sessions, each of which focused on a different aspect of PHIR and specific questions for consideration. Expert presenters opened the sessions by framing some of the methodological and theoretical challenges researchers face in assessing the impact of population health interventions, while discussants reflected on the presentations with their own complementary and contrasting perspectives. Participants were then given opportunity to add their own thoughts and insights, which are highlighted in the last section of this document.

Sessions

Session 1: Evaluating the Effectiveness of Population Health Interventions

Chair: Dr. Alan Shiell, University of Calgary

Key Questions:

  1. Is it possible to retain the benefits of randomization and yet allow interventions to be adapted to a local context? If so, how do we maintain and assess fidelity?
  2. How do we generalize from intensive case studies?

Presenter: Dr. Michael Oakes, University of Minnesota

The gold standard for population health interventions is randomized group or community trials (i.e., that exploit intact social groups); however, such trials can be expensive and difficult to manage and replicate. One benefit of randomization is the production of exchangeable counterfactual substitutes that facilitate the identification of intervention effects, if the sample size is sufficiently large. There are three elements to identifying these effects: positivity (probability of exposure), exchangeability, and consistency (similar treatment across subjects).

In PHIR, more emphasis needs to be placed on ruling out competing explanations and hypotheses through well-executed experiments and case studies. Case studies are the foundation for the way we think about most things, yet they are vastly underutilized in PHIR. There is a need for more well-researched and well-documented case studies involving critical thinking and the marshalling of evidence to draw credible inference.

Case studies are the foundation of the way we think about most things, yet they are vastly underutilized in PHIR.

Rather than regard generalizeability as the "average effect", it should be thought of in terms of transferring an intervention that works in one place to another place. Care must be taken when generalizing from case studies: knowing and incorporating local details is essential to success. While it is beneficial to tailor interventions to local groups, "perfect tailoring" means there is no replication and "no tailoring" means the intervention may be inappropriate to some communities.

Discussant: Dr. Louise Potvin, University of Montréal

Cluster-randomized trials are not a proper gold standard for population health interventions. Randomization is an imperfect approximation of an experimental counterfactual that requires assumptions, and the constant-effect assumption it requires cannot be met. The fundamental problem of causal inference is that causality is impossible to observe directly due to differences in potential effects resulting from exposure to particular causes (i.e., interpreting a main effect in the presence of an interaction effect). Rather, it is inferred based on scientific work or statistical theory.

The effective component of an intervention is located in the interaction with contextual characteristics.

The effective component of an intervention is located in the interaction with contextual characteristics. Locally, the proper counterfactual for causal inference is a long series of pre-intervention observations. Much larger collections of interventions and intervention studies are needed to develop theories and methods for a proper science of population health interventions.

Session 2: Health Equity Considerations in PHIR

Chair: Dr. Jeannie Shoveller, University of British Columbia

Key Questions:

  1. How can health equity considerations be embedded in our research designs to study population health interventions?
  2. What theoretical and methodological developments are needed to generate the required empirical evidence about how population health interventions effectively reduce inequities?

Presenter: Dr. Margaret Whitehead, University of Liverpool

There are three main approaches to reducing inequities: improving the health of the most disadvantaged, narrowing the health divide, and reducing the social gradient in health throughout the population. National and regional strategies in Britain have tended to focus on the first two and to drift downstream toward individual lifestyle interventions rather than toward upstream approaches on wider social determinants. These trends have reduced their potential effectiveness.

Three main approaches to reducing inequities are to improve the health of the most disadvantaged, narrow the health divide, and reduce the social gradient in health throughout the population.

While information on both relative and absolute reductions in inequalities resulting from public health initiatives is important to policy making, differences between the two can be difficult to explain. The intent in public health is not to reduce inequalities in health by levelling down, but rather to improve overall population health and reduce inequalities by leveling up.

The goal is to reach equitable objectives in the most efficient way possible. Efforts are needed to advocate for the increased visibility of health inequities and the differential impacts of policies; the intelligent application of targets and performance indicators; goals that level up and not down; the use of logic models for strategies; and realistic timescales for achieving targets.

Discussant: Dr. Jason Robert, Arizona State University

Public and population health research is fundamentally an exercise in uncertainty that faces the dual challenge of incomplete knowledge and dynamic populations. While public health action should be informed by evidence, mere evidence that health inequities exist reveals nothing about them. If and when science comes into play, it does so alongside prevailing values, priorities, and other factors in the decision-making context. In Canada, for example, detailed descriptions of patterns of inequality or disparity in health have led to an ethical imperative to address them, yet some regard such efforts as beyond their responsibility, misguided, wasteful, or otherwise. The question is how to influence people in a pluralistic democracy where there may be few or no universally shared ethical values.

Advocacy and lobbying are needed to influence policies and achieve public health action—and while such action should be informed by evidence, decisions depend on prevailing values and priorities.

Such challenges demand collaborative and sustained efforts in the area of empirical assessment and normative and political work. It is not sufficient to ask people what they think; they also need to be asked why they think that way and perhaps why they should consider thinking another way. The Frameworks Institute suggests framing fairness in terms of disparities between places rather than people. Those with vested interests and perspectives to share must be brought to the table to create a deliberative forum.

It is not sufficient to ask people what they think; they also need to be asked why they think that way and perhaps why they should consider thinking another way.

Session 3: Perspectives from Research Funders

Chair: Dr. Shawna Mercer, US Centers for Disease Control

Key Questions:

  1. Based on what you heard today, which research areas does your organization see itself supporting to further methodological and theoretical underpinnings in population health intervention research? What makes these research areas more compelling priorities?
  2. How might we best encourage international collaborations to advance such novel developments in PHIR?
  3. Where do you see the field of PHIR five years from now?

Presenter: Dr. Jean-Baptiste Herbet (Institut national du cancer, France)

France is a newcomer in this area of research, having issued its first call for applications for PHIR on cancer prevention in 2010. Efforts are being carried out under a government action plan focused on three main areas, one of which is to reduce inequalities in health (social inequalities, in particular). France's national cancer institute is funded by the national department of health and research, so is in a good position to inform politicians about what works and what doesn't in terms of interventions. The PHIR symposium and workshop were a step toward promoting international collaboration in such areas as peer review and strengthening ties between researchers and policy-makers.

Presenter: Dr. John McCallum (NHMRC, Australia)

In Australia, where epidemiological studies receive the largest proportion of health research funding, a new preventive health agency is being established to focus on social and translational research. There is interest in collaborating with Canada on large, complex interventions to optimize impacts on a range of structural changes without widening socio-economic gaps and also in the area of peer review—possibly through the sharing of panel members and common training opportunities. Advances envisioned in five years include public health researchers providing training and advice to panelists, chairs, assessors, and reviewers; improved health performance in marginalized groups; and the implementation of findings from epidemiological research in public health efforts.

Presenter: Dr. Mark Petticrew (London School of Hygiene and Tropical Medicine)
with remarks prepared by Dr. Peter Craig (MRC-UK)

Existing theory and methods for PHIR should be put to better use, as radical changes are unlikely compared to small refinements from lessons learned. In addition to methodological advances, clear thinking is needed to improve understanding of complex interventions. International collaboration is a way forward, with the International Tobacco Control policy a potential model for such efforts. Given differences in contextual constraints, collaborations should focus on developing common resources to support PHIR, including infrastructure that can be used internationally (e.g., shared datasets). Research priorities should be developed systematically using formal evaluation and implementation techniques.

Presenter: Dr. Eduardo Simoes (US CDC)

The Prevention Research Centers (PRC) program at the US Centers for Disease Control has distributed about $250 million over five years to 37 universities for investigator-initiated research. The program's research priorities include health promotion and disease prevention issues; policy and system-wide changes; and continued funding for evidence-based prevention research. More than a dozen new and effective interventions have been developed through the PRC program, many of them already adopted widely in the US. Another focus of the program is on stimulating the implementation of natural experiments of policy and environmental changes that, intentionally or not, affect health outcomes. Envisioned five years from now is the development of innovative approaches in design and methods to more effectively and efficiently incorporate the complexity of intervention context interaction; the development of new metrics and indicators; and greater collaboration in PHIR implementation involving two or more countries.

Discussion

Promoting Population Health Intervention Research

  • International collaboration is essential.
  • The principles of and considerations for strong PHIR need to be defined.
  • Researchers can promote support for PHIR by serving on peer review panels, submitting grant applications, conducting high-quality studies and publishing results, and improving their ability to advocate and communicate effectively with decision makers—in particular, telling compelling stories, highlighting different perspectives, and speaking to different audiences is essential.

Priority Setting

  • The voice of community members is needed to help to define research priorities.
  • A more fulsome discussion is needed about the importance of embedding notions of equity and the policy, program, and funding implications of population-attributable risk and the global burden of disease into priority setting for PHIR.
  • Being thoughtful up front about what is being examined can help prevent funding from being misdirected by highlighting important areas and steering attention away from those with a dearth of information.
  • Epidemiologists should be encouraged to build in health equity considerations into their work.
  • We need to distinguish between maximizing health and economic advancement and put more thought into how each affects the other.

Randomized Control Trials

  • Greater consideration of proof of concept and a better understanding a priori are needed to show causal pathways.
  • Cluster RCTs may not have the capability to link context to outcomes; however, they enable the examination of a dose-effect approach.
  • RCTs are useful for structural interventions; however, for agentic interventions, where social theory comes into play, an understanding of why people are changing or not is required.

Design and Methodology

  • Every design has flaws.
  • A wide range of study methods available for PHIR are underutilized or have failed simply because they were not appropriate to the situation or the mindset for using them was not there.
  • There will never be consensus on a gold standard.
  • If the question is about whether an intervention can work, experiments are usually more useful; if it is about whether it does work, that is usually easier to determine through observation.
  • If a process can be adapted across contexts and there is good measurement and knowledge of its key processes and active ingredients, it may be replicable even if it looks different in different contexts.
  • Recent work on the use of models to define what would happen in the absence of treatment and that replicate what we see in the real world has led to multiple new methods. Setting boundaries around what defines optimal PHIR locks research into quantitative methods and randomized trials.
  • No model can begin to comprehend the complexity of a group of people because so many contextual factors cannot be controlled for (e.g., social, emotional, cultural, historical). Other methods, especially natural experiments combined with strong, qualitative case studies, are much better for PHIR.
  • Any kind of natural experiment in which there is an intervention reveals more theoretically than can be learned from a stable system. Perturbation of the system may reveal more about the causal mechanism.
  • If one looks at a naturally developed intervention through the lens of two or more distinct observational research methods and they point in the same direction in terms of outcome and intervention relationship that is a strong indication of a possible intervention effect. Following up by using more traditional research methods may then increase understanding of the causal association between the intervention and the health outcome.
  • Research is focused on things that are measurable at the environmental level and on individual-level behaviours. Thinking about more complex, location-based variables (e.g., relational ones) would be more helpful.

Interactions

  • In looking for interaction effects, it is helpful to ensure that potential effects within sub-groups (e.g., sex, place, and ethnicity) are pre-specified and theoretically driven.
  • There is interaction between treatment and context; the question is how idiosyncratic is the context.
  • The distribution of adverse effects is systemic, not random.
  • Interventions that aim to prompt interaction effects should be examined more closely because feedback must be amplified in order to change a complex system.

Health Inequities

  • Equity discussions have focused mainly on reducing disparities among the most disadvantaged. More attention needs to be paid to what people are doing to themselves collectively.
  • Society is diverse, so talking about what is collectively known or valued is impossible.
  • The focus should be on social determinants of health inequities rather than social determinants of health.
  • Specifying or singling out sub-areas of deprivation ignores the fact that some people experience oppression at multiple levels.
  • It needs to be determined if and where to draw the line between improving health and improving health equity and whether interventions should be discarded simply because they increase health inequities.
  • The measurement of equality or inequality does not indicate if a policy is good or bad. The only way to go beyond a distributive approach is to incorporate not only normative dimensions but also other ways of thinking informed by other disciplines in our approach to conceptualizing inequalities.

Relative vs. Absolute Risk

  • Two groups moving in parallel toward lower risk but growing apart a bit in the process would be viewed largely as a positive outcome—therefore, absolute differences matter, not relative ones.
  • Discussions should focus on real rates, trends of real rates, and risk differences rather than on relative risk.
  • Stratification means that more effort may be needed to achieve the same target in different sub-groups.
  • There is very little fluidity through the social stratification index, so there is no reason to expect differences in the health realm.

Values in Decision Making

  • In addition to developing new theories and methods, there is a need to redefine innovation and challenge existing values.
  • Instead of asking people for their opinions on slogans about equity, it is better to ask them about their values because they are more likely to endorse equity from that perspective.
  • Whether one takes a utilitarian perspective to increase overall health or an equitable perspective to redistribute health more equitably, it is all a question of values.
  • Sometimes, the way in which the importance of a problem is communicated causes people to do the right things for the wrong reasons (e.g., out of pity or misplaced compassion, rather than values anchored in social justice).
  • Evidence does not necessarily give rise to solutions.
  • There will always be some people worse off as the result of a policy than they were beforehand.
  • Even no policy is a policy.
Date modified: