Section 2.2: Knowledge Synthesis
Jennifer Tetzlaff, Ottawa Hospital Research Institute
Andrea C. Tricco, PhD, Ottawa Hospital Research Institute
David Moher, PhD, Ottawa Hospital Research Institute
Definitions*
- Knowledge syntheses (KS) consist of a clearly formulated question and use systematic and explicit methods to identify, select, critically appraise, and interpret data from relevant research
- A meta-analysis is a statistical technique used to quantitatively integrate the results of included studies in a KS
- A KS does not necessarily include a meta-analysis
* From the Cochrane Collaboration
Knowledge Synthesis
- KS is used to interpret individual study results within the context of global evidence
- KS can be used as a starting point for practice guidelines and new primary research (e.g. trials)
- KS bridges the gap between research and decision-making
Types of Evidence
- Many groups worldwide conduct KS and methods involved usually depends on the question(s) being considered
- Questions regarding the effectiveness of interventions will usually include quantitative evidence (e.g. odds ratio for a particular drug versus placebo)
- Contextual questions about why an intervention worked in a particular context will usually include qualitative evidence
General Methods of a KS
- Incorporating qualitative evidence into KS can be:
- Challenging
- Difficult to locate qualitative evidence
- Difficult to integrate qualitative evidence with quantitative evidence
- Methods are just emerging
- As such, the focus will be on general methods applicable to most KS
Overview
- Assembling the review team
- Formulating the question, protocol, and eligibility criteria
- Identifying relevant studies
- Selecting studies
- Risk of bias assessment
- Data extraction
- Data analysis
- Presenting results
- Interpreting results
The Review Team
- Determined by the question
- May include the following people:
- Clinical experts with knowledge of the topic
- Methodologists who know the KS process
- Librarian to help locate relevant studies
- Researchers who conducted studies on the topic
- Funder or commissioning agency for context
- Statistician if meta-analysis is being considered
- End-users (e.g. policy makers, patients) to increase relevance and uptake
Formulating the Question
- Most important step because it guides the KS process
- PICO (S or T): Population, Intervention, Comparators, Outcome, and (Study design or Time period) facilitates question development
- May not fit all KS (e.g. Intervention sometimes replaced by Exposure) but still useful template to consider
Formulating the Protocol
- Pre-specifies the review process
- Important because it decreases post-hoc changes to methods and selective outcome reporting
- Elements include primary versus secondary outcomes, search methods, appraisal of the literature, and data abstraction
- Any changes to the protocol should be transparently reported in the review write-up
Formulating the Eligibility Criteria
- Should extend from the question
- Based on PICO (S or T)
- Consider language of publication
- Consider publication status (e.g. published vs. unpublished material)
- Needs to be thoroughly considered, properly defined, and transparently reported
Identifying Relevant Studies
- Based on the question and PICO (S or T)
- MEDLINE, EMBASE, and The Cochrane Library electronic databases are commonly used for health-related research
- At least 2 relevant databases should be searched
- Advisable that a librarian guides this process
- Should search for unpublished and difficult to locate (i.e. grey) literature (e.g. trial registries, public health agency websites)
Selecting Studies
- Based on the eligibility criteria
- 2 stages: broad screening of titles and abstracts and stricter screening of potentially relevant full-text articles
- 2 independent reviewers should screen at all levels to ensure relevant studies aren't missed
- Agreement between reviewers can be assessed using the kappa statistic
Risk of Bias Assessment
- Many assessment tools have been developed to asses the risk of bias for different study designs
- Only reporting a summary score is not advisable; the assessments for each criterion from the quality appraisal should be reported for each study
- Sensitivity analyses on risk of bias can be conducted versus excluding studies based on risk of bias
Data Extraction
- Primary outcomes should be differentiated from secondary outcomes
- Data extraction form(s) should be developed a priori and pilot-tested to increase reliability
- Potential errors are decreased if more than one reviewer independently extracts the data
- Authors of included studies should be contacted for missing or unclear information
Data Analysis
- Depends on the question and type of data collected
- All KS must have a narrative synthesis of results and risk of bias
- Standard effect measures (e.g. odds ratios, hazard ratio) may be used
- Meta-analysis may not be possible or advisable if outcomes were assessed inconsistently and clinical, methodological or statistical heterogeneity is observed
Presenting Results
- Screening process should be described in the text and/or presented as a flow-chart
- Characteristics of included studies should be described in the text and/or a table (e.g. participant populations, interventions)
- Results of risk of bias assessments should be presented in a table and/or text
- Quantitative data should be presented as summary data (e.g. effect estimates with confidence intervals for each study) and may be presented for each outcome in a table or in a forest plot figure
- Qualitative data can be presented visually (e.g. conceptual framework)
Interpreting Results
- Discuss risk of bias, strength, and applicability of the evidence for each outcome
- Relevance of the results should be considered for key stakeholders to increase applicability
- Qualitative evidence can help explain how the intervention worked and whether it will work in a different setting
- Should consider study and KS limitations
Disseminating Results
- Most common form of dissemination is publication in peer-reviewed journals
- Open access journals will increase dissemination
- Targeted dissemination via media for the public, brief reports for health care providers, policy makers and consumers, and decision-aids for patients
Uptake of Results
- Much attention has been paid to enhancing the quality of KS but relatively little has been done regarding the format for presentation to enhance uptake
- Resources are available to make KS more user-friendly (e.g. Clinical Evidence and Program in Policy Decision-Making/Canadian Cochrane Centre database).
Future Research
- Increasing the uptake of KS
- How best to update KS
- Comparability between different types of KS (e.g. rapid reviews versus conventional reviews)
- How to prioritize KS topics
Contact Details
- Ottawa Methods Centre, Clinical Epidemiology Program, Ottawa Hospital Research Institute
- JMT: jtetzlaff@ohri.ca
- ACT: atric060@uottawa.ca
- DM: dmoher@ohri.ca
Acknowledgments
- We thank CIHR for their funding and Dr Sharon Straus for her funding and useful feedback on our paper
- ACT: CIHR (Frederick Banting and Charles Best Canada Graduate Scholarship and a Rising Star Award from the Institute of Health Services and Policy Research)
- DM: University of Ottawa Research Chair
- Date modified: