Introduction to Evidence-Informed Decision Making


Developed by Donna Ciliska, RN, PhD
Professor, McMaster University, and
Scientific Director, National Collaborating Centre for Methods and
Tools

How do I use this learning module?

Estimated total time: 5 hours

Objective: To understand the components of evidence-informed decision-making.

Process: This module is built on a scenario that will allow you to understand and apply each stage of the Evidence-Informed Decision Making process. Once you read the scenario, you can go to any section. (Time estimates are in brackets).

Links: Each time you see the word scenario, it is linked to the actual scenario and will take you there if you click on it. Similarly, the key terms are linked to a definition in a glossary.

Overview

  1. What is Evidence-Informed Decision making? Why bother? (0.5 hours)
  2. Ask. How do I frame the question? (0.5 hours)
  3. Acquire. How can I find the best evidence in five minutes or less? (1 hour)
  4. Appraise. How can I decide if the particular study is good enough to apply? (0.5 hours)
  5. Integrate. How do I decide which of multiple studies to use? (0.5 hours)
  6. Adapt. How do I use the information from #5 in decision-making/policy brief? (1 hour)
  7. Apply. How do I develop the implementation plan? (0.5 hours)
  8. Analyze. How do I evaluate if the plan worked? (0.5 hours)
  9. Glossary

Scenario:

Obesity is on the rise in Canada. You have been invited to be part of a national panel of practitioners, researchers, and program-level and government-level policy makers drawn together for a think tank to come up with evidence-informed recommendations for the prevention of obesity in children. Your own expertise is related to school-based programs for children in primary schools (roughly 5-13 yrs old). You have heard the words evidence-informed-decision-making but, before you go to the panel, you want to be sure you understand the process. You find this on-line educational module to help you!

Part 1. What is Evidence-Informed Decision making? Why bother? (0.5 hours)

Consider:

  • Billions of dollars are spent annually on health-related research
  • It takes approximately fifteen years to get research into recommended policy and 40% implementation in practice (Antman et al, 2001)
  • 30-50% of people received recommended care (Schuster et al, 1998)
  • 30-40% of patients do not get treatments of proven effectiveness (Grol, 2001)
  • 20-30% of patients received acute care that was not needed or that was potentially harmful (Schuster et al, 1998)

Is there any reason to believe it is any different in your own area of expertise?

These statistics give you some perspective on the current
research → practice gap
that exists. The evidence-based medicine (EBM) movement was born of the realization of the gap, and began an effort to bridge the
research → practice
and
research → policy gaps.

A brief history:

EBM as a term was first coined by Gord Guyatt in 1992, and a commonly used definition was offered by Dave Sackett and colleagues:

the conscientious, explicit and judicious use of the current best evidence in making decisions about the care of individual patients (Sackett, 1996)

The Evidence-Based Medicine Work Group, under the leadership of Guyatt, published a series of 25 articles for the Journal of the American Medical Association (JAMA) between 1993 and 2000 that outlined criteria to evaluate current evidence to support clinical decisions. These Users Guides have formed the basis of most of the critical appraisal tools that exist and were revised and published together in book form (Guyatt & Rennie, 2002). EBM was recently named one of the top ten most important innovations in health care in the last 150 years, in a poll held by the British Medical Journal (BMJ), arguing that EBM allows the other innovations to be implemented.

EBM has expanded to include many disciplines such as evidence-based nursing, veterinary medicine, dentistry, policy-making and pastoral care. In every case, evidence-based healthcare involves practitioners or policy makers using their expertise to combine the best available evidence, knowledge of available resources with patient or population circumstances, values and preferences in decision-making.

How could you argue with the goal of bridging that
research → practice gap?

There has been a backlash to EBM related to the strongly held myth that the only acceptable evidence is from randomized controlled trials or meta-analyses. We have begun to use the term Evidence-Informed Decision Making (EIDM) to attempt to get beyond some resistance to EB practice, and to connote that other types of evidence are useful in making decisions.

In your reading of other material, you will see many other terms such as knowledge translation or knowledge exchange. In a project to set up search terms to find this literature, Ann McKibbon and colleagues at McMaster University have come up with over 60 terms for this idea of getting research evidence into practice and policy! Just think of them as terms related to this enterprise of getting the research evidence into decision-making.

Practice!

Take a few minutes to jot down some answers to the following questions.

  1. What are the barriers for you in getting research evidence into your work (whether patient care, program design, policy design or research study design)?
  2. What are the blocks that have to do with you?
  3. What are the blocks that have to do with your job design?
  4. What are the blocks that have to do with your organization?
  5. What are the blocks that have to do with your community?

Barriers:

Much research has gone into studying the barriers to using research evidence in practice. Here are the big ones:

  • Time
  • Inability to access research
  • Inability to understand the language of research
  • Lack of critical appraisal skills
  • Lack confidence in making change based on research evidence
  • Lack of sense of control over practice
  • Culture resistance to change, decisions based on history
  • Lack of organization valuing or supporting evidence-based practice
  • Lack of consensus on what constitutes evidence.

Note: many hours have been spent debating what constitutes evidence. There are many different definitions from many different disciplines. In the context of this learning module, the term research evidence will be used when discussing studies. Otherwise, evidence will connote the broader definition, including other forms of knowing.

A few studies have looked at characteristics that influence whether or not a change in practice happens as a result of new research evidence. They have to do with issues related to the:

  • Individual e.g., age, education, years in same position
  • Organization e.g., size, complexity, research done in that organization, affiliation with a university
  • Environment e.g., rural/urban, university or research centre in geographic area.
  • Evidence itself e.g., complexity, difference from current practice, resource requirements.
  • So we are beginning to know what is blocking us from putting evidence into practice. How do we get beyond those barriers? There are some active areas of research, today, on how to change practice and less on how to get research evidence into policy. More on that in section 7, Apply.

How does knowledge get research evidence into practice and policy? - the CIHR (2007) Knowledge to Action Framework

Graham and colleagues developed a framework entitled Knowledge to Action to integrate knowledge creation and application (Graham et al., 2006). They described a knowledge funnel where knowledge is created, aggregated, distilled and tailored into clear and practical products for use in the field. Central to the Knowledge to Action Process is that, at each phase of knowledge creation, producers customize knowledge products to the unique needs of potential users. A process of collaboration throughout the process, including the question phase, has the potential to make the research more meaningful and useable for practitioners, managers and policy developers.

The action cycle circles the knowledge creation funnel. The action cycle contains 7 activity phases that are recommended to support the translation of knowledge to users. The activities are taken from a review of 60 theories of planned action (Graham et al 2006).

The model recognizes the nonlinear process of knowledge translation: the authors expect that each action phase can be influenced by the preceding action phases, and by feedback between the phases.

You will see that this model fits with the steps of Evidence-informed decision-making. As a clinician, manager or policy-maker, you will be starting with the box at the bottom of the diagram identifying a problem, identifying, reviewing and selecting knowledge, then moving to the left on the circle, adapting the knowledge to the local context, considering barriers to the implementation, selecting interventions, monitoring their use and evaluating outcomes. The final stage feeds back to where we started with the identification of problems.

Other Resources

References

  • Antman, E.M., Lau, J., Kupelnick, B., Mosteller, F., Chalmers, T.C. (2001). Acomparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA. 1992 Jul 8;268(2):240-8.
  • Graham, I., Logan, J., Harrison, M.B., Straus, S.E., Tetroe, J., Caswell, W., & Robinson, N. (2006). Lost in Translation: Time for a Map? The Journal of Continuing Education in the Health Professions, 26(1), 13-24.
  • Grol, R. (2001). Successes and failures in the implementation of evidence-based guidelines for clinical practice. Medical Care, 39, 8 (S2), II46-54.
  • Guyatt, G. & Rennie, D. (Eds) (2002). Users Guides to the Medical Literature: A manual for Evidence-Based Clinical Practice. American Medical Association.
  • Sackett. D.L., Rosenberg, W.M., Muir-Gray, J.A., Haynes, R.B., & Richardson, W.S. (1996). Evidence-based medicine: what it is and what it isnt. BMJ, 328, 535-536
  • Schuster, M.A., McGlynn, E.A., & Brook, R.H. (1998). How Good Is the Quality of Health Care in the United States? Milbank Quarterly, 76 (4), 517-563

Part 2. Ask. How do I frame the question? (0.5 hours)

The first stage of evidence-informed decision-making is to frame the question. Using the scenario about developing recommendations regarding prevention of obesity in children, you decide to update your own knowledge.

  1. You search PubMed on obesity and get 17,826 hits. Not a chance you are going there!
  2. You try to limit your search by adding children, resulting in 15, 238 hits. Still no way you can get through those. Most people give up at this stage, and decide to go with whatever studies they have in their personal files.

Practice, research and policy questions can be of many types incidence/prevalence; burden of suffering and quality of life, experience and meaning of illness, causation, harm, assessment (or diagnosis), prognosis, costs. Frequently, however, most of us are concerned with effectiveness of therapy or interventions.

When considering intervention or therapy questions, you can use the following PICO format to help you be very specific about your question. This also helps to formulate your search of the research evidence.

Patient / Population
Who are the people involved individuals, families, populations? Age? Specific problem or prevention issue?
Intervention
What specific preventive, therapeutic, health services strategies are you considering?
Comparison
What is current usual care or usual circumstance or standard of care?
Outcome
What are the patient or population level outcomes of most significance?

When looking at context or experiences, these questions are best answered by qualitative research, so the P S question frame is used:

Patient / Population
Who are the people involved individuals, families, populations? Age? Specific problem or prevention issue?
Situation
What circumstances or experiences do you want to know about?

Practice!

  1. Using the scenario about developing recommendations regarding prevention of obesity in children, you decide to update your own knowledge related to school-based interventions. How will you frame the question? Jot down how you would apply the PICO format before going on to the next page:
    • Patient / Population
      Intervention
      Comparison
      Outcome
  2. From your answers, compose a focused question with all the above parameters.
  3. Possible answer:
    • Patient / Population: school children, aged 5-12
      Intervention addition of a daily physical activity program
      Comparison usual health and physical education curriculum alone
      Outcome rate of obesity, rate of overweight, mean body-mass index (BMI), mean weight for height

    So you might write the question as:

    What is the effectiveness of daily physical activity programs done in the schools, compared to the usual health and physical education curricula alone, on the rate of obesity in school children?

  4. A related qualitative question might be:
    • Patient / Population obese school children
      Situation school physical activity program

    You might want to know if obese children are teased or ridiculed by other children or teachers when they participate in physical activity programs. Do they experience discrimination? Are they made to feel inadequate?

    One possible qualitative question (phenomenology) would be:

    What is the experience of school children when they participate in physical activity programs at school?

Other Resources

References:

  • Fineout-Overholt, E., & Johnston, L. (2005). Teaching EBP: Asking searchable, answerable clinical questions. Worldviews on Evidence-Based Nursing, 2, 157-160.

Part 3. Acquire. How can I find the best evidence in 5 minutes or less? (0.5 hours)

Using the question you framed in Part 2, What is the effectiveness of daily physical activity programs done in the schools, compared to the usual health and physical education curricula alone, on the rate of obesity in school children?, how would you begin a search to get yourself the best evidence?

As mentioned in Part 2, just searching PubMed on obesity or obesity in children yielded over 15,000 hits. One of the downsides of the information age is clearly information overload!

A hierarchy of quantitative evidence has been proposed to indicate that some study designs are considered stronger, that is, more free of bias, than others. No study of people, designed by people, could ever be totally free of bias, but as you go down the hierarchy, there is a likelihood of greater bias. The use of this hierarchy has led to one of the unfortunate myths of evidence-informed decision making, that only systematic reviews or randomized controlled trials can be considered evidence. The reality is that all study designs are forms of evidence. When you are making a decision about instituting a change in practice or policy, however, you would want to use the evidence from the highest point in the hierarchy that you could find. For some questions, that may very well be case control studies or expert opinion!

Hierarchy of quantitative evidence

  • Systematic reviews of randomized controlled trials
  • Randomized controlled trials
  • Systematic reviews of nonrandomized or cohort studies
  • Cohort studies
  • Case control studies
  • Case series
  • Case report
  • Ideas, editorials, opinions
  • Animal research
  • Invitro (test tube) research

It is very important to note that this hierarchy does NOT include qualitative research. Qualitative research answers questions about experience or meaning. It is a parallel to the hierarchy of quantitative evidence.

Brian Haynes has described a pyramid (the 5S Pyramid), for finding the best evidence with the least amount of time and effort (Haynes, 2007).

Systems are electronic systems that might be so sophisticated as to be linked to patient records and prompt practitioners regarding guidelines for care (e.g., what tests to order or what interventions need to be done). For example, for a patient with type II diabetes, it would prompt the care-giver that blood work, eye exam, foot exam and diet review need to be done.

The systems level also includes guidelines. You can access guidelines from sites such as the National Guidelines Clearinghouse or the Registered Nurses Association of Ontario Best Practice Guidelines. Many of the guidelines are free full-text.

Summaries are usually text-based and are related to a specific disease or condition. An example includes Clinical Evidence. Unfortunately, you must subscribe either individually or via an institution.

Synopses are brief reports (1-2 pages) of pre-appraised individual studies or systematic reviews which give key methodological details and results, along with an expert commentary on issues of applying the results in practice. Examples of synopses are found in the evidencebased journals (eg: Medicine, Nursing, Dentistry, Health Policy). There are currently 23 such journals. For example, see Evidence-Based Nursing or Evidence-Based Dentistry. Some of these are open access, whereas others require a subscribtion, either individually or via an institution.

Syntheses include systematic reviews of all studies that could be found on a particular focused question. These include The Cochrane Library, the Agency for HealthCare Research and Quality (AHRQ) Evidence-based Practice Centre Reviews. Both groups have similar, rigorous methods for review.

Another source for systematic reviews and summaries about interventions related to public health in Canada is the Effective Public Health Practice Project. Further, Health-Evidence.ca rates reviews that are relevant to public health in Canada, summarizes them and provides recommendations for practice and policy that arise from the reviews.

Studies are individual studies related to a particular focused question. There are several searchable databases that would help you find individual studies. The most used in Canada include Medline, PubMed and CINAHL, but there are many other specialized databases. PubMed is good to know as it is a free-access version to Medline. PubMed provides a full citation and abstract, as well as links to any full-text articles that are free to the public.

As mentioned in Part 2, when we searched PubMed for any type of study related to obesity in children, we came up with over 15,000!

Exploring search strategies

Using the search Pyramid above, start any search from the top.

Back to the question from our scenario: What is the effectiveness of daily physical activity programs done in the schools, compared to the usual health and physical education curricula alone, on the rate of obesity in school children?

  1. Systems:

    Most of us do not have sophisticated systems where we work but we can get free on-line access to guidelines. If you find a well-done guideline that was recently published, you might start and end your search there!

    Open up another window in your browser so that you can keep this window open and search for some information at the same time. In your 2nd window, go to National Guidelines Clearing.

    Guidelines are usually done for broader questions, such as prevention of obesity in children, or treatment of obesity in children rather than focused on something like daily physical activity. So, you type on the search box obesity and children. You will get over 100 hits or references a bit too much to sort through! Your next step would be to use their Detailed Search that prompts you for keywords, conditions, age. As those come up, try the following limits.

    • Keyword: physical activity
    • Disease/Condition: obesity
    • Treatment/Intervention: physical activity
    • Age Range: Adolescent (13 to 18 years), Child (2 to 12 years)
    • Sort Order: Relevance

    What you get now is between 10-20 hits. Much more manageable!

    Here are a few titles:

    • Increasing physical activity in schools: kindergarten through eighth grade. University of Iowa Gerontological Nursing Interventions Research Center, Research Translation and Dissemination Core - Academic Institution. 2005 May. 42 pages. NGC:004518
    • Primary prevention of childhood obesity. Registered Nurses Association of Ontario - Professional Association. 2005 Mar. 88 pages. NGC:0042
    • Active healthy living: prevention of childhood obesity through increased physical activity. American Academy of Pediatrics - Medical Specialty Society. 2006 May 1. 9 pages. NGC:004962 65
    • Overweight in children and adolescents: pathophysiology, consequences, prevention, and treatment. American Heart Association - Professional Association. 2005 Apr 19. 14 pages. NGC:004277

    All look like they have potential, but the 1st and the 3rd actually have physical activity in their title, so you will check those out first.

  2. Synopses

    One barrier to using evidence in decision-making is lack of skills in critical appraisal. Synopses are available where different groups, like the McMaster Health Knowledge Refinery, have gone to all the work of finding and critically appraising articles, including reviews and single studies.

    For public health policy and practice questions, one of the best sources of information is from Health-Evidence.ca, which is described in more detail with practice tips under syntheses. Staff conduct a thorough search for reviews, then critically appraise the and rate them as strong, moderate or weak. Strong reviews get a 2 page synopsis a summary of key methods issues with recommendations for practice and policy.

    Other sources of synopses are the evidence-based journals. Unfortunately, you need to belong to a consortium through your local university or hospital or government system to access those. However, these journals are a fantastic source of pre-appraised information. The staff rate every study in a wide range of healthcare journals using pre-set quality criteria, select only the best quality studies and summarize those studies in a 1 or 2 page abstract with the implications added by an expert in the field. For policy makers, Evidence-Based Health Policy and Management and Evidence-Based HealthCare deals with more policy and systems issues, where as Evidence-based Dentistry or the Evidence-Based Nursing are obviously more discipline focused, and Evidence-Based Mental Health is more condition-focused.

    Pratice!

    If you have access to a library consortium, search a few of the evidence-based journals for a synopsis, review, or single study that answers the question posed in Part 2:

    What is the effectiveness of daily physical activity programs done in the schools, compared to the usual health and physical education curricula alone, on the rate of obesity in school children?

    One example found that was relevant for the scenario is:

    • Kerr, C.M. (commentator) (2000). A school based, interdisciplinary curriculum in grades 6 and 7 reduced obesity in girls. Evidence-Based Nursing, 3: 13.

    Note that the title gives you the conclusion of the study.

  3. Syntheses

    Syntheses may be summarized into a synopsis, as discussed in step 2 above. However, they involve a group of people doing a thorough search on a particular therapy for a particular problem. There are many different terms used, but, most often, a systematic review means that the review team undertook a very broad and in-depth search for published and unpublished studies, had at least two people independently rate relevance and quality and do data extraction for the primary studies, then put together the results in a meaningful way that goes beyond just describing each study in detail. A systematic review must be done in order to get to a metaanalysis, where, if appropriate, statistical results from individual studies are combined to give an estimate of the overall effect.

    The Cochrane Library is a prime source for syntheses. Review groups from around the world follow a rigorous process to find, critically appraise, then synthesize the results from all the relevant studies. (See the Cochrane Collaboration. The Cochrane Library is a searchable database of these high quality reviews. There is currently free access in only a few provinces in Canada, but efforts are underway to make it freely accessible across Canada, so keep checking if you do not have access through a library service. There are several different sections of the library:

    1. systematic reviews
    2. protocols (Someone has developed the plan for a systematic review on the topic. An expected completion date is usually given.)
    3. DARE (Database of Abstracts of Reviews of Effects) a database of other reviews that have not been done by Cochrane groups

    When you look at a review in the Cochrane Library, you first get a structured abstract, followed by the plain language statement, which is useful if you do not understand the language of research.

    Practice!

    If you do have access, go to the Cochrane Library and to the search box. Explore what you get when you type in obesity; or physical activity and school . What did you find that is relevant to our scenario?

    Several come up more specific ones with the latter search. For example, this one looks useful:

    • Summerbell, C.D., Waters, E., Edmunds, L.D., Kelly, S., Brown, T., & Campbell K.J. Interventions for preventing obesity in children. Cochrane Database of Systematic Reviews 2008 Issue 2.

    FYI: The results of this review indicate that several studies were found. The authors divided the results by short versus long-term (more than 12 months), and by the type of intervention (diet plus activity versus either alone). Two long-term studies of physical activity alone showed that children were more likely to have a lower BMI after a physical activity program, but the results were larger if diet and physical activity were combined and the effect was greater in girls than boys.

    Health-Evidence.ca

    As noted in the synopses above, for public health policy and practice questions, one of the best sources of information is from Health-Evidence.ca. Using a very thorough search, the team broadly searches every six months for any new reviews. They critically appraise the review and rate it as strong, moderate or weak, then provide a two-page summary of key methods issues, with recommendations for practice and policy.

    Keeping this page available, open a 2nd window in your browser and click on Health-Evidence.ca

    Practice!

    1. Click the top right where the magnifying glass says SEARCH articles. It leads to a refined search page, where you will see Focus. There, you can specify physical activity and nutrition (with subfields like healthy weight); Intervention Location where you can indicate school, and age fields to check. Several reviews come up; all have quality ratings and some have the summary with recommendations in a pdf document. This is a gold mine where all the panning has been done for you. If the particular review has a summary (same as the synopsis), you get recommendations for policy and practice, available in English or French.
    2. Explore how the results changes with different search box fields.

    Of interest, the same systematic review comes up as identified in The Cochrane Library search above:

    • Summerbell, C.D., Waters, E., Edmunds, L.D., Kelly, S., Brown, T., & Campbell K.J. Interventions for preventing obesity in children. Cochrane Database of Systematic Reviews 2008 Issue 2.

    However, here the review has already been critically appraised for you (pre-appraised) and rated to be strong. In addition, there is a summary of the study and implications for practice and policy that is located in an attached pdf document.

  4. Studies

    Sometimes the topic of your interest is one that has some individual studies, but no one has made the commitment to pull them together into a systematic review. At the other extreme, you may be searching for an intervention that is so novel, there have been no published studies.

    For individual studies, there are several possible databases that you could check: Medline, CINAHL and other more specialized databases such as CancerLit. This unit will focus on PubMed (public access Medline) as it is a free-access searchable database that will even give you full-text provided the journal where the paper is published has allowed that. Otherwise, you at least usually get the abstract, which will give the idea if you want to go further to retrieve the paper. PubMed allows you to search for reviews as well as individual studies.

    Open PubMed now in another browser window.

    1. Using Clinical Queries

      This PubMed search option has a powerful, tested, built-in and lengthy search strategy that allows you to find answers to your questions quickly. What follows is a practice run at using this powerful tool.

      Down the shaded left side box on the opening page above, you will see PubMed Services. There, click on the section that says Clinical Queries. From your actual open web browser or the screenshot below, you can see that there are several options, including Search for Clinical Studies by Category and Find Systematic Reviews.

      Following the hierarchy, you go to the Find Systematic Reviews option and type in school interventions to prevent obesity and get approximately ten hits all systematic reviews. You can quickly scan and find, from the titles, that one is for 0-5 years, one for preschool and another is for adolescents. The other 6 are relevant and have been published between 1993 and 2007. Interestingly, the same systematic review comes up here, as in The Cochrane Library and in Health-Evidence.ca:

      • Summerbell, C.D., Waters, E., Edmunds, L.D., Kelly, S., Brown, T., & Campbell K.J. Interventions for preventing obesity in children. Cochrane Database of Systematic Reviews 2008 Issue 2.

      If you do not find a good systematic review, return to the Clinical Queries section and, under Search for Clinical Studies by Category, click the circle on Therapy. Type in the box some terms such as physical activity and obesity prevention and children. You get approximately 40 studies. For most people making clinical and policy decisions, this number is probably not manageable!

    2. Using Special Queries

      Of note, other types of studies get (mis)classified for easier searching in PubMed. Back to the opening page, click on Special Queries. Here you will find Health Services Research (HSR) Queries, which would be of special interest to managers and policy-makers. Once again, click there and you will see there are several options, such as:

      • economics
      • outcome assessment
      • process assessment
      • qualitative research
        Note a strange place to put qualitative research as most qualitative research is not about health services research, and most people looking for answers to qualitative questions would not think to look under health services research.
    3. Using PubMed Limits

      If you have no luck with the Clinical Queries or Special Queries, another option is to use Limits in PubMed. Back to the opening page, you can type in the words to describe your topic, then click on limits just below the search box. You will see that you can limit by several factors author, year, type of article, age. Play with using the same terms in the box, but changing the limits to see what you get.

Other Resources

  • Centre for Evidence-Based Medicine. EBM tools. Finding the best evidence.
  • Forrest, J.L. (2001). Enhancing your practice through evidence-based decision-making: finding the best evidence. Journal of Evidence-Based Dental Practice, 1, 127-136.
  • The McMaster Health Knowledge Refinery. A collection of projects related to retrieval, appraisal, organization, classification, dissemination and uptake of evidence from research that is relevant to health care.
  • National Library of Medicine. PubMed Tutorial.

References:

  • Haynes, R.B. (2007). Of studies, syntheses, synopses, summaries and systems: the 5S evolution of information services for evidence-based healthcare decisions. Evidence- Based Nursing, 10, 6-7.

Part 4. Appraise. How can I decide if the particular study is good enough to apply? (1 hour)

Introduction

Unless you have found a pre-appraised article (such as a synopsis from an evidence-based journal or summary in Health-Evidence.ca), you will have to assess the methods of the study. This process is known as critical appraisal and what you are judging is the quality of the methods, trying to answer the question: Was this study done well enough that I can be confident in the findings?

There are key quality criteria for any types of studies that you find in the 5S Pyramid (Part 3 above - systems, summaries, synopses, syntheses, and studies). It is not the intent of this learning module to detail the critical appraisal process for every kind of research report, but to make you aware of the resources where you can get the tools and explanations about their application.

A word of caution! Neophytes to critical appraisal sometimes throw out relatively welldone studies from consideration because they are not perfect. There are no perfect studies. As you become more familiar with the process, you will see that there are some criteria that relate to larger concerns, and would therefore be fatal flaws for which you would reject the study. However, some other criteria are not so critical and, even if the study has not fulfilled that particular criterion, you would still consider implementing the intervention.

Most of the available tools are built from the key criteria developed by the Evidence-Based Medicine Working Group for the series published in JAMA, later collected in a book (Guyatt & Rennie, 2002). The particular tools below are no exception, but are recommended because they are more self-explanatory than most. With each of these recommended tools, you get an explanation of the criterion they are asking you to rate handy to have for neophyte critical appraisers.

Once again, we are using our scenario, about preventing obesity in children with school-based physical activity programs. Use the scenario for every section on appraising below.

  1. Appraising a guideline.

    You know about the 5S Pyramid, so you started at the top and searched for a guideline, which you found at the National Guideline Clearinghouse. How do you decide if the guideline was well done?

    An internationally accepted standard for critiquing guidelines is the AGREE tool, from the AGREE (Appraisal of Guidelines Research and Evaluation) collaboration. The tool is at AGREE collaboration. While it looks intimidating simply due to its number of pages, its length is due to the explanations given for every criterion and for the scoring. It is very self-explanatory, and depending on the guideline, can usually be completed in well under an hour.

    The tool is set up to be scored by individuals, but can also be used in a consensus process, where each rating is discussed. As an individual using the tool, you get a sense of where there are problems that my affect your use of the guideline; you can decide if there is a fatal flaw that would lead you to reject the guideline, or if there are minor flaws, such that you can still get a very good protocol from a guideline. For example, one of the criteria is related to whether the guideline group has set a time for the next review and update of the guideline. Even if "no", this would not be a fatal flaw. Conversely, the lack of thorough search for relevant literature on which to base the guideline would be a reason to reject the guideline.

    Practice!

    Practice rating a guideline:

    • Print the AGREE tool
    • Go back to the Systems part of section 3 and search for a relevant guideline.
    • Try out the AGREE tool to rate the guideline you found in b) above.
  2. Appraising a review (synthesis)

    The Critical Appraisal Skills Program (CASP) of the Public Health Resources Unit in the UK has a series of tools with explanations of criteria that are freely accessible on-line for personal use. As with a guideline, you want to know if you can really rely on the systematic review that you found regarding physical activity in the schools to prevent obesity in children. The criteria ask you about factors such as the clarity of the review question, if the search was thorough, the size of the effect and how precise it was, and if the interventions described in the review could be applied to your population or patients.

    Practice!

    Practice rating a systematic review:

    1. Print the tool to appraise systematic reviews: Public Health Resource Unit
    2. Go back to the synthesis part of section 3 and search for a relevant systematic review or meta-analysis
    3. Try out the CASP review tool to rate the systematic review you found in b) above.
  3. Appraising a primary study about an intervention

    The Critical Appraisal Skills Program (CASP) of the Public Health Resources Unit in the UK has a series of tools with explanations of criteria, that are freely accessible on-line for personal use. As with a systematic review, you want to know if you can really rely on the individual studies you find about physical activity. The criteria ask you about factors such as the size of the effect, the precision of the results, and if the interventions described in the review could be applied to your population or patients. Sometimes in health care research we have information about interventions through the highest level of evidence, the randomized controlled trial; for other types of health care studies, we only have evidence at the level of cohort studies or case-controls. Therefore, there are different review criteria for different types of studies. Randomized trials have the greatest ability to control for confounders or bias.

    For practice rating a primary study:

    1. Go back to the single studies part of section 3 and search for a relevant intervention study, preferably a randomized trial.
    2. Print the tool to appraise randomized trials
    3. Try out the CASP tool to rate the randomized trial you found in a) above.

Critical Appraisal Skills Programme (CASP) (UK)

  • Form to critically appraise systematic reviews
  • Form to critically appraise randomized trials
  • Form to critically appraise cohort studies
  • Form to critically appraise case control studies
  • Form to critically appraise economic evaluation studies
  • Form to critically appraise qualitative studies

Other Resources:

  • Shea B. J., Grimshaw J. M. , Wells G. A., Boers M., Anderson N., Hamel C., Porter A. C., Tugwell P., Moher D., & Bouter L.M. (2007). Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology, 1-7. This article is available from BMC Medical Research Methodology
  • Duke University Medical Center. Introduction to Evidence-Based Medicine. Evaluating the Evidence

References:

  • GRADE Working Group (2004). Grading quality of evidence and stregth of recommendations. British Medical Journal, 328,1490-7.
  • Guyatt, G. & Rennie, D. (Eds) (2002). Users Guides to the Medical Literature: A manual for Evidence-Based Clinical Practice. American Medical Association.

Part 5. Integrate. How do I decide which of multiple studies to use? (0.5 hours)

You do a focused search on the top levels of the 5S Pyramid and find

  • 1 guideline
  • 5 systematic reviews
  • 26 individual studies

about the effectiveness of daily physical activity on obesity in children.

Should you read all this information?

Should you critically appraise it all?

What is the most efficient way to get through this information?

Have a look at Figure 5.1 to get a sense of how you would decide about use of the varying levels of evidence you have found. Using the 5S Pyramid from Section 2, you would start with the retrieval and reading of the guideline; if it is relevant, you would go on to apply the AGREE tool (Section 4) to appraise the guideline. If you and your colleagues conclude that this is a relevant guideline of sufficient quality that you can utilize it, and it is published in the past two years, that may be the extent of your work. However, if you conclude it is not a good review, you need to go to the systematic reviews.

One way to approach multiple systematic reviews of the same topic is to look first at those with the most recent date of searching. The date of publication may not be relevant, but in the methods under search strategy, it will detail the years included in the search. The critical appraisal can then be ordered chronologically, starting with the reviews with the most recent searches. That will give you some confidence in how up-to-date is the publication.

If you find that the systematic reviews are recent and of high quality, the only reason to search for additional primary studies would be to take into consideration those published after the search dates in the systematic reviews. Were their results similar to the results reported in the systematic review or have the more recent studies presented different conclusions?

Finally, in order to get the details of more complex interventions (almost anything beyond drug therapy), you would have to look up the primary studies that were included in the guideline development or systematic review, as these syntheses cannot include enough detail in their publication to allow the clinician to implement the intervention.

What if you find no research evidence at any level of the 5S Pyramid? Some health care questions have not yet been answered by research. Expert opinion is the next best option. Going back to the scenario, your specific question might be about school-based daily physical activity of three hours or more versus one hour or less. That question has not been addressed by research and may never be! If you are really intent on this question, you might get an expert to estimate the calorie expenditure and they might guess which would utilize more energy.

Otherwise, if there are no published studies, you might be seeking more general information from experts of colleagues, like:

  • What have people tried?
  • How did it seem to work?
  • Were patients/populations open to trying the intervention?
  • Were patients/populations satisfied with the results?

Part 6. Adapt. How do I use the information from #5 in decisionmaking/policy brief? (1 hour)

Research evidence comes from studies of particular samples of populations. You will have noticed when looking at the critical appraisal tools that all include criteria related to can I use this research with my patients (or population)? In every instance, you must consider the inclusion/exclusion of participants in the study.

Some people take this to the extreme and claim they can only use studies done in their own institution. A better question might be to ask if there are important reasons why you cannot apply this evidence to your situation.

For example, from our scenario, we may not be interested in studies of physical activities done in boarding schools where the time for activity may have been done outside of the regular curriculum. You know this cannot work without the captive audience as the students at your school would mostly not be available, particularly if busing is used at your school.

In addition, you must consider how this research evidence fits with outer factors. Consider:

  • Magnitude of health issue in local setting
  • Magnitude of the potential reach of the intervention
  • Costs (direct and indirect) of implementing the intervention
  • Availability of resources (personnel)
  • Organizational expertise and capacity
  • Political acceptability of the intervention
  • Social acceptability of the proposed intervention

A tool for assessing applicability in community interventions is available:

Assessment of Applicability & Transferability
Construct Factors Questions to Ask
Applicability (feasibility) Political acceptability or leverage
  • Will the intervention be allowed or supported in current political climate?
  • Will there be public relations benefit for local government?
  • Will this program enhance the stature of the organization?
  • Will the public and target groups accept and support the intervention in its current format?
Social acceptability
  • Will the target population be interested in the intervention? Is it ethical?
Available essential resources(personnel and financial)
  • Who/what is available/essential for the local implementation?
  • Are they adequately trained? If not, is training available and affordable?
  • What is needed to tailor the intervention locally?
  • What are the full costs (supplies, systems, space requirements for staff, training, technology/administrative supports) per unit of expected outcome?
  • Are the incremental health benefits worth the costs of the intervention?
Organizational expertise and capacity
  • Is the current strategic plan/operational plan in alignment with the intervention to be offered?
  • Does this intervention fit with its mission and local priorities?
  • Does it conform to existing legislation or regulations (either local or provincial?) Does it overlap with existing programs or is it symbiotic?)
  • Any organizational barriers/structural issues or approval processes to be addressed?
  • Is the organization motivated (learning organization)?
Transferability (generalizability) Magnitude of health issue in local setting
  • Does the need exist?
  • What is the baseline prevalence of the health issue locally?
  • What is the difference in prevalence of the health issue (risk status) between study and local settings?
Magnitude of the "reach" and cost effectiveness of the intervention
  • Will the intervention broadly cover the target population?
Target population characteristics
  • Are they comparable to the study population?
  • Will any difference in characteristics (ethnicity, socio-demographic variables, number of persons affected) impact intervention effectiveness locally?

Practice!

Once again, go back to the scenario. At this point, you are satisfied that the research that you have found supports the inclusion of more active daily physical activity periods for school children.

You are now developing that recommendation for a policy that will be presented at the local school board. Using the community where you actually live, consider the factors in the Applicability and Transferability Tool above. Where will the strengths/supports be and where will the barriers/weaknesses be?

Develop an argument for why the evidence can or cannot be applied with your own community.

References:

  • Buffett, C., Ciliska, D., & Thomas, H. (2007). Can I use this evidence in my program decision? Assessing the applicability and transferability of evidence. National Collaborating Centre for Methods and Tools.
  • Fervers B, Burgers JS, Haugh M et al (2006). Adaptation of clinical guidelines: A review of methods and experiences. International Journal of Health Care, 18, 167-76.
  • Grimshaw J, Thomas RE, MacLennan G et al (2004). Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technology Assessment, 8:iii-iv- 72.
  • Toman C, Harrison MB, & Logan J (2001). Clinical Practice Guidelines: Necessary but not Sufficient for Evidence-based Patient Education and Counseling. Patient Education and Counseling, 42(3), 279-87.

Part 7. Apply. How do I develop the implementation plan? (0.5 hours)

Back to our scenario, you have now framed the question, searched for, found and appraised the research evidence as good enough to use and you have concluded that the intervention of daily physical activity in the schools is feasible (can be transferred) in your community.

Going back to Figure 1 in Part 1, we are now up to the left outer circle section about Assessing barriers to knowledge use. Some of the barriers may have been identified as you considered the applicability/transferability in Part 6, or in your consideration of general barriers to Evidence-Informed Decision Making in Part 1.

Practice

Make a list of the barriers you think would be active in instituting daily physical activity in the schools (our scenario). Barriers relate to:

  • The individuals who will use this information (e.g., What barriers will teachers have? Students?)
  • The organizations (e.g., School boards? Curriculum?)
  • The broader culture (e.g., Prevailing attitudes)
  • The change itself (e.g., How much time will it take away from the usual curriculum? How complex is the task? Will any school personnel require more training?)

Now use the same list and consider the supports that would be available in each of the categories.

Supports relate to:

  • The individuals who will use this information (e.g., Teachers? Students?)
  • The organizations (e.g., Parents? Board of Health?)
  • The broader culture (e.g., Attitudes toward fitness)
  • The change itself (e.g., A welcome break from in-class activities?)

Together, this assessment of barriers and supports is sometimes called a situational analysis. The analysis will help you plan for strategies to implement change. As with any planned change, it is important to consider both how to build support as well as how to break down barriers. Go back to Figure 1 in Part 1. We are now up to the left outer circle section about Select, tailor and implement interventions.

John Lavis and colleagues (2003, 2004) proposed five key questions to assist in planning for knowledge exchange. They are key questions to answer whether you are a researcher, manager, practitioner or policy-developer.

  • What (is the message)? This translates or transforms research findings into an actionable message.
  • To Whom (the audience)? Be specific when defining who will need to get the message. Understand who are making the decisions.
  • By Whom (the messenger)? Is the messenger credible and is there a chance for the audience to partner with the messengers?
  • How (transfer method)? What is the budget, preferred mechanism for learning new information, is the audience actively engaged in selected the mechanism? Is the transfer mechanism evidence-based? Is the intervention tailored to overcome the audiences identified barriers?
  • With what expected impact (evaluation)? What does the knowledge translation , synthesis and exchange (KTSE) project hope to change?

Practice

Back to our scenario, we have found good evidence related to daily physical activity for children done in the schools. Answering the following questions will go a long way to helping you develop an action plan.

  • What (is the message)?
  • To Whom (the audience)?
  • By Whom (the messenger)?
  • How (transfer method)?

Other Resources

  • DiCenso, A., Virani, T., Bajnok, I., Borycki, E., Davies, B., Graham, I. et al. (2002). A toolkit to facilitate the implementation of clinical practice guidelines in healthcare settings. Hospital Quarterly, 5(3), 55-60.
  • Registered Nurses Association of Ontario. Toolkit: Implementation of Clinical Practice Guidelines.
  • Effective Practice and Organization of Care. This is a collaborative review group of the Cochrane Collaboration, focusing on change in health care practitioner practice and how health care is organized.

References:

  • Bosch M, Van der Weijden T, Wensing M, & Grol R (2007). Tailoring quality improvement interventions to identified barriers: a multiple case analysis. Journal of Evaluation in Clinical Practice, 13, 161-168.
  • Dijkstra R, Wensing M, Thomas R, Akkermans R, Braspenning J, Grimshaw J, & Grol R (2006). The relationship between organisational characteristics and the effects of clinical guidelines on medical performance in hospitals, a meta-analysis. BMC Health Services Research, 6, 53.
  • Grimshaw, J., Eccles, M., Thomas, R., MacLennan, G., Ramsay, C., Fraser, C., & Vale, L., (2006). Toward evidence-based quality improvement. Evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966-1998. Journal of General Internal Medicine, 21, S2, S14-20.
  • Grol R, & Grimshaw J (2003). From best evidence to best practice: effective implementation of change in patients care. Lancet, 362, 1225-30
  • Lavis, J. N., Posada, F. B., Haines, A., & Osei, E. (2004). Use of research to inform public policymaking. Lancet, 364, 1615-1621.
  • Lavis, J.N., Robertson, D., Woodside, J.M., McLeod, CB, Abelson,J. & the Knowledge Transfer Study Group (2003). How can research organizations more effectively transfer research knowledge to decision-makers? The Milbank Quarterly, 81(2), 221-248.
  • Wensing M, Van der Weijden T, & Grol R (1998). Implementing Guidelines and Innovations in Primary Care: Which Interventions are Effective? British Journal of General Practice, 48, 991-997.

Part 8. How do I evaluate? (0.5 hours)

For those who like logical progressions, here is a nice sequence:

Introduce policy → change practitioner behavior → impact population outcome

Sometimes it even works! A hospital EBP group reviewed the literature on skin preparation for venipuncture and concluded the literature favored chlorhexidine. They created a hospital wide policy, got their ordering/stores department to replace the alcohol swabs with chlorhexidine on the venipuncture preparation trays, had some strategies to communicate these changes to staff, and behold, the staff switched to using chlorhexidine for skin prep. A lovely example of the linear sequence above!

But in the scenario, some school boards have had a policy of quality daily fitness for some years already. Do the schools always follow it? No they counter with curriculum overload and lack of time to fit in the fitness schedule.

Where the school principals insist on fitting daily activity in the schedule, there are students who take every opportunity to miss the event or participate with little enthusiasm and less energy expenditure, with equal impact on their body composition! To the opposite effect, you can see that one keen teacher, despite a lack of support by the principal, ensures that her students get fifteen minutes of high level activity each day, even when she is supposedly teaching math. Practitioners can, therefore, introduce changes in their own practice without a policy in place. Likewise, some students will take it upon themselves to get more than the suggested share of high level activity during recess breaks, so will get the fitness benefit without policy or teacher intervention!

If the practitioners follow the policy, is there an impact on the population? In the case of this scenario, can you prevent obesity? Five years later, is the rate of obesity at your school lower than before the policy was introduced? Does the obesity level of the children change?

Every step in this sequence is a fruitful area for program evaluation or a more formal funded research project. Studying how policy development and implementation take place, including factors that affect each stage, would increase our understanding about the policy process. We do not know very much about uptake of policy directives or how they get adapted at the local level. From our scenario, how does the policy about physical activity get carried out by the schools and how much does the principal alter the policy through interpretation, or how carefully do they monitor if the teachers are following the policy, or how many shortcuts do teachers take, because they feel like they are getting behind in the curriculum?

There are some tools to evaluate research utilization in general. However, you can see that, with the scenario, you would be wise to evaluate more specific processes and outcomes, such as how many teachers did the daily activity, what percentage of days, estimated costs of training teachers, students grades, student fitness activities and rate of obesity in students.

Practice!

How would you assess the outcome of your planned change (Part 7)?

What outcomes could you assess in relation to:

  • Policy development
  • Policy uptake
  • Teacher use of the policy
  • Student participation
  • Student outcomes

Resolution of the scenario

You now have a beginning understanding of evidence-informed decision making and are very prepared for your participation in the group. You have framed a clear question about increasing physical activity in the schools, and have considered what the barriers and supports might be both for policy development and implementation. You also have some ideas about outcomes to consider for evaluation in terms of reach, penetration, fidelity to the intervention and outcomes for students. You are a valuable and well-informed group member.

References:

  • Eccles M, Grimshaw J, Campbell M, & Ramsay C (2003). Research designs for studies evaluating the effectiveness of change and improvement strategies. Quality and Safety in Health Care, 12(1):47-52.
  • Grimshaw JM, & Eccles MP (2004). Is evidence-based implementation of evidence-based care possible? Medical Journal of Australia, 180(6 Suppl), S50-S51.
  • Grol R, & Grimshaw J (2003). From best evidence to best practice: effective implementation of change in patients' care. The Lancet, 362(9391), 1225-30

Glossary

Bias:
a systematic error or departure from the truth in results.
Cohort study
a group of people with a common set of characteristics or set of characteristics that are followed up for a period of time to determine the incidence of an outcome; there is no comparison group.

Cullum, N., Ciliska, D., Haynes, R.B., & Marks, S. (2008). Evidence-Based Nursing. An Introduction. Oxford: Blackwell

Case-control study
an observational study that begins by comparing patients who have the health problem (cases) and control participants who do not have the health problem, and then looking back in time to identify the existence of possible causal factors. For example, the identification of patients with and without lung cancer and looking back in time to determine past smoking behavior (exposure to tobacco).

Dawson-Saunders, B., Trapp, R.G. (1994). Basic and Clinical Biostatistics. Norwalk: Appleton & Lange

Case report
detailed report of a person detailing signs, symptoms, progression of illness
Case series
a report on a series of patients with an outcome of interest. There is no comparison group.
Confounder
a variable that affects the observed relationship between two other variables. For example, alcohol consumption is related to lung cancer but does not cause the disease; instead, both alcohol and lung cancer are related to smoking (the confounder), which causes lung cancer.

Crombie, I.K. (1996). The pocket guide to critical appraisal: A handbook for Healthcare Professionals. London: BMJ Publishing Group.

Evidence
information or facts that are systematically obtained (i.e., obtained in a manner that is replicable, observable, credible, verifiable, or basically supportable).

Rycroft-Malone, J., & Stetler, C.B. (2004). Commentary on evidence, research, knowledge: a call for conceptual clarity. Worldviews on Evidence-Based Nursing; 1(2):98-101.

Evidence-based medicine
the conscientious, explicit and judicious use of the current best evidence in making decisions about the care of individual patients.

Sackett. D.L., Rosenberg, W.M., Muir-Gray, J.A., Haynes, R.B., Richardson, W.S. (1996). Evidencebased medicine: what it is and what it isnt. BMJ, 328, 535-536

Evidence-informed decision-making
the use of evidence that contributes to decision making about particular problems or issues about best use of resources within institutions and across the healthcare system.

Canadian Health Services Research Foundation (2006). Weighing Up the Evidence. Making evidence-informed guidance accurate, achievable, and acceptable. A summary of the workshop held on September 29, 2005.

Dissemination
involves identifying the appropriate audience and tailoring the message and medium to the audience. Dissemination activities can include such things as summaries/briefings to stakeholders, educational sessions with patients, practitioners and/or policy makers, engaging knowledge users in developing and executing dissemination/implementation plan, tools creation, and media engagement.

Canadian Institutes for Health Research About Knolwedge Translation.

Knowledge translation
is a dynamic and iterative process that includes synthesis, dissemination, exchange and ethically-sound application of knowledge to improve the health of Canadians, provide more effective health services and products and strengthen the health care system. This process takes place within a complex system of interactions between researches and knowledge users which may vary in intensity, complexity and level of engagement depending on the nature of the research and the findings as well as the needs of the particular knowledge user.

Canadian Institutes for Health Research About Knolwedge Translation.

Knowledge exchange (formerly knowledge transfer)
is collaborative problem-solving between researchers and decision makers that happens through linkage and exchange. Effective knowledge exchange involves interaction between decision makers and researchers and results in mutual learning through the process of planning, producing, disseminating, and applying existing or new research in decision-making.

Canadian Health Services Research Foundation. Glossary of Knowledge Exchange Terms Used by the Foundation.

Meta-analysis
a method for combining the results of several independent studies that measure the same outcomes so that an overall summary statistic can be calculated.

Dawson-Saunders, B., Trapp, R.G. (1994). Basic and Clinical Biostatistics. Norwalk: Appleton & Lange.

Phenomenology
an approach to enquiry that emphasizes the complexity of human experience and the need to understand that experience holistically, as it is actually lived.

Polit, D.F., & Hungler, B.P. (1997). Essentials of Nursing Research: Methods, Appraisal and Utilization. Philadelphia: Lippincott.

Qualitative research
research that aims to generate an understanding of complex, unquantifiable phenomena, such as peoples experiences or perceptions. A few examples of types of qualitative research include phenomenology, grounded theory, participatory action research, and ethnography.

Cullum, N., Ciliska, D., Haynes, R.B., & Marks, S. (2008). Evidence-Based Nursing. An Introduction. Oxford: Blackwell.

Randomized controlled trial (RCT)
a study design in which individuals are randomly allocated to receive alternative preventive, therapeutic or diagnostic interventions and then followed up to determine the effect of the interventions (one of the alternatives might be no intervention).

Cullum, N., Ciliska, D., Haynes, R.B., & Marks, S. (2008). Evidence-Based Nursing. An Introduction. Oxford: Blackwell.

Synthesis
the contextualization and integration of research findings of individual research studies within the larger body of knowledge on the topic. A synthesis must be reproducible and transparent in its methods, using quantitative and/or qualitative methods. It could take the form of a systematic review, follow the methods developed by the Cochrane Collaboration, result from a consensus conference or expert panel and may synthesize qualitative or quantitative results. Realist syntheses, narrative syntheses, meta-analyses, meta-syntheses and practice guidelines are all forms of synthesis.

Canadian Institutes for Health Research. About Knolwedge Translation.

Cullum, N., Ciliska, D., Haynes, R.B., & Marks, S. (2008). Evidence-Based Nursing. An Introduction. Oxford: Blackwell

Date modified: