Background Methodology
This is a brief summary of methods of the Global Health Interventions: A Review of Evidence, a project of the University of California San Francisco and the Kaiser Family Foundation. Global Health Interventions: A Review of Evidence results and additional supporting materials are posted on the KFF Global Health Gateway, http://globalhealth.kff.org/. For further information, contact James G. Kahn.
The primary goal of Global Health Interventions: A Review of Evidence is to translate empirical evidence on the effects of interventions into a quantitative synthesis that is technically accurate while being concise and accessible to non-technical audiences. Global Health Interventions: A Review of Evidence examines both prevention and treatment for each health condition, with a particular focus on data with the most potential relevance for policy and an emphasis on health outcomes (e.g., deaths and disease incidence) over process measures (e.g., satisfaction with services or adherence).
Global Health Interventions: A Review of Evidence findings are presented in three parts:
-
Key Findings Table: for each prevention and treatment intervention type, indicates the risk reduction and strength of evidence for mortality, morbidity, and other indicators.
-
Overview: provides background information on the health condition; summary epidemiology; review of Key Findings (What works?), and future directions.
-
Logic Model: a graphic of the mode of disease acquisition and progression, indicates the location of opportunities for prevention and treatment interventions.
Global Health Interventions: A Review of Evidence methods employ three key techniques:
-
Stepwise, transparent progression: GHIR moves with explicit and small steps from existing systematic reviews and important new studies to “key findings.” (See pyramid below).
-
Relative risk reduction (RRR): We use a standardized outcome metric, RRR, or the percent reduction in the risk of negative health outcomes, to describe the impact of each intervention on outcomes. The RRR is applied to mortality, morbidity, and/or indirect health indicators, according to available evidence.
-
Strength of Evidence (SoE): We rate SoE on a 0 to 6 scale (visually represented by bars as in mobile phone reception). The score is based on the extent and type of studies (e.g., randomized controlled trials, or RCTs), the quality of available systematic reviews, and precision (i.e., narrowness of uncertainty bounds). The typical evidence associated with each score is summarized below. Detailed SoE methods are available.
6 = 3 or RCTs, well reviewed, good precision (very strong).
5 = 3 or more RCTs, minor problems with review and/or precision (strong).
4 = 2 RCTs, good review and precision (moderate strength).
3 = 1 RCT or multiple non-RCTs, good review and precision (moderate strength).
2 = 1 RCT or multiple non-RCTs, problems with review and/or precision (weak).
1 = 1 or more non-RCTs, serious problems with review and/or precision (very weak).
0 = No evidence, due to lack of studies or extreme imprecision.
This pyramid summarizes the Global Health Interventions: A Review of Evidence process to distill evidence on “What works?” for global health interventions.
Global Health Interventions: A Review of Evidence starts at the bottom of the pyramid, with a search for systematic reviews and pivotal new studies. Most reviews come from the Cochrane Library, with others identified through PubMed and other sources. The evidence at this level is massive, diverse in form, and technically complex.
The second step is to select potentially policy-relevant reviews and comparisons within those reviews. By comparison, we mean comparing an intervention to a referent prevention or treatment strategy. Our criterion is: could this information affect major decisions on policy or funding, such as whether and at what scale to support a particular intervention? Important but narrower questions, such as drug dosing or comparisons between very similar intervention designs, are generally excluded. We also exclude universally accepted practices, such as basic oral rehydration salts. Thus, we omit one-third to more than two-thirds of review comparisons through this process. We document all these discussions and decisions (available on request)..
We next extract information from the selected comparisons. We extract the context (e.g., country and type of population), the research methods (e.g., study design and outcome measures), and the quantitative findings on efficacy (to allow calculation of the RRR).
In the following step we rate the strength of evidence, As noted above, this relies on the quantity and type of studies, as well as the precision of findings (i.e., the width of the relative risk confidence interval). The result is a “Summary” table that presents the intervention comparisons, the findings (e.g., mortality), the RRR, and the Strength of Evidence.
Finally, we combine evidence by intervention type where possible. For example, if different insecticides for environmental control of a disease vector (i.e., a mosquito) all work with similar efficacy (as measured with the RRR), we combine the findings into a single row. If one class of insecticides is more effective and more expensive (and thus may be the subject of policy discussion), it gets a separate row.
The first step in combining is the “Intervention Map”, which lists in each row the individual comparisons to be combined. The second step is the “Key Findings” table, which presents only ranges of findings. We document the links between the original extractions and each row in these two summary tables, to allow review of the aggregation decisions.
Before finalizing our results, we consult with subject area experts. We send to these experts our list of included reviews, the Key Findings table, and all of the other evidence synthesis steps described above. We follow up via teleconference. Based on expert feedback, we may add reviews or pivotal new studies, or adjust our interpretation of existing reviews.