Application of meta-analysis to specific research fields: Lessons learned

Abstract

Scientific research is challenged to translate findings from multiple, often conflicting, clinical trials into a simple answer of whether a treatment works or not. The public and healthcare providers alike frequently voice their frustrations when the media reports a treatment working on one day, but seemingly the next day reports a study refuting the previous one. Meta-analyses are being used more commonly by researchers to convey an understandable summary of scientific studies to the general public and healthcare providers. As time goes by, we have learned how to improve meta-analytic techniques to reflect more valid results and when it is appropriate to pool or not to pool results from different studies. Retrospective reviews often don’t acknowledge this learning curve and may fail to recommend the most current valid guidelines. This editorial presents an example of how the current use of meta-analysis has shifted in one field (the therapeutic effects of probiotics) and recommendations on how to correctly interpret the results of such an analysis.

Key Words: Meta-analysis; Study designs; Probiotics; Sensitivity analysis; Meta-regression

Core tip: As meta-analyses are used more frequently and their findings reach a wider scope of people, it is the responsibility of researchers to use current guidelines and appropriately apply their findings to form valid conclusions. As researchers gain experience with this technique, we need to recognize that our methods may change over time. Meta-analysis remains a valuable tool for examining controversies arising from conflicting studies.

INTRODUCTION

The lag-time between a scientific discovery and when it is translated into changes in public behavior and beliefs can be protracted. We have never been so challenged in our ability to sort through the onrush of volumes of scientific data to provide timely clinical advice to the public and healthcare providers. The public and healthcare providers face the challenge of obtaining accurate clinical recommendations when the amount of information available is vast and often discordant. In addition, as meta-analyses are used more often, issues in proper application of methods and interpretation of the results need to be re-visited, for example, when it is appropriate to pool studies.

Public awareness and belief in scientific study results

We researchers work hard to present new findings for clinical therapies, but how much does the general population really understand Patients are often confused by information overload and the ambiguity of clinical results, while at the same time suffer from low health literacy[1]. Health literacy measures the ability of the public to understand healthcare providers, as well as reading and comprehending the vast amount of health information. What is alarming is that only 25%-50% of the American and European general populations are scientifically literate, making it difficult for them to decipher complex scientific articles on new medical treatments[2,3]. A global survey across 57 nations found 56% have “hardly any or only some confidence” in the scientific community[4]. Improving public confidence and awareness of scientific findings is paramount. Science is not a static world. Shifting through studies analyzing similar treatments for the same disease can be frustrating, even for scientists, as newer studies may refute established beliefs and the findings of other studies that seem similar in study design and enrolled participant populations.

Systematic reviews were first developed to provide the public and healthcare providers with a simple summary from multiple studies. The value of systematic reviews is that they allow a detailed examination of many studies, in which researchers try to explain the heterogeneity of the study outcomes, which may to be to a variety of sources ranging from the type of study population, different study designs, differences in study quality and differences in dose and duration of the investigational intervention. But to the disappointment of readers and clinicians, systematic reviews often do not provide a simple answer to the question: “Does this treatment work”

Meta-analysis was developed to allow combining results from different studies in order to obtain a single pooled estimate of effect or risk from various exposures (behaviors, medications, or interventions) associated with a specific outcome. Meta-analytic analysis involves discerning where heterogeneity exists, identifying the sources from which it may spring and modeling the data to obtain a single pooled estimate of effect. The advantages of meta-analysis include: achieving a higher statistical power due to the greater numbers involved in the pooled population, ability to examine sources of variability over multiple studies while adjusting for common measures shared by studies, and providing one pooled outcome across a variety of studies.

If meta-analysis can provide an answer to “Does this treatment work”, why is there still confusion in the literature The problem relates to limitations inherent in the meta-analytic methods and the often too rapid acceptance of conclusions reached by the researchers. If a meta-analysis has pooled dissimilar types of treatments together and the pooled risk estimate shows significant efficacy, people often jump to the conclusion that any of the treatments included in the meta-analysis must be effective. This isn’t necessarily so.

Meta-analyses can be misleading if different types of treatments or different types of outcomes are grouped together. The pooled outcomes are also highly dependent upon how the search of the literature is conducted: including or excluding meeting abstracts, non-English trials, years covered, inclusion/exclusion criteria narrow or wide, how incomplete data is handled (exclude or attempt to contact authors of original papers) and different forms of bias (publication bias, study quality, study size). There is a distinct learning curve as both the field of meta-analysis matures and researchers using this technique gain experience.

LEARNING CURVE

History

Meta-analysis developed in the field of astronomy in the 1700s, with the application to the clinical field in 1907[5]. The use of meta-analysis was largely ignored by the clinical field until the 1980s, when the number of publications increased exponentially (Figure 1). Expert panels were convened to reach a consensus on how meta-analysis should be conducted and reported. The guidelines published as PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analysis, were developed in 1999 and are continually updated, yet the translation of meta-analytic results continues to challenge the general public and scientists alike[6-8].

Figure 1 Number of publications from PubMed (January 1960-October 2014) citing Meta-analysis by decade. A total of 83786 citations.

Meta-analysis is used to assess a wide array of interventions and diseases ranging from the phases of the moon and lunacy[9] to the effects of wine[10,11] and chocolate on cardiovascular disease[12] and the effectiveness of probiotics for many diseases [for example, antibiotic associated diarrhea (AAD), allergies, acute pediatric diarrhea, etc.][13-15].

To demonstrate how we have learned to properly perform and interpret meta-analytic results, let’s focus on the field of probiotics for the prevention and treatment of various diseases.

Examples of inappropriate pooling

In the early phase of scientific enquiry for a new treatment, there is a tendency to combine different types of interventions or different outcomes to generate a pooled estimate of risk, especially when there is a paucity of available clinical trials published in the literature. This was seen in the field of probiotic randomized clinical trials (RCT).

Some of the earliest meta-analyses of probiotics combined different probiotic strains and species, and reported one pooled estimate of effect. An early meta-analysis done in 2002 assessing probiotics for the prevention of AAD reported a significant protective effect (pooled OR = 0.39, 95%CI: 0.25-0.62) for four RCT of “yeast” probiotics[16]. Fortunately, the only available trials used the same type of yeast (Saccharomyces boulardii). However in this same meta-analysis, D’Souza et al[16] also concluded that all “non-yeast” probiotics were effective for AAD, based on a pooled OR from five RCT, which combined the results from four different types of probiotics (Enterococcus faecium, Lactobacillus rhamnosus GG and three trials with two different mixtures of probiotics). Subsequent meta-analyses did not support the efficacy for three of the types of probiotics and only L. rhamnosus GG was found to be effective for AAD in later studies. Another example is from a meta-analysis by Van Niel et al[17], who reported a single pooled relative risk (RR) estimate for three different strains and a mixture of two strains of Lactobacilli for the treatment of pediatric diarrhea, and concluded that “Lactobacilli was effective for treating pediatric diarrhea”. Again, when these different strains were subsequently analyzed separately, some were effective, but some were not. This meta-analysis did not analyze the different species separately. Another meta-analysis by Deshpande et al[18] done in 2010 including 10 different types, either a single strain or mixtures, of probiotics concluded from their pooled RR that a “combination of Lactobacillus and at least one Bifidobacterium species” was effective for pediatric necrotizing enterocolitis. Unfortunately, there only three of the ten trials used a mixture of Lactobacilli and Bifidobacteria species, but none of the mixtures contained the same strains of bacteria. Even recent publications suffer from inappropriately concluding “any of the tested probiotic strains are effective” once as a significant pooled OR is found. King et al[19] pooled the results from 20 RCT using a variety of probiotics (eight RCT using five different Lactobacilli species and 12 trials with different mixtures of probiotic types) and concluded “probiotics reduce the duration of respiratory illness”. They did observe significant heterogeneity in the analysis and, although they did sub-group analysis by degree of bias and type of patient population (adult vs pediatric), they failed to do a separate analysis by type of probiotic strain. Khalesi et al[20] did a meta-analysis with nine RCT using a variety of different probiotic formulations (four RCT used yogurts, two RCT used milk, one each used capsules, drink or cheese) with nine different probiotic species and concluded “consuming probiotics may improve blood pressure”. Although the researchers did do separate analysis by single vs mixtures of probiotics, dose and duration of probiotic treatment and baseline blood pressure, they did not do a separate analysis by the type of probiotic strain. Perhaps the best example of inappropriate use of meta-analysis is seen in the review done by Hempel et al[21] published in JAMA in 2012. The authors concluded that “the pooled RR from 63 RCT indicated a statistically significant association of probiotic administration with reduction in AAD”. However, eight studies were not on AAD, the pooled studies were a mix of treatment and prevention study designs, and the pooled probiotics included 13 different mixtures and 8 different species of single probiotics. The erroneous assumption that once a pooled estimate shows significant efficacy means that any of the treatments included in the analysis are independently effective continues to be a major source of misinterpretation using meta-analysis techniques.

Solutions for "to pool, or not to pool"

We have learned over time that the efficacy of probiotics is species and strain specific. Thus, it became apparent that we need to separately analyze different probiotics and not lump them all together. It is now recommended to not pool studies using different probiotic strains and to either exclude non-identical interventions or use separate sub-group analyses for each group with significant heterogeneity or if the disease or intervention is known a priori to have different effects[22,23]. Although the use of random-effects models may help to statistically control for some of influences of heterogeneity due to study size or bias, other sources of heterogeneity need to be examined too.

Limit to one type of intervention: One of the earliest probiotic meta-analysis limited inclusion to six RCT using the same mixture of probiotics (Enterococcus faecium and Strept. thermophilus) and found a significant reduction in cholesterol in their pooled outcome measure[24]. As research into probiotics continues, more studies are focusing on including the same type of probiotic strain, allowing them to pool only those probiotics similar in genetic make-up, mechanism-of-action and strain efficacy.

Sub-group analysis: Another strategy to examine heterogeneity is sub-group analysis. These sub-groups should be defined a priori in the protocol and several methods may be used, including exclusion sensitivity analysis, stratifying on sub-groups, or use of meta-regression[25,26]. As probiotic meta-analysis research progresses, researchers are separating the analysis by distinct subgroups, including species or strain of probiotic, dose and duration of intervention given, type of enrolled population (adult vs pediatric) or by study quality. Szajewska et al[27] did a meta-analysis of six RCT for the prevention of pediatric AAD, which showed an overall pooled protective RR (RR = 0.44, 95%CI: 0.25-0.77) and used sub-group analysis to show only one strain (L. rhamnosus GG) was independently effective. None of the other three types of probiotics had more than one RCT. The scarcity of published articles can limit the applicability of meta-analyses when sub-group analysis by probiotic species results in only one RCT per species. In these cases, we must wait until multiple trials are available before we can use meta-analysis to determine if a particular probiotic strain is effective or not.

Currently, most researchers are now incorporating current guidelines and performing sub-group analysis separately for each type of probiotic and reporting these pooled estimates separately[13,15]. The meta-analysis of Johnston et al[28] illustrates what happens when meta-analysis is used when the literature can provide only a limited number of trials for the disease being studied. He included six RCTs, but only two trials used the same type of probiotic (L. rhamnosus GG) and although he presented pooled RR for the other trials, this is inappropriate, as there was only one trial each for the other probiotic strains. When I look back at my 2006 meta-analysis of different probiotics for the prevention of AAD, I appropriately reported one pooled RR for six RCT using S. boulardii and another pooled RR for the six RCT using L. rhamnosus GG[13]. But if I were to report these same results today, I would not have combined the remaining single probiotics and the mixtures in a pooled estimate, rather I would have reported separate estimates of efficacy for each probiotic strain sub-group.

Another consideration is whether to pool the type of disease being treated. One meta-analysis limited inclusion to the same type of probiotic (S. boulardii), but included 27 RCT on a variety of diseases, ranging from AAD to traveler’s diarrhea[29]. Each type of disease was assessed separately with a systematic review and a meta-analysis was appropriately done only for the disease indication that had sufficient numbers of trials (10 RCT for preventing AAD).

Inflammatory bowel disease encompasses ulcerative colitis, Crohn’s disease and pouchitis, which have different mechanisms and treatment outcomes. In a recent meta-analysis of 23 RCT for IBD, researchers correctly separated the three different disease profiles and additionally did sub-group analyses by the type of probiotic[14]. Only one type of probiotic mixture (VSL#3) was effective for treating both ulcerative colitis and pouchitis, but not for Crohn’s disease.

This learning curve for meta-analysis is not limited to just the field of probiotics. A similar learning curve to pool and not pool types of interventions is seen in the progression of studies investigating the role of red vs white wine vs beer for cardiovascular health. An early meta-analysis did not separate the different types of alcohol and pooled results from wine, beer and liquors[10]. A later meta-analysis did separate wine from beer and found a dose-effect on reduced cardiovascular risk from wine, but not beer[30]. Further research determined the main effect on cardiovascular health was due to the resveratrol (from red wine) and later meta-analyses focused on just red wine[11]. Similarly, Hooper et al[12] recognized that not all chocolate is alike, as they separated out white vs dark chocolate and cocoa into separate sub-groups in their meta-analysis of chocolate on cardiovascular health.

CONCLUSION

Now, we recognize that different types of interventions (for example, different probiotic species or different types of medications) should not be pooled in a meta-analysis, nor should different disease outcomes be pooled[15,22]. Yet, retrospective reviews often fail to acknowledge the learning curve as both the public and scientific researchers learn how best to perform these types of complex analyses. Table 1 shows my recommendations for the proper conduct of meta-analyses. Meta-analysis has its place, but we must resist the urge to jump to an easy conclusion that any of the treatments included in the analysis work once a pooled estimate shows efficacy. Systematic reviews may offer a more transparent analysis of multiple studies for clinicians and the public, but may be more cumbersome to interpret.

Table 1 Recommendations for performing a valid meta-analysis.

Begin with a systematic review of the literature, carefully reviewing original trial publications

Use a standardize data extraction form and independent reviewers to achieve reliable data for the analysis. Use a third reviewer to resolve conflicts

Use current guidelines for standardized reporting of meta-analysis results and incorporate a statistician into the analysis team

If the exposure/intervention/medication has been shown to have different effects within categories, narrow the groups until similar un-confounded groups are formed (for example, by strain of probiotics not by species or by genus, or by type of drug, not by class)

Use caution when using the overall pooled estimate of risk for concluding any type of exposure results in a similar risk of outcome