EDITOR'S PAGE
Meta-Analysis
Anthony N. DeMaria, MD, MACC, Editor-in-Chief, Journal of the American College of Cardiology*
* Address correspondence to: Dr. Anthony N. DeMaria, Editor-in-Chief, Journal of the American College of Cardiology, 3655 Nobel Drive, Suite 630, San Diego, California 92112 (Email: ademaria{at}acc.org).
The topic of meta-analysis has come up on several occasions during the weekly meetings of the editors, and has provoked some spirited discussions. These systematic reviews integrate and synthesize existing research studies in an attempt to derive new information by quantitative statistical analysis. We are quite impressed with the increasing number of such articles that we are receiving at JACC. A quick check of PubMed confirmed that our experience is not unique. A search for the keyword "meta-analysis" in PubMed yielded over 110 pages listing over 2,100 papers since January 1, 2008, while "meta-analysis and heart" alone accounted for over 130 articles published in the first 6 months of this year. In fact, some articles are now being devoted to meta-analysis of meta-analyses, so called meta-meta analysis (1). It seems clear that we are experiencing a virtual explosion of this type of research activity.
There is no doubt that meta-analyses have many positive attributes (2,3). Busy physicians have difficulty keeping abreast of the huge volume of medical literature, and some may not possess the analytic skills to resolve the often non-definitive or conflicting findings. Meta-analysis provides an attractive solution to this problem. By examining the totality of data available about an issue, meta-analysis can identify inadequacies in existing data and point to areas of needed research, reduce the potential for erroneous findings occurring by chance, and more accurately define the benefit and possible adverse effects of management strategies. In fact, by combining smaller datasets, meta-analysis may establish unrecognized outcomes, may provide evidence of statistical significance where it was previously absent, or may eliminate any possible bias in individual studies. While prospective randomized trials remain the highest level of clinical evidence, meta-analysis can provide generalizability that is often lacking in such protocols.
Having delineated its attributes, it must be acknowledged that meta-analysis carries some significant limitations. Primary among these, at least in my mind, is the total dependence upon the quality of the studies that are included. No amount of statistical manipulation can overcome defects in poorly conducted studies, a concept often expressed as garbage in–garbage out. Heterogeneity among studies is another major limitation. Heterogeneity may involve the number and type of patients included, the location of the study, the methods and/or equipment used, the experience of the investigators, the presence of additional medications, and other variables. Alternatively, the individual studies may differ in the results obtained. Obviously, the more homogeneous the findings of individual reports, the greater the confidence in the results of integration. Meta-analysis is very vulnerable to the effects of publication bias, or the tendency for studies with negative results to be less often submitted and accepted for publication than those with positive findings. Hopefully, the registration of clinical studies at their onset will help overcome this problem. Finally, meta-analysis is subject to the same potential bias of the investigator and/or sponsor as are all research endeavors.
In addition to the above limitations, meta-analysis, like all investigation, is susceptible to flaws in methodology. Accurate meta-analytic methodology requires that a comprehensive literature search be performed and that strict criteria for the exclusion of studies be provided. In general, meta-analyses are graded on the quality of the data included, the quality of data extraction, the statistical methodology used to combine data, and the degree to which the limitations of the study are recognized and addressed. Despite these clearly delineated requirements, evidence exists that meta-analyses are often performed imperfectly. In fact, it has been reported that systematic reviews in fields such as anesthesia, critical care, surgery, emergency medicine, and others often contained deficiencies, and that 80% of such material published on asthma prior in one survey had major flaws (4). This may partially explain why individual meta-analyses on the same topic have been found to yield discordant results (5). Thus, in addition to the limitations inherent in the technique, the validity of the results may be compromised by suboptimal performance of the study.
In view of the above it is not surprising that many individuals, including some JACC Editors, have been doubtful about the value of meta-analysis. They are skeptical that separate studies involving different patients, coming from different locations, and treated for dissimilar time periods according to diverse protocols, can yield any meaningful results. A quip often heard from the skeptics is that "meta-analysis is to analysis as metaphysics is to physics." Even the advocates recognize that combining apples and oranges carries the danger of sometimes resulting in a fruit salad. Nevertheless, there appears to be no diminution in the enthusiasm for this type of investigation, and the manuscripts continue to arrive at a brisk pace and to generally positive reviews.
As for myself, I have mixed feelings about meta-analysis. I recognize all of the attractive attributes of these systematic, quantitative reviews. No one can deny the value of a paper that identifies the best data on a question, subjects the extracted data to valid statistical analysis, and yields a clear conclusion from the whole that was not established by the individual studies. Such papers serve to help you keep up with the literature, and can provide the foundation for evidence-based medical practice. I do worry that such studies make it easy for me to be intellectually lazy and not grapple with the source data myself, but this is not the fault of the study. However, I believe that there are a limited number of settings where new, valid, useful information can be derived by this technique. The optimal situation exists when a reasonable number of relatively well done, generally homogeneous randomized trials exist that do not have adequate power to establish a statistically significant conclusion. Meta-analysis is of lesser value when only a few, small trials are available or when there are a substantial number of trials that all have the same findings. Why do a meta-analysis if there is no disparity in the data? There is a potential role for meta-analysis when multiple studies exist that have different findings. In this situation, however, the heterogeneity often compromises accuracy and the issue is probably better resolved by a large randomized clinical trial.
And so, despite its limitations, it is likely that we will continue to receive an increasing number of meta-analyses. They are popular with readers, well received by reviewers, and increasingly used to define evidence-based practice and guidelines. Although it challenges logic that one could obtain new accurate information from the quantitative integration of a number of very diverse studies, the numerous meta-analyses published speak for themselves. Used in the proper setting, I think they can make a valuable contribution. The job of the Journal will be to ensure that those published are in this setting and are methodologically sound.
 |
References
|
|---|
1. Delgado-Rodriguez M. Systematic reviews of meta-analyses: applications and limitations J Epidemiol Community Health 2006;60:90-92.[Free Full Text]2. Crowther MA, Cook DJ. Trials and tribulations of systematic reviews and meta-analyses Hematology 2007:493-497. 3. Halligun S, Altman DG. Evidence-based practice in radiology: steps 3 and 4—appraise and apply systematic reviews and meta-analyses Radiology 2007;243:13-27.[Abstract/Free Full Text] 4. Boluyt N, van der Lee JH, Moyer VA, Brand PP, Offringa M. State of the evidence on acute asthma management in children: a critical appraisal of systematic reviews Pediatrics 2007;120:1334-1343.[Abstract/Free Full Text] 5. Jadad AR, Cook DJ, Browman GP. A guide to interpreting discordant systematic reviews Can Med Assoc J 1997;156:1411-1416.[Abstract]
|
Top
References