EDITOR'S PAGE
Publication bias and journals as policemen
Anthony N. DeMaria, MD, MACC, Editor-in-Chief, Journal of the American College of Cardiology*
* Address correspondence to: Anthony N. DeMaria, MD, MACC, Editor-in-Chief, Journal of the American College of Cardiology, 3655 Nobel Drive, Suite 400, San Diego, California 92122 (Email: ademaria{at}acc.org).
 Publication bias, manifested most prominently by lack of publication of negative results, has been an increasing source of concern for the past several years. In fact, in 2000 Song et al. (1) published a 120-page monograph devoted entirely to publications bias from the Health Technology Assessment Program of the National Health Service. The issue came to a head recently when Glaxco was sued for withholding results from clinical trials showing a lack of efficacy and potential increase of suicidal thoughts among adolescents on the antidepressant Paxil. The firm paid a fine and agreed to publish both the protocols and results of the clinical studies of its drugs in an open registry. Soon thereafter, a group of editors announced that they would no longer publish clinical trials whose protocols had not previously been placed in a public registry (2). These events have raised a number of questions: What constitutes publication bias? How prevalent is it and what effect does it have? And most importantly here, what is the role of journals in solving the problem?
The initial observation of publication bias dates to 1959 when Sterling (3) found that 97% of articles in four journals reported statistically significant findings, raising the likelihood that studies lacking significant differences were not being published. Primary attention has since focused upon unreported clinical studies with negative results, often referred to as "the file-drawer problem." Concern has recently been extended to incomplete publications that fail to present all the studies undertaken and/or results available. In fact, a recent study found that of 102 randomized trials examined, 50% of efficacy data and 65% of adverse effects were incompletely reported (4). The definition of publication bias has been further expanded to encompass dissemination bias, or the accessibility to research findings related to when, where, and in what format the findings are published. Thus, studies whose publication is delayed, or appears in non-indexed journals or uncommon languages, or is ignored by the media, may also be considered to suffer publication bias. The common denominator, of course, is the failure to bring data of potential importance to the attention of physicians and the public.
The possible consequences of publication bias are also fairly obvious. Of greatest importance is the misrepresentation of the efficacy or adverse effects of a diagnostic or therapeutic modality. Unreported adverse effects are, of course, the most dangerous possible consequence. Current guideline recommendations are usually based upon at least two publications with concordant results. If only two of three or four completed studies are positive, and only those results are reported, one obviously gets a distorted view of the efficacy of the intervention. We live in an era where meta-analysis of multiple small trials is often required to assess significance and efficacy. Clearly, meta-analysis will be extremely flawed if all available data are not taken into consideration. Unreported clinical studies represent wasted resources, and may result in other investigators undertaking the same futile experiment. Finally, failure to publish results is unfair to the patients who have voluntarily participated at some possible inconvenience, discomfort, and risk. There can be no doubt that publication bias is a serious problem for medicine.
The responsibility for publication bias can be attributed to multiple sources. Investigators bear a major responsibility, because publication must begin with them. There is little question that negative results stimulate less enthusiasm for the analysis and manuscript preparation required for peer-review publication than positive findings. My experience has confirmed that reviewers generally share in the lack of enthusiasm for negative results. In fact, most editors are likely inclined to view negative findings less favorably because we are seeking to publish findings that will alter medical practice. When prioritizing limited pages, it is only too easy to downgrade a study "which just shows negative findings." The media and authors of review articles are also less likely to feature negative trials. We at JACC are often counseled not to accept reports of unsuccessful interventions since "they will disappear on their own."
Finally, the role of the medical industry must be considered. The incentives inherent in the profit motive are so intuitive that suspicion nearly always exists regarding the potential of industry to suppress negative results. In fact, evidence of such behavior does exist. However, in my opinion, the role of industry in contributing to publication bias is less, overall, than that of the other sources.
As in so many controversial areas of medicine, precise data regarding the prevalence and actual effects of publication bias are lacking. The literature is replete with direct or indirect evidence of bias in virtually every area of medicine; it almost seems that if you look for it, you will find it. However, the true magnitude of the problem remains undefined (1). (It should be noted that articles on publication bias are susceptible to such discrimination themselves. That is, papers seeking to report the failure to find publication bias may face a lesser chance of being submitted and accepted for publication.) Similarly, it is difficult to measure the effect of publication bias upon clinical care and patient outcome. In most cases, interventions that fail in clinical trials are abandoned. However, the absence of quantitative data regarding the effects of publication bias should not diminish the importance of the problem.
Given the various sources responsible for publication bias, it is clear that no single action will eliminate the problem. Efforts should be made, using either a carrot or a stick, to have investigators prepare and submit manuscripts describing the complete results of clinical studies. Given the need to publish for academic advancement, one would think that this should be easy. Removing any external inhibition arising from sponsors and journals would definitely be of value. Likewise, non–peer-review medical media should be encouraged to disseminate and extol the contribution of negative clinical studies. This may be more difficult to implement as negative studies are often, de facto, less interesting.
Despite the value of these measures, the focus of attention has clearly turned to medical journals to rectify the problem of publication bias. It has been recommended that editors not only remove any references to the desirability of significant findings from their instructions to authors, but proactively encourage submission of clinical studies that fail to reach significance. Editors could minimize publication bias by being especially observant in the selection of reviewers and consideration of their critiques. It has even been advocated that journals extend a provisional acceptance for publication based upon the rigor of a research protocol. Rickard Horton of Lancet has done this for some time, with the rationale that a well-designed protocol will yield useful results whether they are positive or negative.
The intervention to overcome bias that has been most promoted has been universal registration of clinical studies. Such registration would place the trial in the public domain and would not only include the rationale, methods, investigators, and sponsors, but ultimately the results. It has been argued that the best way to achieve such a registry would be for journals to require registration at enrollment to qualifyfor publication. Presumably, no investigator would risk having the results of a trial go unreported for lack of registration.
To implement this strategy, the International Committee of Medical Journal Editors (ICMJE), comprising 10 general medical journals worldwide, published editorials stating that from July 2005on they would require, as a condition for publication, registration in a public trials registry at the start of the enrollment. The registry must be accessible to the public at no charge, electronically searchable, have a mechanism for data validation, and be managed by a not-for-profit organization. The editors specifically mentioned the clinical trials registry sponsored by the U.S. National Library of Medicine. Although these editors spoke for themselves, they urged all other journals to join them in this policy.
I must admit that I have somewhat mixed feelings about this proposal. There is no question in my mind that publication bias is a significant problem and must be addressed. I also agree that medical journals have some unique abilities to bring to bear on this problem. Therefore, I certainly do not object to having JACC adopt the requirement for clinical trials registration advocated by the ICMJE. However, while this action may be of value in managing publication bias, it will neither eliminate it nor is it necessarily the appropriate function for a journal. To begin with, registration alone will not ensure submission and publication of results nor convey the credibility of peer review. Neither will this action foster dissemination of the results when the trial is completed. Journals have significant page limitations, at least with regards to print, and existing journals could not likely accommodate every clinical trial in current pages. Moreover, an important function of journals is to prioritize the most newsworthy information for its readers, findings which will change behavior, so that they can devote available reading time most efficiently. This prioritization is clearly one of the attractions of each journal, and editors will likely be reluctant to compromise it.
Finally, I wonder if it is an appropriate role for a journal to be a policeman. The job of editors is to bring important new information to readers as soon as possible, not to administer punitive measures to investigators who do not adhere to standards for research. If in the course of an unregistered (for any reason) clinical trial the investigators stumbled on a very important finding, would editors really refuse to publish that finding? And if so, would society benefit? It seems to me that to require trial registration to qualifyfor publication is at best an imperfect attempt to address unreported study results, and one that uses journals for a purpose for which they were not intended. Nevertheless, until a better, more comprehensive solution to publication bias comes along, JACC will try to observe the registration requirement. Just as in politics, the best thing about some options is the alternative.
 |
References
|
|---|
1. Song F, Eastwood AJ, Gillbody S, Duley L, Sutton AJ. Publication and related biases Health Technol Assess 2000;4:1-115.[Medline]
2. De Angelis C, Drazen JM, Frizelle FA, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors N Engl J Med 2004;351:1250-1251.[Free Full Text]
3. Sterling TD. Publication decisions and their possible effects on inferences drawn from tests of significances Am Stat Assoc J 1959;54:30-34.[CrossRef]
4. Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials JAMA 2004;291:2457-2465.[Abstract/Free Full Text]
|
Top
References