EDITOR'S PAGE
What constitutes a great review?
Anthony N. DeMaria, MD, MACC*,1
* Address correspondence to: Anthony N. DeMaria, MD, MACC, Editor-in-Chief, Journal of the American College of Cardiology, 3655 Nobel Drive, Suite 400, San Diego, California 92122, USA.
| |
Anthony N. DeMaria, MD, MACC
|
|
The last Editor's Page dealt with the topic of a report card for journals. Just as journals are subject to grading, so are the critiques of manuscripts prepared by reviewers. In fact, the Editors grade each review for quality based on a number of characteristics we have found to be of value. However, these characteristics have not been well disseminated. During our recent Editorial Board retreat, a number of Editorial Board members indicated that they had never received counseling on how best to review an original research manuscript. The consensus was that it would be useful for the Editors to define what we regard as the attributes of an outstanding review. We discussed this topic at one of our weekly meetings, and the following summarizes our thoughts on the matter.
As I have previously indicated, when a manuscript critique is done well, it requires time, effort, and often a certain amount of background research. The rewards for such efforts are meager and generally consist of an insight into the latest developments in the area of research and the opportunity to influence the material being read by the cardiovascular community. Generally, the function of reviewing manuscripts is regarded as a contribution to the academic pool and as reciprocation for having someone else undertake to review one's own articles. Nevertheless, reviews can vary from excellent to cursory, and their value is not limited to the crucial role they play in determining acceptance or rejection. Perhaps even more importantly, they can markedly improve an article for final publication. Therefore, it is hoped that reviewers will attempt to be as proficient in this task as they are in authoring articles.
Many critiques begin with a brief summary of the manuscript being reviewed. This may at first seem redundant, since the Editors have clearly read the manuscript themselves. However, the summary provides us with a valuable insight into the reviewer's perspective of how the study was conducted and the findings reported in the paper. Perhaps not surprisingly, individual reviewers will often come away from the same manuscript with very different impressions of the nature of the study. Even just a description of the hypothesis tested, methods applied, and results found often conveys an indication of the impression the article had on the referee. Therefore, although it is not absolutely necessary, a brief overview of the manuscript is very useful to the Editors in determining its priority for publication.
The heart and soul of a manuscript review is, of course, an evaluation of how well it serves the scientific process. Was there a rationale for the objectives of the study, and was the hypothesis to be tested important? Were the methods used appropriate and accurate, and was the resulting data appropriately measured and analyzed? Were the conclusions drawn justified, and the findings significant? In short, was the article original, accurate, and relevant? It is surprising how often this basic information is never commented on in a critique. Very often the reviewer will merely begin by raising specific issues related to some aspect of the paper. The Editors are left to wonder whether the omission of comments regarding the basic structure of the work implies that it was well done or that it was never evaluated. We ask for specific grades for each of these categories, of course, but as will be discussed below, we often get grades without justification.
The best reviews place the manuscript in proper perspective. Such critiques indicate the state of knowledge in the field being studied and how the manuscript contributes the knowledge in the area. It is, of course, the responsibility of the authors to provide the appropriate background for their work. On occasion, however, the true significance of original research is missed by the author and first identified only by the reviewer. The Editors, who are often not expert in an area, certainly appreciate the orientation that a review can provide for a manuscript.
One of the most consistent characteristics of an excellent review is the inclusion of references and citations from the literature to support statements made by the referee. This is, of course, most important in assessing the originality of manuscripts. Very often a reviewer will comment that there is "little new information" contained in a research article. However, the authors of the manuscript typically disagree, having stated in their introduction that few or no data exist. Moreover, the natural bias of the Editors is to assume that if the information contained is new to them, it is likely to seem original to the majority of readers. The citation of previous work reporting the same or similar data as in the submission under consideration is therefore of inestimable value in determining the priority for publication. In the same vein, literature citations are of great value in substantiating the criticisms leveled by reviewers against manuscripts. The claim in a review that methods are inappropriate or that variables capable of influencing data are ignored is most effective when supported by appropriate citations. The Editors are often adjudicating differing opinions regarding the same manuscript by individual reviewers, and the opinions supported by data are always the ones assigned the greatest significance.
It never ceases to surprise us how often reviewers neglect to provide any comments in the space provided for "Comments to the Editors." Often the referees merely repeat their comments to the authors. The "Comments to the Editors" should provide a summary assessment of the manuscript and justifications for the accept/reject recommendation and the priority scores assigned. Most critiques include both positive and negative statements in the "Comments to the Authors" section. In addition, individuals differ considerably in assigning priority scores, just as some teachers and professors are "easy graders" and others are "hard graders." Therefore, it is of enormous value for the reviewers to put into prose their overall assessment of the article and the reasons supporting this assessment in their "Comments to the Editors." While the ultimate decision to reject or accept an article must rest with the Editors, a review that provides an opinion for or against acceptance, with supporting reasons, is always regarded with appreciation.
It goes without saying that the best reviews will convey the same message to the authors and to the Editors. The number of manuscript evaluations we have received that convey one opinion of the work to the authors and a different one to the Editors was quite unexpected. Such critiques are neither helpful to the authors in improving their manuscripts nor helpful to the Editors in explaining our editorial decision. The Editors dread such reviews and immediately prepare for letters of appeal from the authors.
I have heard it said that editors of medical journals are either "abortionists" or "midwives." I think this statement very well summarizes the charge to the reviewers. Firstly, the reviewer is asked to represent the journal in selecting only those manuscripts that are of the highest quality for publication. The referee must protect the journal from accepting flawed manuscripts that contain errors in methods and analysis or inaccurate conclusions. In addition, given the low acceptance rate of most journals, the reviewer is asked to help the journal prioritize the merit of individual submissions within the limited number of pages available. In this role, the reviewer is clearly sitting as a judge and emphasizing the identification of significant flaws in the paper. However, the reviewer is also asked to seize the opportunity to act as consultant to the authors and improve their papers. The Editors have seen numerous instances in which manuscripts have been enormously upgraded by the peer-review process, and virtually all manuscripts are improved afterwards. In this "midwife" role, an excellent review is one that is objective and constructive, one that avoids antagonism and points out areas in which the article can be improved.
A last issue worthy of discussion relates to reviews judging that manuscripts are not well suited for publication in JACC. Clearly, there is some subject matter that will be of little interest to most of our readers. However, reviews indicating that original research submissions are "not for our audience" are assuming a knowledge of what the bulk of JACC readers will find of value. Although the Editors nearly always agree with those reviews that assess manuscripts as not suitable for JACC, the best reviews include a justification for this opinion and provide evidence that the manuscript will not be of interest either to the reviewers or the general readership.
Just like parenthood, the reviewing of manuscripts is one of those things in life for which there is very little formal training. Although the task itself seems quite straightforward, critiques differ significantly in their quality. Given the importance of the undertaking to both the authors and the scientific community in general, manuscript review is a responsibility that should not be undertaken lightly. From the standpoint of the Editors and the Journal, the peer-review process is fundamental to our existence. When the characteristics previously described are present in a review, it always gets commented upon when the paper is discussed in our weekly meetings. The real evidence of the value of a quality review lies in the fact that the Editors are all familiar with the names of those individuals whose critiques are consistently excellent.
 |
Footnotes
|
|---|
1 Editor-in-Chief, Journal of the American College of Cardiology 
This article has been cited by other articles:
|
|
|
|
 
R. M. Rosenfeld
How to review journal manuscripts
Otolaryngology -- Head and Neck Surgery,
April 1, 2010;
142(4):
472 - 486.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
E. P. Havranek
Characteristics of a great review
J. Am. Coll. Cardiol.,
May 19, 2004;
43(10):
1927 - 1927.
[Full Text]
[PDF]
|
|
|
|
