There are basically 3 types of peer review in the field of medicine:
-
1) Medical peer review, by which organizations are appraised.
-
2) Clinical peer review, by which skills of physicians are evaluated.
-
3) Scientific peer review, by which articles submitted to journals are reviewed and their quality assessed.1
Here, I will concentrate on the last type of peer review. Scientific peer review dates back to 1752, when the Royal Society of London established a committee to assess publications submitted to their journal, Philosophical Transactions (a practice that may actually have originated in Edinburgh before this event).2 The journal editor considered this as an “internal peer review,” meaning that no individuals working outside of the journal looked at the articles. The purpose of the reviewers was to help the editor choose articles considered as appropriate for the general theme of the journal and not to strictly address their quality. Because the journal had considerable page space, rejections were rare and the goal was to fill it. In the early 1900s, scientific peer review, as we now know it, began in earnest in the United States. The first journals to use it were Science, JAMA, and American Practitioner. Peer review was not practical until 1959, when the photocopier was invented. At that time, multiple copies of submitted articles could be mailed to external reviewers without fear of losing them. During the next decade, the overall activities related to the science of medicine and its research increased dramatically, and the previous excess page space in most journals disappeared. Thus, editors saw the need to be more discriminating, and peer review began assessing the quality of submissions and only publishing those that passed a rigorous review. This is the type of peer review most current journals use and, in a poll, over 96% of scientists supported it.3
Two types of peer review dominate scientific journals:
Closed.
1) Unblinded: the names of authors and reviewers are known to each other; less commonly used than other systems because of the fear of introducing bias in the process.
2) Blinded:
Single-blinded: authors know the names of reviewers and may even suggest them (called “author-guided review”) or reviewers know the name of the authors (more common than the former and somewhat more controversial).
Double-blinded: neither authors nor reviewers know each others' names (used by most journals, including AJNR; number of reviewers varies per article between 2 and 3).
Open. Materials are generally posted on the World Wide Web (WWW) and are open to review by all users (even the general public). This may occur before or after publication (vide infra).
The effects of blinding the authors to the reviewers, and vice versa, have been studied. In one such study, keeping the names of the reviewers secret led to 8% more rejections than when they were known to the authors, implying that reviewers are more willing to reject a submission if they know that their identities will not be made known.4 Conversely, publicizing the reviewers' names led to a 5% increase in positive recommendations.5 In an assessment of which type of peer review is preferred, a survey of 838 individuals showed that 68% of reviewers favored not knowing the authors' names and that 72% of authors chose not to know the reviewers' names, findings that support the use of double-blinded peer review.6
Dr. Richard Smith, a former editor of the British Medical Journal, called peer review “expensive, slow, prone to bias, open to abuse, anti-innovatory, and unable to detect fraud.”7 In addition, the following problems have been noted with scientific peer review8:
1) Blinding reviewers to the authors' identities does not improve the quality of evaluations.
2) Passing reviewers' comments to their coreviewers has no effect on improving the quality of future reviews.
3) Spending more than 3 hours doing a review does not increase its quality.
4) Measuring the quality of peer review is challenging.
Who is a good reviewer? According to one study, the best manuscript reviewers are individuals younger than 40 years of age (I would have thought that the more experienced, and thus older reviewers, were better, but that is not the case), from top academic institutions (makes sense to me), personally known to the editor (I call this the “shame factor”), and those who are blinded to the identity of authors.9 If an individual has all 4 characteristics, 87% of his or her reviews will be judged as being excellent. If, however, the reviewer possesses only 1 characteristic, only 7% of his or her reviews will be excellent. In a different study, aimed at evaluating how carefully the reviewers analyzed submissions, the editors introduced “8 areas of weakness” (read: errors) into 1 article and sent it out to 420 reviewers.10 Of them, 53% completed the review, but only an average of 2 errors was detected by all. The investigators concluded that “neither blinding reviewers to authors and origin of the paper nor requiring them to sign their reports had any effect on the rate of detection of errors. Such measures are unlikely to improve the quality of peer review reports.” Another article's goal was to investigate which errors were detected by reviewers and if prior training in reviewing improved their ability to spot these.11 Nine errors were inserted into 3 articles and given to trained and nontrained reviewers. Overall, only 2–3 errors were detected, with biased randomization being the most frequently recognized. Training the reviewers did not significantly improve their reviews.
As alluded to before, misconduct also occurs in peer review in the form of personal vendettas, abuse of anonymity, false praise of submissions due to fear of vindictive authors, and, more importantly, plagiarism of nonpublished data for personal benefit.12 Dr. J. Eisen, editor of Plos ONE Biology, had the following to say about peer review: “If you asked someone today to design from scratch a peer review system, they would not design it the way it is,” and “Having 2 or 3 reviewers and one editor as gatekeepers of scientific knowledge is a mistake. It has too much potential for limiting the spread of scientific knowledge.” In 2008, the Cochrane report stated the following13:
1) There is no clear effect of author or reviewer blinding.
2) There is no evidence that reviewer training improves the quality of the process.
3) Different ways of communicating with reviewers have no effect on the quality of reviews.
4) There are little data to support that peer review improves the quality of published articles.
Thus, the next question that comes to mind: Is there a better process? The quest continues and today “open” peer review is receiving a lot of attention. As peer review was only possible after photocopies were widely available, open review was only possible after the WWW matured and morphed into Web 2.0. Benefits of open review are increased number of reviewers, increased transparency, more constructive criticisms (the last 2 observations are based on the fact that the names of reviewers are generally known and that all discussions remain archived on the Web), and higher quality submissions.14 Drawbacks include that sometimes it is not possible to get enough reviewers (because it is a completely voluntary and unsolicited activity), delayed publication, and, overall, a system that is more complex than double-blinded peer review. There are 2 types of open reviews: pre- and postpublication. A detailed explanation as to how each type works would be too long for this Perspectives, and I have included these in pictorial form (Fig 1). While some journals have successfully implemented open review, others have failed. In 2006, Nature allowed authors to choose between their traditional peer-review system and open review, and only 71 of 3000 opted for the latter.15 At the end of the trial, the editor commented as follows: “From informal feedback, it was clear that the trial generated a lot of casual interest, but no hostility or enthusiastic endorsements in any quantity. Unsolicited comments posted on the Web were less useful than those from designated referees but in principle could draw attention to something not spotted by the referees.”
Diagram comparing 3 different types of scientific journal peer review. On top (white boxes) is our traditional double-blinded peer review. In the center (pink-shaded boxes) is a hybrid system that incorporates external open review into the double-blinded one. This results in a time penalty and increased expenses and thus is not practical. On the bottom (red boxes) is an open-review-only system that may save time and expenses in publication but remains controversial.
How about a hybrid peer-review system? This implies open review for only selected articles. The Proceedings of the National Academy of Sciences uses such an approach and open reviews occur after acceptance and posting of articles. Open reviewers like it because they are not responsible for acceptance/rejection, and the assigned editor and original referees remain anonymous. This is true of postpublication peer review, and even journals outside of the sciences, such as Shakespeare Quarterly, have successfully implemented it. An easy way to go about postpublication open review is to use blogs and social media. Benefits include openness, keeping articles fresh by continuous evaluations and changes, and publication schedules that are not significantly affected. Drawbacks are amateurish evaluations and the fact that people are reluctant to blog (similar to our experience with ajnrblog.org).16 Plos ONE has used postpublication open peer review successfully.17
Mr. Viter Tracz (responsible for Gower Medical Publishers, the Current Opinions Journals, and BioMed Central) considers himself the father of open access and said the following of peer review: “Except for a tiny little part at the top where it is done seriously, peer review has become a joke. It is not done properly, it delays publication unnecessarily, it is open to abuse, and is being abused. It is seriously sick, and has been for a while.” He believes that a large group of volunteer scientists should do open review and thus created the “Faculty of 1000.”18 This Web site specializes in postpublication rating of articles and contains over 112,000 as of this writing. The neuroimaging section of F1000 is mostly composed of basic science, but the articles rated include clinical ones, too. Articles are rated on a numeric scale and given comments such as “Must Read,” “Exceptional,” and so on. They also rank journals (AJNR is ranked 813 of 1129) based on the number of articles F1000 reviews, article grades, and total yearly publications of that journal. So, they ranked AJNR based on only 2 articles! Other similar Web sites, such as Evidence-Based Medicine, use a star rating system a la Amazon (this type of rating became available in AJNR in March 2012). On both sites, the articles tend to be older, as they only rate those that are open access (1 year for AJNR but 2 years for most other journals). The Radiology Best Evidence Newsletter from Medscape is more contemporary, but because most articles rated are not open access, only their abstracts are found there.19 Facebook may also be used as a means of postpublication open peer review, and the New England Journal of Medicine has been successful with this method. Conversely, AJNR has not, but maybe as neuroradiologists become younger, it will.
Whatever peer review system we continue to use, we need to be careful, as governments are starting to look into this issue. In England, the House of Commons Science and Technology Committee issued a report on “Peer Review in Scientific Publications.”20 They concluded that there are many ways of doing peer review, that publishers should offer a variety to suit the needs of different publications, and that the importance of prepublication assessment is crucial and this always requires subjective judgments that may result in errors. They encouraged different research groups to optimize review systems and foster innovations, and stated that openness and transparency are attractive, and, at the end, they congratulated Plos ONE on the quality of their on-line programs. More importantly (although not directly related to the topic of this Perspectives but one that is the result of peer review), they stated that the use of the Impact Factor to measure the quality of articles is questionable (I imagine this is where the importance previously placed on postpublication review comes in) and that the Impact Factor should not be used when assessing individuals for career progression.
To end on positive note, a former editor of JAMA said, “Peer review represents a crucial democratization of the editorial process; incorporating and educating large numbers of the scientific community, and lessening the impression that editorial decisions are arbitrary.”21
AJNR uses the time-honored double-blinded peer-review system, but our readers and the general public are welcome and encouraged to use our blogsite and Facebook page as a means of postpublication open review.
References
- © 2012 by American Journal of Neuroradiology
