How to Remove Bias From Peer Review

This article originally appeared in The Chronicle of Higher Education on May 7, 2015.

By Teri W. Odom

The ugly side of peer review was on full display last week when a scientific paper was rejected for reasons that smacked of sexism. Two female authors had submitted a paper to a journal that is part of the open-access PLOS family. A negative decision was made based on a single review stating, “It would probably also be beneficial to find one or two male biologists to work with (or at least obtain internal review from, but better yet as active co-authors). … ”

The reviewer has since been removed from the PLOS reviewer database, and the editor was asked to resign from the editorial board. But the quandary concerning overt sexism — even misogyny — in academic journals remains.

In scientific journals, protection of peer-reviewer identity is a key tenet to ensure unbiased and fair assessment of the submitted work. In its best form, reviewers offer helpful critiques to improve the paper by pointing out possible errors in analysis, asking for more interpretation, or suggesting that other data is needed to fully support the findings.

Such candid comments are usually better received on paper than in person, which is why reviewer identity is protected. Abuse of this protection, however, allows reviewers free rein to provide their opinions, fair or not, about the work without facing consequences for inappropriate reviews or for needing to be responsible for their words.

Taken to an extreme, abuse of peer-review protection results in outcomes very similar to those of Internet trolls: Harmful comments can be made without recourse. Everyone knows anonymity breeds contempt.

I believe this unfortunate incident — which generated deserved outrage — is a failure of editorial leadership. The onus should not be simply on the reviewer.

As executive editor of ACS Photonics, part of ACS Publications, my role is to serve the scientific community by identifying expert reviewers, inviting them to offer unbiased and fair assessments of the submitted paper, and then making a decision based on those reviewer recommendations as well as my own reading of the paper.

To ensure consistency in the review process, typically two or more reviews are needed. What is critical to note is that although we invite reviewers to make recommendations on the paper (e.g. accept, revise, reject), it is the prerogative of the editor to actually make the decision.

In the unfortunate case above, editorial leadership was clearly missing. The editor should never have passed those comments along to the author, and there should have been more than a single, unsubstantiated review before a decision was made. And yes, that the editor stepped down from the editorial board was appropriate.

How systemic is this problem?

Although such data may be hard to come by, one significant change to the publishing landscape has been the rise of open-access journals. The idea behind these journals is appealing, where new scientific results can be made freely available to the public — and, indeed, there are some excellent ones.

However, the majority of open-access journals have focused on technically sound results of already reported findings, which may not necessarily translate to scientific impact. But the bigger challenge is that for many of these journals, the editorial process is not as rigorous, which, when unchecked, can result in the publication of papers on topics that are absurd or with no new content.

Editors of reputable scientific journals — practicing scientists and professional editors — have an obligation to ensure a fair review process and to protect authors from inappropriate and unsubstantiated comments. If editors are not held accountable, increased incidents of poor peer review and bias are bound to occur.

Early in my scientific career, I received a review that contained disparaging remarks such as my being “out of touch with modern developments” with “no elements of novelty … that could warrant publication … in any good journal.” Even as I told my students not to take such remarks personally, it was hard not to. And, although this particular reviewer might write like this for everyone, the intent was clearly to harm the chance for possible publication (which worked, for that particular journal).

Perhaps a system of review for editors and reviewers would provide a check for demeaning comments. For example, if an editor fails to communicate that such commentary is not acceptable, then she or he should step down from that position. And if a reviewer makes outrageously biased comments more than once, then he or she would be barred from publishing in and reviewing for that family of journals.

Perhaps only then will it become common practice to concentrate on the science and the content, not the sexism.

- Teri W. Odom is a professor of chemistry at Northwestern University.