Reviewer 3 | A semi-serious discussion

I guess that if you landed here, you know what I am referring to, but let me clarify the subject of this post for the benefit of the youngest scientists. During peer-review, we get good and bad feedback, either deserved or not. We can then respond and revise our work. However, it is not rare to get the reports from the mythological reviewer 3 (also known by a different number), one that dismisses your hard work in ways that you will find particularly unfair and difficult – if possible at all – to rebut. There are various flavours of Reviewer 3, but common traits – not necessarily all present in an individual report – might be the condescending tone, vague unreferenced criticisms, the request of impossible experiments, a deep misunderstanding of the manuscript, accusations of various type. The important aspect of referee 3 is that, generally, responding to their critique is either factually impossible or would not improve the quality of your work. Of course, there is a lot of subjective interpretation here, and some of referee 3’s suggestions might be proper, or some report that at first glance are good (negative but well done) might be written by reviewer 3 in incognito. In any case, most scientists agree that Reviewer 3 does exist and, some of us asked for an independent assessment of a controversial refereeing report, might even know the identity of some of them, however ever-shifting they are.

***

Why Reviewer 3? Well, this is very anecdotal and indeed people might do the same ‘joke’ but changing the numbering. The argumentation I am going to elaborate on (I beg you, Reviewer 3, if you are reading, please remember this is not a completely serious discussion), does not depend on precise numbers, certainly not by the cardinality of the referee. In any profession, there are very skilled and bad professionals; this applies also to the academic world, of course. However, referee 3 does not have to be particularly bad scientifically, they might be the smartest of all, but for the scope of the refereeing process, referee 3 is doing this particular job and at that particular time, particularly badly, perhaps for lack of time, hubris, a particular emotional state, ignorance or for a genuine misunderstanding: it does not matter. It exists. Then, let’s take the anecdotal report of Referee 3, for a moment, at face value.

***

Let’s say that each time an editor nominates a reviewer, it is like the toss of an unfair coin (i.e., the probability of heads is different from the probability of tails) – either we get a Referee 3 or we do not. The probability to get at least one referee 3, is then the complement of the probability to not get one at all, i.e. p1=1-(1-p0)^n, with n the number of referees nominated by an editor. Which is the probability p0 for referee to be referee 3?

There might be some data out there, but as data is relatively unimportant both to make my point and to reviewer 3, I will assume that as Reviewer 3 is often called Reviewer… THREE, it is a frequent occurrence to observe one out of three referees been, well you got it, referee three. Then, after ‘careful consideration’, I assumed that one out of three is the most frequent occurrence.  The mode of a binomial distribution is floor((n+1)p0)=1 or ceil((n+1)p0-1=1. We can thus infer that between 1/4 to 1/2 of all referees could provide a Reviewer3-like  response. Hence, which is the probability to get at least one Referee 3 for your submission? Well, although a rare occurrence, if the editor asks the opinion of just one expert (perhaps as a preliminary inquiry) this probability is somewhere between 1/4 and 1/2, of course, identical to p0. For two referees, we will get a 43-75% probability and for three (the most common case), almost a 60-90% probability. Therefore, getting a Referee 3 report might be a rather obvious outcome of the peer-review process.

Now, let’s do another outrageous assumption. Let’s assume that also the editor, when handling a manuscript, could make the same mistakes as a referee 3 and that the journal has a very high bar for a manuscript to be accepted, i.e. any substantial negative feedback causes a rejection. In this case, the probability that the Referee 3 syndrome might negatively affect your submission is between 70-95%. Unrealistic? Maybe.

***

Now that several weeks passed, the referees’ reports are back in the hand of the editor. This is a very complex stage where so many objective and subjective factors might change how referee 3 is handled.

One possible outcome is that you get two Referee 3s… a rare outcome… isn’t it? If three referees have sent reports in, the probability to get at least two Referee 3s is actually between 15-50% Let’s say that – on average – a quarter of papers could be rejected because of Referee 3s, as if you get at least two of them any editor would, legitimately, dismiss the idea that those are ‘bad’ referees.

Let’s assume now you got just one referee 3 report. Again, with no intention to be accurate, these are the possible outcomes I can think about:

1. The Editor considers Referee 3’s points valid and the paper is rejected. Unexperienced authors will give up this submission at this stage, address any valid point raised during the refereeing and move to a second journal. Keep in mind now, that at the next journal, you will get the same probability of getting a Referee 3. However, if Referees one and two were positive with a few criticisms that could be addressed with new data, the experienced author would appeal. Until recently, I did not realize that Editors are quite open to this option assuming they find the manuscript interesting and that you get only a single problematic referee. Unfortunately, journals have mechanisms to discourage this path. However, if you can disregard emotions and humbly reassess your work on the basis of the Referees’ critique and you still find that the main issue is a Referee 3, engage – positively – the Editor. In most cases, you will find nice people trying to help out.
2. The Editor considers Referee 3’s points invalid and in one way or another, you will be allowed to address only the solid scientific point of Referee 3. It is very rare this will be written to you explicitly. I still find difficult to handle this situation. In most cases, this is the more likely situation you will get published even with a Referee 3 in the cohorts of referees. My suggestion is to speak with a senior colleague to decide how to proceed, or again to engage in a polite and proactive way the Editor.
3. The Editor considers Referee 3’s points invalid and asks for the opinion of Referee 4. This is the most sympathetic and proactive response that an Editor can have. However, this is also a situation that does not protect you from Referee 3, as the shapeshifting nature of Referee 3 might make them reappear with a differently numbered T-shirt. You will have between 25-50% to get another Referee 3 and being rejected not on merit. On a positive side, you might have up to 75% probability to replace a Referee 3 with a more objective peer.

***

Which is the point of this post? As I stated in the title, this is not a serious and quantitative analysis of peer-review. But I wished to address with outrageous simplifications a basic issue. Does the attitude of Referee 3 play an important role in peer-review? There are several reports showing how peer-review, despite its importance and the several mechanisms to establish a formal and objective process, give rise to a high degree of randomness in the outcome. Here, I just wished to point out that the probability to get a random and unfair report might be high. I leave to others the study of how high this value really is. However, while very experienced Editors and Authors might know how to handle the situation, there are two issues that concern me:

1. We are accustomed to harsh criticism. Often, a solid scientific debate is confused with been tough, and assertiveness is confused with freedom to not be polite. Who manage peer-review, academic or professional Editors, or managers in funding agencies, might consider this the natural and obvious rules of the game. Being a scientist has become something of a high-pressure job and it seems everyone has to accept this. Most of us are good and well-intentioned people, but the gears of this heavy machinery that is science are difficult to change, at least while the machine is in action.
2. The authors, or grant applicants, should have a very balanced approach. On the one hand, they should always make an effort to learn from criticism, even unfair criticism. This is a bit tricky with Referee 3. However, we always have to dissect Referee 3 to identify any useful critique. This is the trivial advise, trivial as it should be obvious. There is something more about this, that if you are a younger scientist with no proper mentoring, you might not know. Referee 3s can have a huge psychological impact on you. I’ve seen this happening to group leaders, and I have experienced this on my own.

*** UPDATE ***

After the publication of this blog-post, Reviewer 3 contacted me privately with the following message.

1. The assumptions the author does are all wrong and WordPress should not have allowed the submission of this article
2. The conclusions of the authors are clearly impossible as they conflict with a large body of literature
3. The authors do not cite any literature, but particularly the papers I published in 1965 that clearly and unequivocally demonstrate the opposite finding or the same findings.
4. The article is written in English, Latin would be the preferred choice for this field
5. Even if the authors could address these shortcomings with a major revision, this article should not be even posted on LinkedIn
6. Moreover, the article is poorly written, for instance, for instance ‘my own’ is not Korektly PhrammatiKalleee

*** UPDATE 2 ***

Hi Donald,

yes, that is sarcasm… not, you know…

Take care,

A.