Fame, Journals, and Blinding

A couple of weeks ago I blogged about a paper on the (possible) effect of double blinding on the bias against female authors. The paper had stirred up a few other comments, which got me thinking a bit more about why I'm a bit sceptical about double-blind reviews. Then I started thinking too hard, and ended up playing around with a simple model.

It's generally agreed that there is a bias towards better-known authors, so that a well-known author is more likely to have a manuscript recommended for acceptance than someone unknown. The argument for double-blinding is that it removes this bias, because the referee doesn't know who the author is. The problem with this is that it is often possible to guess who the author is (hell, it's sometimes possible to guess who a reviewer is) - a study 10 years ago (Cho et al. 1998) found that reviewers could work out the identity of the authors in about 40% of cases.

Presumably the authors who are recognised are the better known ones. We therefore have a situation where fame (whatever it is exactly) affects both whether a paper will be recommended for acceptance, and also whether the authors will be recognised. What effect does this have on the pattern of acceptance? Rather than just indulging in arm-waving, we can build a model, and indulge in arm-waving with numbers!

The model is simple, but hopefully captures the main points. Each author of a manuscript (for simplicity I will assume that each paper only has one author) has a fame. If the author's identity is known to the reviewer, then the probability of acceptance increases with their fame (the solid black line below).

If the reviewing process is double binded, then the probability that the author is recognised increases with the fame (the red dotted line). Note that it starts from a lower point, but increases more rapidly. if the author's name is not recognised, then the probability of acceptance is equal to the minimum probability. This is the the solid red line.


The technical details are below, for those who care. I have also scaled the probabilities of acceptance, so you can see them.

What does this show? Well, if you're a nobody, then the double-blind process means that you do as well as anyone else who isn't recognised, i.e. all but the famous. The famous do well under both systems, as they're recognised anyway. The people who lose out are those in the middle: the ones who are just starting to make a name for themselves, but are yet to be well known. With single blinding, their fame is enough that it helps them. Under double-blinding, though, they are not famous enough that they are recognised, so they are treated the same as a novice.

What this suggests, then, is that double blinding doesn't remove the biases: it just shifts them. So, the very famous actually do better under double blinding, as do the very obscure. Playing around a bit with the model suggests that the general result is robust, but it depends on the probability of recognition starting lower and having a steeper slope.

This is a model, using numbers that were plucked out of the air. But how does it compare to reality? My guess is that the effects are not as severe as shown here, but what is needed is data which can be used to estimate the parameters of the model. In the mean time, I'm not going to submit to any double-blind journals until I have my FRS.



The Maths
Fame, f, is uniformally distributed between -1 and 1. The probability of acceptance for a fame f, p_a(f), is modelled like this:


if the identity of the author is known, otherwise it is the minimum value. If the manuscript is double-blind reviewed, then the probability that the reviewer correctly recognises the name of the author, p_r(f), is


If a manuscript is reviewed double-blind, the probability that it is accepted is proportional to

p_r(f)p_a(f) + (1-p_r(f))p_a(-1)

The final probabilities are normalised, so that they sum to 1, by dividing by the sum of the probabilities.

References
Cho, M.K. et al. (1998) J. Am. Med. Assoc. 280, 243–245.

Read more!