ArXiv and double-blind reviewing revisited

Joakim Nivre
Uppsala University
President of the ACL

It is with great interest that I have followed the discussion in connection with ACL Vice-President Marti Hearst’s post last week on the conflict between pre-acceptance arXiv publishing and double-blind reviewing. Just to recap, most people agree that the former has a number of advantages such as fast and easy access, “time-stamping” of results and ideas, and an open feedback and revision process. At the same time, posting papers on arXiv prior to acceptance for publication clearly undermines double-blind reviewing, which in a number of studies has been shown to promote fairness and prevent bias in the selection of papers.

I draw two conclusions from the discussion so far. The first is that there is no simple or obvious way out of the dilemma, which just confirms what I have thought for a long time. The second is that opinions are quite divided in the community, something that was perhaps less clear to me previously. Some people consider double-blind reviewing worth saving at almost any cost because of the proven advantages. Some people feel just as strongly about the merits of fast and open dissemination of research results using arXiv. And some people are actively looking for compromises to save the best of both worlds.  

The ACL Executive has discussed this issue on and off since I joined the committee two years ago, and it has become clear that we need to do something about it. We need both a long-term and a short-term strategy. We need to work long-term because it is a hard problem where a viable solution is unlikely to be found quickly, and because it is part of a larger complex of issues having to do with how we want our models of publishing and reviewing to evolve in the future. In addition, being a relatively small community, we probably need to work in synergy with neighboring communities to find a viable solution, especially if we want to impose changes to existing systems, and this will also take time.  

At the same time, we need to find a workable short-term solution. Because another conclusion I draw from the recent discussion is that the policy currently adopted by TACL and most of our conferences is not really working. According to this policy, a paper can be submitted for double-blind review despite being posted (non-anonymously) on arXiv as long as the authors declare this. This may have been an appropriate policy when arXiv was a marginal phenomenon, but to most people today it appears either confusing, ineffective, or simply unfair (because we keep rejecting papers whose anonymity has been compromised in other ways). And a policy that lacks support from the community is not going to work in the long run.  

So what is the ACL Executive planning to do about this? At our recent winter meeting, we decided first of all to gather more information by organizing a membership poll. The details are still to be worked out, but the poll is likely to contain questions about your current practice – to what extent you post papers on arXiv prior to acceptance, for example – as well as about your views on possible alternative systems and policies – such as whether you are willing to sacrifice pre-acceptance publication to save double-blind review or vice versa. While you are waiting for this poll, you may want to check out an earlier poll organized by Kyunghyun Cho:
(http://www.kyunghyuncho.me/home/blog/toarxivornottoarxiv).

We also have plans to organize a special session at ACL 2017 in Vancouver to discuss this and related issues. Except for our regular business meeting, we normally do not have meta-sessions at our conference, but we feel that this is important enough to the community to set aside a bit of time somewhere in the program. Again, the details are still to be worked out, but one of the things to discuss will surely be a new and better policy that can be put in place for next year’s conferences. If you have ideas about how this session should be organized or would like to contribute, please get in touch with me or any of the other members of the ACL Executive – or simply make a comment on this post!

Based on the input that we gather from the poll and the meta-session, and hopefully with a better short-term policy in place, we should be in a better position to map out a long-term strategy towards a viable model for publishing and reviewing in our field. Deciding how to deal with the conflicting pressures of arXiv publishing and double-blind review is only one of many strategic decisions that we have to face, but it is clearly a very important one and we look forward to getting your input on this. Because a long-term strategy can only be successful if it is well anchored in the community.