Dear readers!
As comments published in response to our last blog illustrate, the issue of arXiv and blind reviewing is controversial. The only part on which everybody seems to agree is the inadequacy of the current policy. We are rejecting papers for non-anonymised submissions where the authors genuinely forgot to remove their names, but let in papers previously submitted to arXiv, if they are declared during submission time. While this declaration supposedly gives reviewer an option to avoid seeing author names, in practice it is almost impossible. As a result, a large fraction of submitted papers are reviewed under different conditions than the rest. If we continue with the current policy, arXiv will be a death warrant for double-blind ACL reviewing.
I am glad to report that this Saturday, Min and I participated in the winter ACL executive meeting, where this issue was discussed in depth. The big question is how to formulate the policy moving forward. The ACL exec is planning to carefully study members’ feedback on this question, and we hope that you will use this forum to express your opinions. For now, I would like to share with you a very thoughtful informational piece on this topic written by Marti Hearst, the Vice President of ACL.
Executive summary: Numerous studies have shown that single-blind review leads to bias in favor of certain types of researchers over others when the objective merit of the work is held constant. All ACL conferences and most workshops make use of double-blind reviewing for this reason. The rapid rise in popularity of online pre-print servers such as arXiv, while presenting the community with many benefits, has the potential to threaten the double-blind review process. We as a community need to make a policy decision about how to handle public pre-posting of papers that are under review at ACL conferences.
In more detail:
The arXiv online pre-print repository service has become enormously popular for distributing NLP research. Many members of the ACL community subscribe to the daily mail alert listing the latest papers that have been posted on the service in order to keep up with the most recent research.
ArXiv has the advantage of making the full text of articles easily available in open access form, and allows authors to timestamp “technical reports” of as yet unpublished work. Authors can post work-in-progress versions of papers on arXiv, receive feedback, and post revisions of those papers. If a version of a paper is eventually accepted for publication at a conference or journal, an author can indicate the citation information and the DOI in specific fields. (However, in many cases authors unfortunately fail to go back and update their paper’s arXiv entry with this information.)
The main challenge that arXiv posts for conferences is the threat to double-blind reviewing.
Many studies through the years have shown the biases that result from reviewers knowing information about the authors of the papers. The latest in a long line of such studies was ironically posted on arXiv itself and has been circulated among those who discuss conference reviewing. Its title is “Single vs. Double Blind Reviewing at WSDM”, by Andrew Tomkins, Min Zhang, and William Heavlin. https://arxiv.org/pdf/1702.00502.pdf
This paper recounts how WSDM 2017’s program chairs conducted a controlled study in which each paper was reviewed throughout the process by 4 reviewers, 2 of whom were assigned to a double-blind, and 2 to a single-blind condition, through the processes of bidding, reviewing, and entering scores. Analyzing the results, the authors found a strong biases.
Single-blind reviewers showed measurable preferences for bidding for papers from “top” institutions over double-blind reviewers. Once papers were allocated to reviewers, single-blind reviewers were significantly more likely than their double-blind counterparts to recommend for acceptance papers from famous authors (p < .0006) and top institutions (p < .0004) and were significantly more positive about those papers. (Top institution = top 50 CS departments; famous author = at least 3 WSDM papers and 100 DLBP papers).
In this case, the authors did not find differences in bidding or reviewing behavior with respect to gender of authors; other studies have shown varying effects in terms of gender bias, some finding strong effects and some finding weak to no effects. They note however that their community does not have fixed conventions for “first authors” and so they count the presence of just one female author as signifying a female author for the entire submission. (NB: I think they should have measured female-majority papers.)
I want to emphasize again that many studies across different fields and several decades have found similar bias effects, and often gender bias effects as well; the Tomkins et al. paper provides a nice overview of several such papers.
I communicated with Andrew Tomkins, asking if since release of this paper he’d heard any responses about how other entities are dealing with arXiv. I would paraphrase his response as saying that other conferences are still feeling their way with this problem. For those wondering: “What about journals? They are single-blind review.” Tomkins et al. point out that the careful selection of reviewers, the opportunities for revision, and the lack of competition for limited slots in a journal potentially mitigate the concerns in journal reviewing.
In this post I am not proposing a solution, but rather am making a point. As a community, we need to consider carefully the relationship between posting to arXiv and the benefits of double-blind reviewing, and decide where we want to go from here.
ACL 2017 
Double-blind is ideal, but might dim lighting be almost as effective as blindness? In other words, the Tomkins study confirms that it’s a problem if reviewers are shown authorship. But we don’t currently show them the authorship.
The fact that a paper is posted on arXiv doesn’t mean that its reviewers actually know the authors. A determined reviewer could clearly discover the authorship if the paper is on arXiv — and often, even if it isn’t. But reviewers are on their honor not to go sleuthing. (At least as I see it – they should probably be told that explicitly.) Even if a reviewer has seen the paper on the arXiv feed, they may not remember the authors.
To find out, we could add a question to the review form. Just *ask* the reviewers whether — as they reviewed the paper — they were fairly sure who wrote it. And whether they knew because of arXiv vs. other clues.
LikeLike
I think at least half of the students in our group either receive email push notifications from cs.CL or are in the habit of checking it online every day. The volume is low enough that you can reasonably skim every abstract and title, so all arxiv submissions get de-anonymized with or without any sleuthing.
Most of the value of arxiv is in seeing things right as they come out—if members of the community are not allowed to check it for fear of de-anonymizing one of their reviews, why should anyone bother to use it at all?
LikeLike
@Daniel Beck’s comment 10 below: To clarify, my suggestion of adding a question to the review form was not to reduce bias. It was just to gather info. The post by Regina and Marti assumes that reviewers generally know the authors of arXiv papers, and do not know (or suspect) the authors of non-arXiv papers. To what extent are these things currently true?
LikeLiked by 1 person
If arXiv does turn out to be a problem, what should we do? Prepublication is helpfully increasing the rate of scientific communication — so we should be reluctant to ban it! But notice that the problem isn’t the papers, just the names. So the minimal solution is to make prepub anonymous.
We would need the help of arXiv, or another versioning prepub server such as openreview.net. The server would need a new option that allows authors to temporarily mask their identity, revealing it in a later version (e.g., upon publication) or after a time limit has expired. ACL could then set a policy that any prepublished version of a submission must have been anonymous.
Anonymous arXiv papers would still be publicly available, advertised in the daily arXiv feed, and citable. Alas, under present practices, a citation like Anonymous (2017) can’t be automatically updated when the names are revealed; rather, a human or a citation indexer would have to follow the URL to get the latest version.
p.s. As it happens, there is another good use for anonymous prepublication: it could resolve the impasse over supplemental material in double-blind journals like TACL. Our community’s page limits are meant to keep paper sizes manageable for authors, reviewers, and readers. However, good scientific practice demands that authors fully document their methods and results. Many conferences have therefore set up an escape valve by allowing supplemental documentary material, which most readers won’t read and reviewers are not required to review. But TACL doesn’t want to publish unreviewed material, so they currently disallow supplements altogether. I think the solution is to use the citation mechanism. Authors are obviously free to cite any work, including their own work. So if they could — without breaking anonymity — cite an appendix that they’d self-published on arXiv, the problem would be solved. Indeed, many details of a project are ordinarily provided by citation — you don’t have to explain the LSTM architecture if you can cite Hochreiter & Schmidhuber, and you don’t have to include a dataset if you can give its URL. The point is that these citations have always acted exactly like supplemental material: reviewers are not required to follow citations nor to evaluate the work therein, though they might choose to (and if they do, it could affect their opinion of the submitted paper).
LikeLiked by 5 people
Great points, Jason! The study by Tomkins et al. is a great start, but let’s not stop there. There are many different variants of semi-blind reviewing to try and evaluate scientifically. The sooner we move forward to rigorously study this issue, the sooner we can hope to have a solution. Just blaming arXiv, which many of us obviously find immensely useful, or proclaiming the death of blind reviewing, which obviously is important to keep our field fair and healthy, will lead us nowhere.
LikeLike
I am a little confused by the arXiv’ing declaration this year. In the instruction, an arXiv URL is provided, but this reveals author identity with a very low expense: just a click.
From my point of view, I prefer previous conferences’ practice: declaring arXiv preprint, but keeping anonymity in a surface form, i.e., no author information in the review form or in pdf. If a reviewer wants to know the author information, he/she has to search the paper actively, which should not be done by reviewers’ honor.
Another concern is preprinting papers after the submission due but before notification; then a reviewer subscribing arXiv mailing lists may be notified the authors of an under-reviewing paper. This bad practice could be banned without hurting much scientific communication: if the paper is accepted, there’s little difference in earning credit of the work; even if not, the actual review process only lasts a couple of weeks.
As discussed, it is virtually impossible to achieve full anonymity because of the research topic, the writing style, or even the drawing style. Even in double-blind reviewing venues, there are bifurcations in the instructions of self-citations: some encourage third-person citations, whereas others encourage anonymous citations.
Therefore, I believe the goal of double-blind review is to increase the difficulty of revealing identities, as well as to keep a balance between the unbiasedness of the review process and the efficiency of scientific communication (and maybe proper credit assignment to researchers).
LikeLiked by 1 person
Personally, I think it would be good practise to only publish on arxiv after a paper has been peer reviewed and accepted to a conference. Maybe ACL could encourage this. IMHO, double blind reviewing is an essential ingredient of academic integrity and has shown to reduce (unconscious?) bias.
LikeLiked by 4 people
When writing this, I mainly thought about bias towards academic institutions (i.e. their “reputation”). However, there is also evidence that female authors are disadvantaged in STEM subjects. See:
“The Matilda Effect in Science Communication: An Experiment on Gender Bias in Publication Quality Perceptions and Collaboration Interest”
Science Communication
In this study, authors Silvia Knobloch-Westerwick, Carroll J. Glynn, and Michael Huge asked graduate students in communications programs across the country to evaluate conference abstracts, and found that students gave higher ratings to identical abstracts submitted with male author names, associating them with greater “scientific quality.” Female-authored abstracts on topics such as “parenting” and “body image” (rather than “politics”) were perceived as gender-appropriate and rated more highly than abstracts on gender-neutral research topics. Male-authored abstracts that related to topics deemed “masculine” such as communication and political participation or democracy and journalism (as opposed to studies of gender, infancy, and parenting) were judged to have the greatest “scientific quality.” Graduate student evaluators were also most enthusiastic about potentially collaborating with the male authors of abstracts on “masculine” topics.
LikeLike
I agree with Jason that if arXiv allowed temporarily hiding author names and only revealing them after acceptance, that would be a great solution. I don’t know if arXiv would help with that though.
In case it’s not possible, I’d like to make a remark that I already made in a previous post of this blog, and thus is partially copy-pasted content, but it’s most relevant here: I don’t agree with the widespread position that blind review is a boolean, all-or-nothing thing. This is implicit in claims that arXiv will be the “death” of double-blind, and I have seen people say that either non-anonymous prepublication is forbidden or we should dispense with blind review altogether, which doesn’t make sense to me.
I think this kind of all-or-nothing position fails to consider one of the advantages of blind review. Blind review is not only about preventing positive bias when you see a paper from an elite university, it’s also about the opposite: preventing negative bias when you see a paper from someone totally unknown. Being a PhD student from a small group in a little known university, the first time I submitted a paper to an ACL conference I felt quite reassured by knowing that the reviewers wouldn’t know who I was.
In other words, under an arXiv-permissive policy like the current one, authors still have the *right* to be reviewed blindly, even if it’s no longer an obligation because they can make their identity known indirectly via arXiv+Twitter and the like. I think that right is important. So the dilemma is not a matter of “either we totally forbid dissemination of the papers before acceptance in order to have pure blind review (by the way, 100% pure blind review doesn’t exist anyway because one often has a hint of whom the authors may be, and this is true especially of well-known authors) or we throw the baby out with the bathwater and dispense with blind review altogether”. I think blind review should be preserved at least as a right for the author (as it is know), and the question is whether it should also be an obligation or not.
LikeLiked by 1 person
The solution seems to be obvious – strongly encourage ArXiv to allow anonymous submissions (or switch to an alternative platform which does). I believe banning ArXiv, or any free public sharing, for papers under review would be a step backwards in the current speed of research.
LikeLiked by 1 person
Speaking as an arXiv moderator, anonymous SUBMISSIONS to arXiv would be unthinkable. We have a major problem policing spam and plagiarism, so we are already using a simple reputation system.
A related issue is that the entire philosophy of arXiv is about openness. I’m not part of the arXiv leadership, but I think it will take a lot of convincing to persuade the leadership to allow publication with embargoed authorship (aka anonymous PUBLICATION).
Finally, there is the issue that a good reviewer should check for prior work on the topic, and if our search engines are any good at all, they will serve up the arXiv preprint. So asking reviewers not to search is asking them not to do their job.
For these reasons, I think authors should not post papers to arXiv while they are under double-blind review.
LikeLiked by 1 person
Hi Tom — Embargoed authorship would solve the other problems you mention. Can you explain a bit more why it would go against the arXiv philosophy?
arXiv is a vehicle that *allows* people to be more open in the way they do science. It doesn’t *require* anyone to publish their work before they wish. So I’m not sure why it should require them to reveal their names publicly before they wish.
Many people who like openness would argue that anonymous speech promotes openness (and that it is an essential component of free speech): https://www.eff.org/issues/anonymity . As evidence, they’d only have to point to your final sentence, which says that because we don’t have anonymity, we’d better avoid some openness! 🙂
Does it help if the embargo is temporary (see comment 2 above)?
LikeLike
What is crucially so good and important in arXiv and prepublishing so that discarding it will jeopardise a good research?
I’ve already written about this on another topic, but my points are:
– I can publish my results on arXiv whenever I want, whatever I want, it doesn’t have to pass the reviewing process;
– people are then obliged to cite me, although I didn’t pass the reviewing process;
– why I would ever try to publish something at a conference (journal??), especially at a “difficult” conference such as ACL?
Another point is jeopardising double-blind reviews — I agree that this policy also prevents negative bias toward the “non-important” persons/institutions, but it can still be unfair and biased, albeit partially and implicitly.
Say, there are 1000 submissions and only 200 can be accepted. Half is from top groups, half is not. If the review is completely* blind, the accepted papers will consist of, say, 120 top authors and 80 not-top authors. If the top authors have a positive bias through arXiv and the other authors are anonymous, then 160 top authors will be accepted because they got better reviews and, ha, not too much place left, only 40 others.
*of course, sometimes it is difficult, but those are completely different factors
LikeLiked by 1 person
Echoing my comment on the previous post, I think anonymous submission is intended to avoid the direct and immediate bias a reviewer gets when the author’s identity is thrown in their face. It’s not intended to prevent the reviewer from going on a search to discover the author.
As for prepublication, I agree that holding a paper for the few weeks of the review process is not a big deal. But what if the paper is ready (or can be made ready) 6 months before the review process? Or if it gets rejected and needs to wait a full cycle for a revised version?
So here’s a more radical idea: move ACL to a year-round, journal-style review process. We’re going in the direction of journal-style reviewing anyway, with more of an emphasis on post-review discussion, and there are voices that want even more back-and-forth between authors and reviewers (this sentiment was clearly articulated at the last NAACL business meeting). Personally I’m not very fond of this extended dialogue in conference reviewing, because without actual paper revisions, people tend to talk about what could be in the paper rather than what is in the paper. My position at the NAACL business meeting was that we don’t need extended dialogue in conference reviewing because we already have a venue for that: TACL. And TACL, with its year-round reviewing and fast turnaround, also offers a solution to the quandary of whether to hold publication until the next review cycle comes along.
So why not move all ACL reviewing to TACL? One nice thing about the current conference review process is that the community pulls together and puts in an enormous amount of effort into writing and reviewing in a short period of time. This definitely is a catalyst for large quantities of work, though the hurried nature of the process probably affects quality. If we move away from the current model to year-round reviewing and publication, we’ll need to direct our energies to TACL instead. It might be too late to ramp up in time for 2018, but we could decide that for 2019 there will be no ACL reviewing: set a deadline on TACL acceptance (not submission), and any paper accepted by that date will be presented at ACL. We’ll get better paper and better reviewing.
LikeLiked by 4 people
Maybe all of the hand-wringing about arXiv is a symptom of having invested too much importance in the peer review process generally.
The story that I was presented when I was starting out was basically “research reports are meaningless and non-credible until they pass peer review, at which point they become True”. But of course this is not how things actually work—plenty of research that is unreproducible or incorrect makes it through peer review. And a paper can provide value to the community before it’s been through the review process—this is the point of arXiv!
What we’re actually trying to decide when we pick up a paper is “is it worth my time to read / use / base my next project on this?”. Or “should I hire the author?”. In practice having made it through the review process is just one feature to use when making this decision, which also involves questions like “Do I recognize the author? Is there published code? Is there already followup work?”.
So the question of whether to allow arXiv / single blind review is basically: does the value provided to the community by arXiv outweigh the harm that results from decreasing the usefulness of the peer review feature (by making it noisier and more biased)?
Bias is worrisome. But this is the kind of thing we can correct for explicitly: if we know that being famous or being at a top institution results gives you half a point for free in a single-blind process, we could try handicapping people in those categories by half a point.
W/r/t noise, I guess my claim is that the stamp of approval was never that great a signal to begin with, and a little more noise won’t hurt. arXiv provides a huge amount of value, and I think the bar for restricting people’s ability to use it should be quite high.
LikeLike
To amend this slightly:
I think I meant “faith in” rather than “importance of” peer review above. We’re an engineering discipline and we have common tools that make it easy to reproduce things. We know the conference review process is noisy, but this is OK—type I errors are ultimately not a big deal. The fact that arXiv has not totally destroyed the field is an indication that even our formal conferences might benefit from a more inclusive acceptance standard (as Carlos GR has suggested below).
And again, I think it’s important to emphasize that double-blind review is *not* totally unbiased. It is often pretty easy to approximately de-anonymize papers from prominent research groups using some combination of subject matter, regional spelling conventions, figure style, and the distribution of footnotes and section symbols. (As Jason has suggested, it would be useful to try to quantify just how easy this actually is.) This means that all the pernicious effects of moving from double- to single-blind reviewing are *already present* to some extent in the existing review process.
So it’s all about tradeoffs, and it may well be that—even for researchers who are negatively impacted by single blind reviewing generally—the increased visibility enabled by arXiv makes this particular tradeoff worthwhile.
LikeLike
What exactly is this “value” that arXiv brings to the community? If a platform helps to accelerate science but ends up imbuing the whole process with numerous biases, is that a “valuable” thing? Is it completely out of question to “slow down” the process of scientific discovery so we can at least make it fairer for unprivileged people?
Double-blind reviewing is not perfect but it still helps reduce bias. Personally, I think fairness is important enough to “sacrifice” science acceleration if that’s the case. If that means we have to go to the extreme of banning arXiv then I am all for it. Of course, there is still a lot of discussion to be made so there might be compromises to be discovered. But sometimes I feel the community is too adamant on the idea of “accelerating science” without even questioning the consequences of it.
Finally, some people made interesting suggestions to help reducing bias in the reviewing process, such as handicapping or making explicit if you think you know the authors of a submission. But, as in double-blind reviewing, there are also not perfect and are prone to some degree of subjectivity (how to define famous vs. non-famous, for instance?). I can see them being useful as additional “layers” in the reviewing process but I am still a bit skeptical of them as *alternatives* to double-blind reviewing.
LikeLiked by 3 people
Exactly my thought: what is this essential “value”? What we would be stripped off from without arXiv?
Right now, I see only introducing some kind of anarchy on many levels. Not only double-blind reviewing.
For example, one of the essentials of each conference: *previously unpublished work*.
Furthermore, until now, for me the research process looked like this: I have some ideas. Then I find suitable data, think about suitable experiments, run them and see what happens, run some more depending on the results, maybe finding more data, etc., and eventually if there is an interesting story about al that, I try to publish it on a conference.
Now with arXiving, I can simply stop at the stage “I have some ideas”, write a few pages about them, “publish” it on arXiv, and there you go.
But then, why only arXiv? I can publish my things on my Linkedin profile, Research gate profile, personal web page, etc. etc.
Of course, peer reviewing is far from perfect, and not all accepted papers are good / all rejected papers are bad, but I really don’t see how arXiving is overcoming this.
And speaking about the reviewing, ther are more and more reviewer comments “you should have cited this and this arXiv paper”, sometimes the paper being uploaded only few days before the submission deadline.
For all these reasons, for me arXiv does not look like a great thing which accelerates research (whatever that exactly means — there are several good NLP conferences each year), it only looks like a chaotic factor on many levels.
LikeLiked by 2 people
Interesting discussions.
I believe suggestion of citing very recent arXiv papers can be discouraged by review instructions.
Preprinting papers provides a chance of fairer competition between established researchers and newcomers because even double-blind review is biased and highly random:
1. There have been examples where two papers were essentially the same thing, but the paper from a famous group was accepted, whereas the other was rejected. It’s not fair. arXiv at least provides a way of disseminating research on a fairer basis.
2. In other cases, a good paper is rejected by incompetent reviews; then it would take a full submission cycle before the paper can be accepted (and thus announced if preprint mechanisms are not allowed). 0.5–1 year seems to be too long for fast-developing areas.
Regarding idea-only papers: it’s researchers own honor to make their paper substantial and rigorous. (Yes, maybe fewer experiments but main conclusions should hold with rigorous experiments and data.) If one preprints idea-only papers or fundamentally incorrect papers, he/she actually loses credit. It’s also very easy to figure out those idea-only void papers, so it’s not a problem to the audience either.
LikeLike
I agree with your comment, point 1 is something that worries me as well. Banning arXiv altogether could put non-famous groups in significant disadvantage when doing research on highly “scoopable” work.
Regarding point 2, I would like to add that a good paper often doesn’t even need incompetent reviews to be rejected. ACL acceptance rates float around 20-25%. I’d say there are more good papers than that, so it’s inevitable that some good material is rejected. We’ve probably all seen essentially the same paper rejected from a conference and accepted at another (not necessarily worse) one, not necessarily by incompetence (although it happens sometimes) but due to the normal variance/luck factor with such demanding acceptance rates.
LikeLike
To me there seems to be an analogy to business innovation problems, i.e., of course you want to be the first and you want to be perfect at the same time. In business, you can file a patent to your idea long before you develop the final product, although only bringing the product to the market will show its real value. There is no point to register the idea only after the market approved the product is great – by then, somebody else might have done it already. In the academic case, peer review is some sort of a “minimal quality norm” stamp, without which it is difficult to enter the market. The number of citations is pretty much the number of “products sold”. And ArXiv is then the patent claim combined with some sort of “crowdfunding request” (for citations). If the “product” is not any good, let it perish – papers probably should not be widely cited if they appeared only on ArXiv and nowhere else within, say, the next 12 months. But when the idea is accepted in the community after the long and rigorous process, ArXiv provides a reference to when and from whom the idea really appeared. Maja has a fair point that it can be any other medium – ArXiv just has a suitable format for this and is widespread in the community.
This might be particularly relevant because there is a huge discrepancy between the amount of resources available in different teams. Experiments in small labs may take months simply due to sharing the already limited computing capabilities, while the prominent & industry players can reproduce the same within few days, and in a fraction of time test the same idea on a much larger data sample, producing a more persuasive paper to the same conference deadline as a result. For similar reason, by the way, it is not the same delay for everyone when reviewers ask for additional experiments. Hence, I think it is meaningful to use ArXiv for saying ‘Here is my idea, based on these preliminary results it seems to work and it might be useful to you. Now give me half a year to test all the parameter configurations because I just have a CPU, or do it yourself if you want, but don’t pretend you thought about it as a first one’.
As an another (minor) point, from a purely subjective perspective I prefer a continuous feed of research ideas and experiments through the year, rather than trying to get up to date with the entire ACL proceedings in one week and spending the rest of the year ruminating over it and wondering where others are taking it in parallel. In this respect I would agree with the suggestion of Ron Artstein above.
LikeLiked by 1 person
I do not understand why is the issue still being discussed.
The arXiv issue has been shed light for years.
The first time I have known this is 2015.
At that time I do believe the society will make effort to keep diversity and reduce bias.
What have been changed since then?
We should be honest with ourselves.
We have made choice: we don’t like double-blind, while we emphasize that double blind is escessary.
LikeLike
Why not allow putting papers on arXiv as long as they are anonymized too. That allows
sharing the research without delay and also preserves anonymity. One can always use an
anonymous email address for answering questions and names can be added (or not added)
after the decision has been made.
LikeLike
We need just two components: anonymous publications on arXiv and ability to update anonymous citations in already-published conference papers.
LikeLiked by 1 person
1) fair bias — anybody can publish anything, there is no bias in favor of famous authors.
Still, for the same reasons that the famous authors can get better conference reviews, the famous authors on arXiv are likely to be cited more than non-famous.
2) competition — to let everybody know who was the first one.
What if two, three or more groups have the same ideas at the same time? (which happens quite often)
How arXiv would help there?
3) enabling publications of interesting preliminary/incomplete results by authors with no enough resources
This one is fine. Some authors obtain nice preliminary results which could very probably not pass the reviewing process (or it has been rejected already), but they do not have enough resources to make the story complete, and they don’t see that they will be able to do in the near future.
Then, why not publish it to arXiv and, if they at some point can carry on with this — great, if not maybe somebody else will be able to do that, but then the other group should cite the arXiv paper.
So yes, you got me there, this one is convincing!
Even though of course, if the arXiv publications is from “small guys” and then some “big guys” get better results and publish it on a conference, there is still possibility that the big guys will be cited more, but life is not perfect. This point makes sense in any case.
Btw pre-penalising “famous groups” in any manner just for being famous is also not fair.
So, I can agree that arXiv can be used for 3), and in a way it covers the points 2) and 1)
But:
Why on Earth to publish *identical* (or almost identical) papers on arXiv and on a conference?? If one has something good enough for a conference, forget arXiv and go for conferences.
There are enough of them. Plus a number of specialised ones. Plus workshops. Plus TACL which accepts submissions every month.
Unfair reviewers? Yes, it happens. But there are many conferences and workshops.
Low acceptance rate on some conferences? Sure. But there are many conferences and workshops.
arXiv should be a way for alternative publishing.
Not to be mixed up with conferences — otherwise too much anarchy, chaos and hyperproduction will take place.
LikeLike
> Why on Earth to publish *identical* (or almost identical) papers on arXiv and on a conference?? If one has something good enough for a conference, forget arXiv and go for conferences.
This argument would work if you knew in advance that you have “something good enough for a conference”, but then, if you could be so sure in advance, why reviewing at the first place? More often than not, you aim high with the submission and gradually decrease, getting “maybe”s and “almost”s, until the paper eventually matches the venue level and the reviewers’ taste, which can easily cause a delay of one year. If you want community feedback on your work, in which you believe to be of interest, why having to stay limited to the sparse comments of reviewers exclusively in quarterly intervals? Either the work is mature enough and gets popular, or it is not, and then you crowdsource some useful criticism to improve it.
Moreover, submitting to ArXiv is definitely cheaper than attending C-level conferences and workshops, being helpful for students and researchers from more remote regions of the world.
LikeLike
For conferences in social computing, each conference has an explicit policy towards arXiv papers. Most of them allow submission of previously arXived papers, and a few don’t. Having an explicit policy would make authors’ options clear before they submit. For areas that are changing quickly, arXiv provides a great service to authors and readers alike as it cuts down publication time by many months. Perhaps, reviewers can be instructed not search for papers on arXiv before reviewing — though this has its own problems too.
LikeLike
I just got a note from Andrew Tomkins; they’ve posted a new version on arXiv with updated gender results; his team manually annotated all the names with gender (except for about 3% which they could not determine).
They then defined female papers in three ways: any female author, first female author, and majority female author. No significant bias effect was found for any of these conditions.
Thanks to Andrew and his co-authors for following up with this analysis!
LikeLike
Thank you for starting this much needed discussion. Reading through the previous comments, I have tried to summarize a few key points for and against pre-publication and added my own opinions below (a long post will follow).
A1) Biases
If we, as a community, are principled about fairness then presented with the current evidence (including the WSDM study) of existing biases, we should not even discuss this subject. We should simply not allow pre-publication. However, to be consistent, we should make the reviewing process fully double blind – the area/program chairs should not see the author names as well (like in the WSDM experiment).
If there are doubts about the WSDM study, we should perform a controlled experiment ourselves and see the effects.
Imposing penalties raises far more important biases than it solves: how do you define famous/non-famous, not all people suffer from biases or have them with various extents, etc.
A2) Asymmetry
The benefits include immediate publication, claiming an idea, de-anonymization if this favors the authors, racking up citations before actual publication.
So, why would someone *not* first publish on arxiv if the community (tacitly) allows this? Some authors prefer not to do this because they fear biases against them caused by de-anonymization, others do not think its ethical (including myself) and some do not care/know about this option.
Everything favors a certain group, so I think the benefits that come with arxiv submissions should be balanced out, for example with a ban on *ACL submissions. The authors who want to publish on arxiv right away can still submit to journals, conferences that accept pre-publication or can use a part of the results in a submission (<25% threshold).
A3) Citing arxiv
As this is un-reviewed work, I think there should be explicit guidelines that should prevent making claims based on or comparing to arxiv-only publications.
F1) Giving everyone a chance to publish
If there is no quality check on the papers, then in the deluge of papers readers will devote more attention and ultimately citations based on author names/institutions or marketing in various forms: over-statements, catchy method names/titles, social media publicity and through press releases/press offices. Everything we will end up with the 'rich getting richer'.
F2) Science acceleration
Honestly, are we seeing a uniform stream of papers coming through arxiv? Actually, most papers are conference submissions, which are posted to arxiv soon after submission.
How many of these papers are actually revised after feedback? This is easy to check on arxiv and having access to lists of *ACL submissions and acceptances. There is a quick analysis for NIPS papers already, which we can easily replicate: http://nlpers.blogspot.com/2015/10/a-small-observation-for-prepubs-on-arxiv.html
F3) Reviewers not searching for the paper online
I think we overly assume the fact that reviewers do not search related work online. But do we really want these reviewers to rate our submissions? How often do we see papers that do not compare or cite (with or without knowledge) a very relevant paper of ours? I think we should encourage reviewers to be diligent and search for previous and potentially missing related work.
I do not think there is a good solution suitable across all CS communities. I agree that publishing on arxiv is useful in areas where publishing takes long (i.e. years), an idea can be taken without much effort (like a proof in theoretical CS) or when reviewing is single-blind. Most NLP papers have thorough experiments, sometimes using new datasets and resources, so in most cases there is arguably not that big of a hurry for pre-publication since there is no big risk for ideas to get scooped.
Ideally, there any many things we could have in publishing, but pragmatically: we all need peer-reviewed publications for applying for jobs and grants with our colleagues from other disciplines, the conference program and space is limited, conference ratings need a basis to be computed on (hence the acceptance rate), etc.
I agree that a rejection is not just a few weeks delay, its on average a 4 month delay while impact, novelty and citations can be lost. We should strive to do the best and most objective job we can in the reviewing process rather than adopting side solutions that are proven to favor certain groups.
LikeLike
double-blind review is quite good , which I hope ACL can keep it. For the papers uploaded to Arxiv, of course, it is known who are the authors, so the Arxiv papers shall not be considered as eligibal for double-blind review. so it shall be the authors choice whether to upload the papers to Arxiv then submit it to a non-blind review journal / proceedings, or choose to keep their works for some time and submit to ACL for doubloe-blind review.
LikeLike
double-blind review is quite good , which I hope ACL can keep it. For the papers uploaded to Arxiv, of course, it is known who are the authors, so the Arxiv papers shall not be considered as eligibal for double-blind review. so it shall be the authors choice whether to upload the papers to Arxiv then submit it to a non-blind review journal / proceedings, or choose to keep their works for some time and submit to ACL for doubloe-blind review.
LikeLike
Thank you for starting this much needed discussion. Reading through the previous comments, I have tried to summarize a few key points for and against pre-publication and added my own opinions below (a long post will follow).
A1) Biases
If we, as a community, are principled about fairness then presented with the current evidence (including the WSDM study) of existing biases, we should not even discuss this subject. We should simply not allow pre-publication. However, to be consistent, we should make the reviewing process fully double blind – the area/program chairs should not see the author names as well (like in the WSDM experiment).
If there are doubts about the WSDM study, we should perform a controlled experiment ourselves and see the effects.
Imposing penalties raises far more important biases than it solves: how do you define famous/non-famous, not all people suffer from biases or have them with various extents, etc.
A2) Asymmetry
The benefits include immediate publication, claiming an idea, de-anonymization if this favors the authors, racking up citations before actual publication.
So, why would someone *not* first publish on arxiv if the community (tacitly) allows this? Some authors prefer not to do this because they fear biases against them caused by de-anonymization, others do not think its ethical (including myself) and some do not care/know about this option.
Everything favors a certain group, so I think the benefits that come with arxiv submissions should be balanced out, for example with a ban on *ACL submissions. The authors who want to publish on arxiv right away can still submit to journals, conferences that accept pre-publication or can use a part of the results in a submission (<25% threshold).
A3) Citing arxiv
As this is un-reviewed work, I think there should be explicit guidelines that should prevent making claims based on or comparing to arxiv-only publications.
F1) Giving everyone a chance to publish
If there is no quality check on the papers, then in the deluge of papers readers will devote more attention and ultimately citations based on author names/institutions or marketing in various forms: over-statements, catchy method names/titles, social media publicity and through press releases/press offices. Everything we will end up with the 'rich getting richer'.
F2) Science acceleration
Honestly, are we seeing a uniform stream of papers coming through arxiv? Actually, most papers are conference submissions, which are posted to arxiv soon after submission.
How many of these papers are actually revised after feedback? This is easy to check on arxiv and having access to lists of *ACL submissions and acceptances. There is a quick analysis for NIPS papers already, which we can easily replicate: http://nlpers.blogspot.com/2015/10/a-small-observation-for-prepubs-on-arxiv.html
F3) Reviewers not searching for the paper online
I think we overly assume the fact that reviewers do not search related work online. But do we really want these reviewers to rate our submissions? How often do we see papers that do not compare or cite (with or without knowledge) a very relevant paper of ours? I think we should encourage reviewers to be diligent and search for previous and potentially missing related work.
I do not think there is a good solution suitable across all CS communities. I agree that publishing on arxiv is useful in areas where publishing takes long (i.e. years), an idea can be taken without much effort (like a proof in theoretical CS) or when reviewing is single-blind. Most NLP papers have thorough experiments, sometimes using new datasets and resources, so in most cases there is arguably not that big of a hurry for pre-publication since there is no big risk for ideas to get scooped.
Ideally, there any many things we could have in publishing, but pragmatically: we all need peer-reviewed publications for applying for jobs and grants with our colleagues from other disciplines, the conference program and space is limited, conference ratings need a basis to be computed on (hence the acceptance rate), etc.
I agree that a rejection is not just a few weeks delay, its on average a 4 month delay while impact, novelty and citations can be lost. We should strive to do the best and most objective job we can in the reviewing process rather than adopting side solutions that are proven to favor certain groups.
LikeLiked by 1 person
I have a few replies/minor comments, but let me start with saying that I think that “banning arxiv” is a non-starter. I’m a fairly loud anti-arxiv-er, but I really think it’s basically impossible.
I have a few comment about “anonymous arxiv” and then a second proposal that I’d like to get feedback on, and then finally some replies to other things people have said.
Anonymous arxiv: This is probably my prefered route, but Yoav Goldberg made a pretty good case on twitter that this doesn’t work that well (see thread beginning at [1]). I post a paper to arxiv anonymously, then either tweet it out myself or send it to famous friends who then tweet it out. This is also very logistically complicated. I know that ICLR is going to be trying to figure out if arxiv can be done anonymously; it might be worth joining forces with them (I can put in touch).
An alternative might be to think about separating the “stamp of approval” component from the “public advertising” component of conference acceptance. Currently conferences do both simultaneously, but I could imagine a model in which you’re allowed to post to arxiv during review, but if you do, then you do not get a presentation slot at the conference. That is, you’ve chosen to do the advertising on your own, so all the conference will do is give a stamp of approval. This does NOT address @Gómez-Rodríguez’s point about arxiv being helpful to people at less-well-known institutions, but perhaps this could be balanced somehow (i.e., PCs can have discretion to give talks to papers that might not have visibility because of whatever, but I can also imagine this devolving).
The other issue with this model is that it *will* bias the reviews of people who choose to arxiv, but because I think that banning arxiv is a non-starter, I think we just have to accept that. The nice thing about this model is you don’t have to change anything in the system, and basically if you’re famous enough that you think having your paper out on arxiv earlier is better than giving a talk, then you get to make that choice.
Minor comments:
Using web search (@Eisner, @Dietterich): Like Tom, I think it’s part of my job as a reviwer to look for and read related work. Telling me I can’t do a web search for keywords from a paper doesn’t work for me.
The SGD argument (@Eisner, others): There’s a claim that prepub speeds up research, but is there evidence that this is true? I blogged a while ago [2] some weak evidence it might not be. Also, if we’re going to go with the SGD argument, it’s important to point out that SGD works when you have /unbiased/ samples from the gradient, so the analogy breaks.
(Although it hasn’t come up here, there also the idea of public comments and how it intersects with blindness; I highly recommend Izabella Laba’s post [3] on this topic, h/t Suresh Venkatasubramanian)
[1] https://twitter.com/haldaume3/status/832128093695270912
[2] http://nlpers.blogspot.com/2015/10/a-small-observation-for-prepubs-on-arxiv.html
[3] https://ilaba.wordpress.com/2016/04/10/arxiv-comments-and-quality-control/
LikeLike
Follow-up here for the record: Kamalika Chaudhuri convinced me that my proposal about not getting presentations breaks for the following reason. People from smaller groups really need to get their work out sooner in competitive environments, and if you disallow them from presenting, it’s a double whammy.
LikeLike
The current policy forces the area chairs to compare a double-blind submission with a (possibly) single-blind submission. I wonder if we can address the fairness issue by putting each set of papers in a different pool (e.g., three tracks in each area: long papers, short papers, arxiv papers). The bias will still be there for papers in the arxiv pool, but you will only suffer/benefit from the bias if you decide to submit your paper in the arxiv pool.
I prefer anonymous arxiv submissions though, if arxiv is willing to support them, because it is less work for the area chairs.
LikeLike
I won’t argue over the benefits of rapidly publicising research results on arxiv (these must be peer-reviewed to count as citable work of course), these are clear. At the same time, it is more than clear that it corrupts the double-blind reviewing process. What would happen if everyone posts their papers on arxiv? Then we’re back to a form of single blind reviewing. It is very important for me to be part of an organisation that respects and tries to preserve the equal opportunities for publishing in top venues for its members, since these publications are the basis for getting tenure, grants, a position as a postdoc/lecturer as well as allowing “university social mobility” (i.e. moving to better universities). When it comes to choose between equal opportunities and rapid publicising of “research results”, we (as a community) should always stick to the former.
Therefore, I think that banning eponymous arxiv and workshop papers from *ACL conferences would be a **great** starter. Researchers will still have the choice to rapidly publish a paper on arxiv if they wish to but that will mean the paper cannot be submitted to a *ACL conference. Simply, you can’t have your cake and eat it too.
LikeLike
It confuses me that arXiv is being singled out. ACL *does* allow submissions where a previous version has been
* released as a workshop paper,
* presented as a talk whose abstract/slides/video are easily found on the web,
* sketched on a blog post, or
* discussed in person with other members of the community.
Don’t these also break anonymity?
Indeed, ACL actively encourages such discussions of work in progress. That’s why our conferences have workshops and coffee breaks. For non-CS conferences, in fact, discussing unpublished work is the entire point — publications are what happen in journals. Doesn’t arXiv merely help to expand and democratize such discussions, so that they are not so dependent on chance conversations or the ability to travel?
I have not released much myself through arXiv / blogs / workshops. But I actually feel a bit guilty about that, because I have sometimes benefited from early knowledge of what other people are working on. (Is my idea sufficiently different that I should bother working on it? If so, what datasets and metrics should I use to facilitate comparison? Does the arXiv paper contain citations or techniques that I didn’t know about?)
I do agree that the rise of arXiv culture has had several troubling effects, but it also seems to fill a need. So I would prefer to address its defects more constructively — not by banning it with no alternative in sight, but by designing alternatives that can do an even better job of promoting an open and vigorous scientific discussion in the Internet age.
I’d like to see mechanisms that can draw other researchers into conversation and even co-authorship before the final version of the paper — and which permit revision (as arXiv does) so that no version is ever really final. Our mechanisms should bring the community’s attention to both interesting new proposals and thoroughly supported conclusions. They should reward those who make genuine contributions, rather than rewarding people who are merely well-positioned or canny operators. Such a reward mechanism, clearly, would need to be smarter than just counting acceptances (sort of double-blind at present) and citations (totally non-blind at present).
I’ve heard Michael Nielsen speak on Open Science, and would like to read his book, https://en.wikipedia.org/wiki/Reinventing_Discovery …
LikeLike
I completely agree with the notion that we need to find good solutions, and that it’s possible that there are positive effects of arxiving in our community. I really like and empathize with these comments by Siva Reddy (https://twitter.com/sivareddyg/status/832005160771653633/photo/1). I’m copying it here because I think it’s really important to highlight (thanks Siva):
1. Show your research to potential employers.
2. Remove the stress of waiting until the paper is accepted.
3. Moving on to new problems without feeling like a failure.
4. Make writing an agile process just like software.
5. Get feedback from public — also an opportunity for reviewers to consider critical feedback from public.
6. Copyright your idea without worrying about reduplication
7. Conference as a venue to meet researchers than a place to reward/predict innovation.
I want to highlight these because even though I don’t believe the “make science faster” argument, I do believe all of these points, and have personally experienced them all.
The four items Jason lists (workshop papers, talk that’s online, blog post sketch, in-person discussion) do seem fundamentally different to me in three ways:
1. For some of these media, we would naturally expect citations, and for the submitted paper to build on that work in some way. This is not the case for arxiv duplication. If I see a talk, and cannot find a corresponding paper, I will cite the talk. (Eg., the now well cited Matti Kääriäinen talk in which he gives lower bounds for greedy learning-to-search.) I will always cite workshop papers. If a blog post were sufficiently detailed, I may cite it (e.g., I’ve cited Chris Manning’s post on my blog about F-measure and NER). I probably wouldn’t cite in-person discussion, but if it was in depth enough I’d just co-author the paper. This is not at all the case for arxiv papers, where there’s a “have your cake and eat it too” model, in which we publish papers on arxiv and then don’t cite them in our conference submissions and therefore do not have to extend the work in any way.
2. With the exception of in-person discussion, anyone could build on work from workshop papers, online talks and blog posts, and so while you often think you might know the identity of the authors, you may be wrong. (There have been studies of this, but not in our own field, that show people are often confidently wrong about who the authors are.) Even if not, there is doubt, and my impression is that even a little doubt can go a long way (there must be psych studies of this that I don’t know). If there’s a near identical paper on arxiv, there’s no doubt.
3. Although it’s sub optimal, bean counting exists, and citations to my papers matter, but citations to my blog posts and discussions and online talks don’t. Maybe we need to fix that problem, but I think it’s just as real.
LikeLike
> so that no version is ever really final.
This is yet another problem of arXiv.
Obligatory citing of arXiv papers is seen to be a great advantage — “I marked this territory first, everyone has to cite my arXiv paper about this. This is though not reviewed, this can change who knows how many times, but everyone has to cite it”.
LikeLike
I won’t argue over the benefits of rapidly publicising research results on arxiv (these must be peer-reviewed to count as citable work of course), these are clear. At the same time, it is more than clear that it corrupts the double-blind reviewing process. What would happen if everyone posts their papers on arxiv? Then we’re back to a form of single blind reviewing. It is very important for me to be part of an organisation that respects and tries to preserve the equal opportunities for publishing in top venues for its members, since these publications are the basis for getting tenure, grants, a position as a postdoc/lecturer as well as allowing “university social mobility” (i.e. moving to better universities). When it comes to choose between equal opportunities and rapid publicising of “research results”, we (as a community) should always stick to the former.
Therefore, I think that banning eponymous arxiv and workshop papers from *ACL conferences would be a **great** starter. Researchers will still have the choice to rapidly publish a paper on arxiv if they wish to but that will mean the paper cannot be submitted to a *ACL conference. Simply, you can’t have your cake and eat it too.
LikeLike
Thank you for all the comments! While Min and I will continue publishing other posts in the blog, this discussion is far from closed. In a few days, Joakim Nivre, the president of ACL, will write a guest blog post on this matter. Please continue posting your opinions — they will help the ACL exec to understand the spectrum of opinions in our community.
LikeLike
As area chair, I could see the problems with the current policy at first hand. One of the papers had already been presented—in identical form—at conference X and was available online on the conference Web site. Now, the ACL 2017 call says :
The ACL 2017 conference invites the submission of long and short papers on substantial, original, and unpublished research in all aspects of automated language processing.
However, this is followed by a “Multiple Submission Policy,” which states:
Preprint servers such as arXiv.org and workshops that do not have published proceedings are not considered archival for purposes of submission.
I find it striking that in this section we’re suddenly no longer talking about “publishing” but about “archival.” I would think that the primary goal of preprint servers and workshops is public dissemination, not archiving. I have never seen ACL conferences advertising “archival” either. What is more, I have not seen any explanation of what “archival for purposes of submission” is supposed to mean, because the requirements mention “substantial, original, and unpublished,” but not “unarchived.” So I have the suspicion that “archival” here really means “published.” So the Multiple Submission Policy really says that if you publish your paper on a preprint server or at a workshop that does not have proceedings, it is considered unpublished for purposes of submission.
In order to verify the status of the paper mentioned above, we (well, Min) contacted the organizers of conference X. They responded:
We do not consider work submitted to conference X as published work and do thus not publish any proceedings. So the conference X webpage should not be seen as an archival page.
Here is a quote from the call for papers: “The conference does not publish any proceedings, but accepted contributions will be made available on the conference web page as extended abstracts.”
So the paper in question is clearly acceptable for submission to ACL 2017, as conference X explicitly declares that it does “not publish any proceedings.” I think the discussion here has already shown that the current policy is
not tenable, and this is another good example. For all practical purposes, the paper *has* been published, but since conference X states that it “does not publish any proceedings” it’s OK. The papers are only “made available on the conference web page.” Please, what exactly is the difference between a Web page containing links to the full text of all submissions and proceedings? Suppose another conference has a Web page containing links to the full text of all submissions, but puts the title “Proceedings” on top of the page—a paper presented at *that* conference would *not* be acceptable for submission to ACL 2017.
In my view, this is not only arbitrary and unfair. It is also highly problematic in other respects, because it is no longer ACL who defines what’s acceptable, but the acceptability of papers is now determined by *other conferences’* self-definitions. One can have different opinions about preprints, double-blind reviewing, etc. In any case, however, you must specify clear, unambiguous rules based on objective criteria, which are then followed without exception. Only this can guarantee a level playing field for the whole community.
LikeLike
Is there any thought/practical guideline for submitting datasets/software in accordance with double-blind review?
I reviewed one paper, whose author(s) declared to publish accompanying software upon publication.
During review discussion, one peer reviewer says, because no software is released in the current form, so that he/she cannot know further information.
However, releasing software is highly risky of revealing author identity: IDEs may auto-fill meta-info as comments during coding; the operating system might also auto-fill meta-info to the files. If one wants to fully anomymize his/her code, there would be tremendous extra efforts, not to mention its risk of revealing identities (and thus rejection without review).
I also see some researchers put an anonymous link (e.g., http://anonymous.link) in the submission version. While I fully trust most researchers, in rare cases, however, unscrupulous researchers may mis-use such mechanism, as the monitor between submission and its camery-ready version is vacuum.
So I wonder if there’s—or we can figure out—any guideline for publishing datasets/software, which is also useful for reviewers. For example, my current mind suggests:
1. For dataset/software papers, submitting accompanying resources is mandatory and it’s authors’ duty to make sure the anonymity of the resources.
2. For other papers, declaring a dataset/software is mostly considered sufficient/void(?), but should not be a drawback of the paper as many ACL papers publish neither datasets nor software.
3. Other thoughts?
LikeLike
For what it’s worth, I suspect that the policies that Michael Piotrowski cites (#29 above) have little to do with double-blind reviewing. Similar policies appear all over academic publishing. I think they originated generations ago with print journals — which aren’t traditionally double-blind.
The policies were more about competition, conservation of resources, and credit:
* Academic publishing houses are competing with one another to sell a product. Each journal wants to be seen as essential reading: they all want to publish great novel work, and be the exclusive publisher of that work. A publisher wants the copyright for itself, and doesn’t want to run afoul of some other publisher’s copyright.
* Journal editors don’t want to spend their limited pages and their editorial services on work that is also being published elsewhere, nor spend the time of its reviewers on work that is being simultaneously reviewed elsewhere.
Thus, journals have always had policies against double publication and double submission. Without these policies, authors would have had an incentive to publish the same work in multiple journals, in order to get wider dissemination, as well as a CV that might be more impressive to the casual onlooker.
Of course, authors were still able to publish closely related papers that built on one another, as long as their text didn’t overlap enough to violate copyright. But “how close is too close?” was a matter of editorial judgment about novelty, where the threshold might vary from journal to journal and wasn’t governed (in Michael’s words) by “clear, unambiguous rules.”
The requirement to publish each paper in just one place led to a culture in which authors would first try submitting to the most prestigious journal where they had a chance, and worked their way down the prestige hierarchy until the paper was finally published somewhere.
But this could take years. That wasn’t great for authors or the progress of the field. Fortunately, in the meantime, everyone could talk about their work at conferences, in dissertations, in institutional technical reports, etc. Journals were generally okay with this as long as they had the right of first print publication. The journals just didn’t think of conferences or technical reports as providing serious competition. Glossy, well-edited, selective journals were still the product that libraries and individual academics would subscribe to, and it was still the imprimatur of a journal that authors craved.
In the CS world, this played out a little bit differently. CS conferences operated like journals, with prestigious print proceedings that were often the final destination of a paper. In communities like ACL, workshops arose as the less-selective venue for work in progress. Again, there was a convention that these forums or preliminary work didn’t count as real publication. ACL wouldn’t want to undermine its own workshops by forcing authors to choose between workshop and conference. In the same way, I think, ACL shouldn’t force authors to choose between arXiv and conference.
As Michael points out, the lines have blurred so that there’s no longer an obvious distinction between “preliminary” (unpublished, non-archival) and “final” venues. That blurring started decades ago — ACL workshops had print proceedings, at least by the time I joined the field in the 1990’s, and libraries could order print copies from the ACL office in New Jersey. Yet there were clear conventions about which venues “counted” as real publications. There are more venues now, so maybe it’s getting a bit confusing what counts, who should decide, and on what grounds. Surely self-publishing on arXiv is about the least likely to count, but how about other cases?
To update the rules wisely, we have to ask why these rules exist in the first place. And in a world of internet-based open-access non-profit publishing that potentially allows versioning and commenting, we might consider whether the old publishing-house prohibitions on double publication are still serving us well. Scientific communication has changed a lot in the past 25 years, and it may look very different in another 25, for better or for worse! We should try to channel the changes in a good direction, toward a system that provides lot of signal and a high signal-to-noise ratio.
Many people (e.g., Yann LeCun) have advocated for a system in which publication is separated from endorsement. In essence, authors would publish a succession of versions under their own authority, uploading them to some server such as arXiv.org, OpenReview.net, or even aclweb.org. Readers could spontaneously post comments, questions, and reviews (perhaps with anonymity and/or moderation). Organizations like ACL could choose to commission additional reviews. Ultimately, such organizations would bless some version of the paper with an endorsement if it was good enough, and maybe a starred endorsement or award if it was strongly recommended within a particular category. This isn’t unlike how book reviewing works today.
In such a system, the old concerns about double publication nearly vanish. There’s nothing wrong with double endorsement: ACL and CVPR might both be willing to consider the same paper (or different versions of it) and endorse it to their respective audiences as worth reading. It’s not obvious how to set up the obligations and incentives here – when is a paper entitled to a review? A re-review? But like our current system, it would have to depend on a lot of goodwill and professionalism from community-minded reviewers, and some mechanism for avoiding duplicate effort (e.g., CVPR might decline to assign additional reviewers while the paper is flagged as being under review by ACL).
An obstacle to such a system is the desire to conceal authorship from reviewers – the original topic of this thread. That’s indeed a good idea, but not a sacred principle. We’ve never been able to perfectly hide authorship. Doing so would only ameliorate biases based on author identity, not biases based on topic or style. Most journals (including our own CL) and grant-making agencies have never even tried to hide authorship, sometimes arguing that reviewers need that context to decide where to trust the authors. Even at ACL 2017, the decisions will be made by program and area chairs who do see the author names.
If I could, I would make the current process even blinder in the interest of fairness. But I wouldn’t hold the whole scientific communication system hostage to that principle. We have to find some engineering solution that balances different values … as the community always has.
LikeLike
I fully agree. The system used to be somehow consistent in the environment in which it had developed, but the environment has changed quite drastically. I fear that patching the system here and there will not fix it but only introduce more inconsistencies, in particular as double-blind review, preprints, new publication types such as datasets, and probably other issues are all closely related. For example, if we consider double-blind reviewing paramount (and thus crack down hard on authors how fail to anonymize their papers), we can’t at the same time accept preprints and publication types that can’t be properly anonymized. We thus have to discuss what we really want and how to balance conflicting requirements; having an open discussion here is certainly a good start.
LikeLiked by 1 person