Invitation to contribute to a public release of anonymous peer reviews

A few months ago, I reached out to Regina and Min suggesting that anonymous peer reviews in ACL 2017 are made publicly available on an opt-in basis. After discussing the pros and cons, they agreed to carefully experiment with the idea.

Similar to previous years, ACL 2017 will continue to adopt a double-blind review process. By default, reviews on an ACL submission will only be visible to its authors and assigned reviewers (and the area and conference chairs, of course). Opted in reviews will be anonymized then included in a dataset of peer reviews, to be released twelve months after the accept/reject decisions are made, under the Creative Commons Attribution 4.0 International License. The dataset will also include peer reviews from other conferences such as NIPS, ICLR and CoNLL. For a review to be included in the dataset, both the author who submits the paper to ACL and the reviewer who writes the review must opt-in.

Our hope is that publishing peer reviews will have a positive impact on our research community. Many reviewers put a considerable amount of time and critical thinking in the review process, and would like their reviews to have a wider readership. A good review gives a quick, unbiased summary of the pros and cons of a paper. Publishing peer reviews will also increase the transparency of accept/reject decisions. Most importantly, analyzing the reviews may lead to interesting observations that help improve the efficiency and quality of the review process in future conferences. It is conceivable, however, that publishing peer reviews might have unintended consequences. For example, published reviews might unduly bias a reader’s opinion in favor or against a paper before reading it.

Please consider opting in your paper draft (in the submission form on softconf) and your reviews (in the review form on softconf) if you think the advantages of publishing peer reviews outweigh the disadvantages. Whatever you decide, it would be great if you could share your reasoning in comments. 

Waleed Ammar — research scientist at the Allen Institute for Artificial Intelligence

Advertisements

3 thoughts on “Invitation to contribute to a public release of anonymous peer reviews

    1. Only the review text and the categorical scores were included when we did this for CoNLL 2016. I leave it to Min and Regina to decide whether accept/reject decisions should also be exported.

      Liked by 1 person

      1. Since this information could be construed as sensitive and we did not disclose beforehand whether it would be public, we’d prefer to err on the conservative side and not publish the accept/reject decision with the dataset.

        Liked by 1 person

Comments are closed.