Dear invited Area Chairs:
[TL;DR: actually you need to read the whole post. So no summary! ]
To increase transparency of the process, we are making this an open letter, which we hope you (and perhaps the general public) will comment on to improve the reviewing process.
First, thank you for considering the charge of being an area chair (AC) for ACL 2017!
We are writing this letter to you to inform you of several important changes to the ACL 2017 organization that will affect how you recruit reviewers and manage the deadlines for the conference. Since these changes could potentially affect whether you decide to take up the decision to be an AC, we decided to write this post first.
Major highlights of the changes to the ACL 2017 reviewing process:
- The deadline for short and long papers is joint. This affects the number of submissions and also the coordination and reviewing load.
- The initial review period is shortened to two weeks (13-27 Feb 2017). This affects how responsible reviewers need to be to ensure that the reviews are completed in time.
- Authors will have a textbox in the author response form to talk directly to ACs, only if needed. This will also affect ACs’ workflow during the author response period.
- We will use fairly broad areas and allow reviewers to review for multiple areas. This hopes to address the problem of “gaming” the selection of areas in the review process. However, this increases the coordination load among ACs assigned to manage their area.
- We are planning to use the Toronto system (used in NIPS) for assisting with assignment of papers to reviewers. This affects the workload of reviewers, as they will need to provide details on their past scientific work that will help build their reviewing profile to best match them to papers of their interest and qualification.
- There will be a meta AC per area to serve as a single point of contact for making recommendations for the area. This AC will be a senior member of the community that accepts this extra responsibility for organization and coordination matters. All ACs are, by default, assumed to have equal share in the scientific decisions on all matters for their respective areas.
Let’s look at each of these changes one by one, in more detail.
- Joint deadlines. With our community and the number of scientific submissions growing, we believe that ACL will attract a record number of submissions. We believe this is more so as in 2017, ACL will act as a both NAACL and ACL together, being the rotation where ACL is back in the Americas. This will put a burden on finding enough high quality reviewers to execute the review process faithfully for each submission.With staggered deadlines that have been the feature of ACL in recent years, this stretches out the review period and encourages more paper submissions. In contrast, we chose a joint deadline to try to lessen the workload per reviewer and have the reviews over a shorter time span. We believe that the joint deadline will not have much impact in lessening the number of lower quality submissions, but it may affect short paper submission numbers. This change will affect the demands on your time in coordinating reviews more carefully. We see the importance of emphasizing this need to be communicated carefully during the reviewer recruiting process.
- Shorter review period. After observing reviewing in many venues, our conclusion is that the length of the reviewing period doesn’t greatly matter for many reviewers — reviews tend to be completed very close to the deadline. A shorter reviewing period allows for more time for the other activities, especially meta reviews and considering author responses.We understand that reviewers do schedule their availability such that they can complete the reviews early, especially if they are not available close to the deadline, and that such reviewers would benefit from a longer review period. However, we believe this to be the minority of cases, and we firmly believe that the responsible reviewer needs to be involved in the dialogue that ensues from the initial review — the author response, the discussion with peer reviewers and the ACs, all the way until notification (i.e., from 6 Feb-30
- Direct to AC text box on author response form. This is a new pilot that we are conducting for ACL 2017, in which the author response form will have a text box for the authors to write about their concerns about any misunderstandings of the reviewers. This idea originated from the problem where deserving works were rejected by reviewers who may have misread or misunderstood the paper and who might not be receptive to the author response mechanism. We will try to ensure that this facility is only used in the case where the authors feel that the reviewers’ comments are out of line, and need to take the dialogue directly to ACs.
- Broad Areas. We have heard reports where the review quality in different areas can differ drastically, such that the authors feel the need to strategically select reviewing areas. Occasionally this can happen when having many, small areas that have fined-grained differences that may not matter for most submissions. In such cases, the decision of which area to assign a paper to can be difficult and cause bias. By having larger, broader areas, we hope to level the review process to ensure a more uniform review quality.Of course, this comes with a cost — areas will have to be monitored with more area chairs, which increases the coordination load among ACs to get their work done on schedule and to resolve conflicts within a reasonable timeframe. Again, some prescheduling and coordination work ahead of the reviewing period will be important to make this work, and the devotion of all ACs of an area to agree and adopt the schedule.
- Toronto system. We are planning to adopt the Toronto system for helping to assign reviewers to papers. The system, used in the Annual Conference of the Neural Information Processing Systems (NIPS), suggests reviewers based on an analysis of their published works (i.e., language model with LDA). You as part of the AC team can decide which recommendations to approve of, if any. It will substitute the current system where the matching is done on keywords alone. Based on feedback from the NIPS community, it has enhanced their reviewing process by providing a better fit of papers to reviewers, increasing the overall conference quality. To familiarize yourself with the system, please read the documentation on it:http://www.cs.toronto.edu/~zemel/documents/tpms.pdf
However, it does require work to use. Aside from work on our side to ensure it can be integrated in a timely fashion for ACL 2017, it requires reviewers to provide a webpage where the reviewer’s publications can be found, downloaded and analyzed. This affects the reviewers’ workload, as they will have to do some work to provide the necessary information for the recommendations to be of use. The good news is that the work is done one time only and when the system is used again in future ACL conferences, the work for any repeat reviewer is negligible (if any). This information needs to be carefully communicated to prospective reviewers.
This is a major change that we hope will continue to feature in ACL conferences and a pilot experiment backed by the ACL Executive board. It may not influence your workload much, but is important to disclose to you. Its usage is not finalized yet, because it depends on the technical team at Softconf to ready the system for integration. We plan to discuss the system in more detail in subsequent posts when the system is close to ready for use.
- Meta AC. With the expanded pool of ACs needed to handle the expected workload for ACL 2017, we will be appointing an initial meta AC for each area as well. This will be a senior member of the community that will be the main point of contact for making recommendations for the area, and will organize a discussion for papers among other ACs. The ACs for an area may decide to change the meta AC to another AC in the area — it’s not binding.
We assume that all ACs in an area carry the same weight in the scientific aspects of the program. To be clear, the meta AC role is for contact and organization only. We do not mean for a meta AC to be assumed as a head AC, whose role might be construed to mean moderating fellow AC decisions. As PC co-chairs, we will take that responsibility upon ourselves.
There are more details that are important for ACs but these are the major changes that we think will affect whether you accept the charges of being an AC. The above changes show that ACs will have a much more involved role in ACL 2017 in deciding the technical program, and much added responsibility. Most importantly, we need all ACs to be responsive during the critical time frame of 6 Feb – 30
If you can agree to these responsibilities, and dedicate time during this period to be able to join us as an area chair, we will gladly accept your help in making ACL 2017 a success. However, we completely understand if you cannot make the time commitment that will be needed for the expanded AC role that we are asking for. If you agree to these terms, please do respond to our personal invitation that will be emailed to you in the next few days.
We have also created a Google calendar containing the important pertinent dates. This is a global calendar for all ACs and will be updated when changes to events and deadlines occur. Where possible, please ensure you have access to the live calendar to keep apprised of relevant ACL 2017 AC events. Here’s the address:
We look forward to your favorable decision!
– The PC chairs