Does the author response make any difference?
Well, we won’t be able to give a definitive answer to this, but in our empirical drive to run an evidence-based programme committee chairing, we have downloaded the review data at different time points from our conference management software, to seek data that could answer this question as best as we can thus far.
We downloaded the review data directly before the beginning the author response period (March 12) and recently, after reviewers had sufficient time to read and react to author responses (March 23).
First some basic statistics. For both long and short submissions, over 80% of submissions filed an author response, and 20% filing a direct-to-AC communication.
Does author response change the scores? Setting aside the probably significant confounding variables, and attributing changes in score only to author response, the answer is yes: in about 15-20% of cases, scores changed. Did they change for the better? On average, yes, it does help to respond; we see a ~2:1 ratio of positive to negative changes in score (also, but probably not significant, if authors don’t respond, there’s a stronger tendency to have a negative score change).
I also wanted to know whether score change trends are significantly different with different initial scores. Definitely yes. Faceting the results by quartile, there is a significant positive trend for submissions that have a high initial score, and a less strong negative trend for submissions that have a low initial score. At the borderline cases (2nd and 3rd quartiles), the boost that author response yields is still there, but less strong. We leave it to you to make conclusions as to the cause (we know, we are crunching only numbers, despite ACL being a NLP/CL conference — alas!).
Direct to Area Chair communication(a new feature we introduced this year) didn’t seem to have much effect — the pattern follows the author response fairly well. We’ll decide whether this helped in the process or not, pending your opinions.
The fine print: the statistics are not exact. Some papers had multiple score changes that averaged out to zero, so are present in the total but not the upward/downward trends. Much of the score changes are probably due to consolidation between peer reviews and discussions initiated by the area chairs. The average score change is a bit over 0.33 (as we have three reviewers, this is the minimal score change), but not enough to change most papers’ quartile rank. We used quartiles specific for each paper format (long and short), as they are different. The “(subset) w/ + direct-to-AC” column, is a subset of the middle column, as all submissions that filed a direct-to-AC text response also filed an author response.
As always, we welcome your insight and quest for deeper insight. If you have specific questions or stats that you’d like to see, let us know by commenting or raising your voice on Twitter or Facebook on these ACL 2017 posts.