yudai.py

My thoughts on the broken state of AI conference reviewing: Years ago, when I was in graduate school and a postdoc in Information Theory, I always felt fortunate to be invited to review for IEEE Transactions on Information Theory or IEEE Transactions on Signal Processing. I felt I was being considered an expert on the subject matter and that my opinion helped make a difference in the review process. I was treated with respect, and I was happy to be part of it. I started publishing in mainstream ML venues in 2017 and was invited to be a reviewer for ML conferences in 2018. I jumped in with the same passion, but it soon became clear that being a reviewer didn’t really matter. I would spend time engaging with a paper only to be overturned by an AC who didn’t read or understand my review, or didn’t care to engage and discuss their rationale. I witnessed terribly wrong papers being accepted and good papers being unduly rejected. I quickly started to feel my role as a reviewer was irrelevant. I wasn’t treated with respect, and I couldn’t make a meaningful difference in the system. I probably would have stopped reviewing for ML long ago if I hadn’t been promoted to AC, and I know a lot of senior researchers who stopped engaging and caring. When I became an AC, I tried to engage reviewers, treat them with respect, and make sure they felt empowered and valued (especially if I made a decision that went against their recommendation). This is crucial for a functioning review system because reviewers need to feel their work matters if we want them to keep coming back and engaging. Now, we’ve reached a point where many reviewers feel their work doesn’t matter. The more experienced reviewers aren’t excited to engage, and the review quality keeps going down to the extent that we really lack qualified reviewers in the system. What solution have we found to this problem? Force everyone to be a reviewer. This is what EMNLP, NeurIPS, CVPR, etc., did this year, which (in my opinion) is a terrible way to address the root cause. We should have thought about how to bring back dignity to the reviewer’s role instead. Fast forward to today, and we are taking steps to make the role of an AC irrelevant as well. When PCs/SACs overturn an AC’s decision without even engaging the AC in the process, the AC doesn’t feel respected or useful. This has been happening in the NLP conferences for a while and happened this year at NeurIPS. Not only does an AC likely feel irrelevant, the SACs will feel the same. We will quickly see experienced members of the community stop accepting AC/SAC roles because they feel it’s a useless activity. The quality of ACs/SACs will go down, and more of the burden will shift to the PCs, who have to decide on 20k+ papers. The end result is that the entire system will soon crumble.