Program committee (PC) meetings are a strange thing. As a graduate student you
will surely have experienced the outcome from PC meetings: a message telling
you your paper got accepted or the dreaded message that it got rejected and
you'll have to send it to another conference.
But how does the review process itself work? During my time as graduate student
and my time as a post doctoral scholar I've had my fair share of rejects and a
good share of accepts and I have always wondered how the decisions were carried
out in the background. I got my first experiences into the process as an
external reviewer for my adviser where I had to sum up papers and their
contributions for a set of conferences. Later in my studies, I got invited to
review some articles for journals where I have apparently become an expert.
Only when I started my postdoctoral studies I got invited to my first PCs,
starting with a systems conference (ACM SYSTOR'13 - thanks Mary and Dilma) and a
workshop (ACM PPREW'14 - thanks Todd). Unfortunately, the PC meetings for both
these conferences were only virtual, meaning that there was no real-world
meeting where all the papers were discussed but we used an online forum to
discuss which papers got accepted and which ones rejected. Earlier this week I
finally went to my first real-world PC meeting at IBM Watson in NY to discuss
the program for the upcoming ACM VEE'14 conference (thanks Dilma and Dan for
thinking of me).
For all these conferences the review process is surprisingly similar: first,
shortly after the submission deadline the reviewers read through all the paper
titles and abstract and bet for papers they want to review. I tend to group
papers into 4 sets, namely I want to review this paper, I could review this
paper, I don't care, and I don't want to review this paper. The system then
clusters all bets and suggests a selection of papers for each reviewer that the
program chairs can manually optimize. Second, the actual review phase lasts
between just a couple of days for small workshops to several weeks. In this
phase, the reviewers actually review the papers, write their reviews, adjust
their scores over all their papers, and submit the reviews into the submission
system. This process usually lasts until a couple of days before the PC
meeting. After submitting their reviews, the reviewers usually have access to
the reviews of the other reviewers. Third, sometimes there is a rebuttal phase
where the authors get access to the latest version of the reviews and get a
chance to defend their work from the usually anonymous reviewers. The fourth
step is the actual PC meeting.
In the meeting the merit of all submitted papers is discussed. In my first PC
meeting the chairs ordered the submissions into 3 groups: early accept (papers
with only positive reviews), early reject (papers with only negative reviews),
and remaining papers. We voted to follow these suggestions and accept the
positively reviewed papers and reject the negatively reviewed papers. All
remaining papers were discussed in detail. For each paper we had a discussion
lead that summarized the paper in 3-4 minutes, mentioning strengths and
weaknesses as identified by the individual reviewers. In the open discussion
that followed, the actual reviewers weighed in their opinions first and later
all PC members were allowed to join the discussion. I enjoyed the open
atmosphere and we had some great technical discussions over some of the papers.
It was also really interesting to see the nuances how different reviewers judge
the scientific contributions of a paper.
So as not to embarrass myself I prepared well, I finished my reviews early (I
knew that I would get busy with the reviews for ASIACCS'14 in due time), wrote
up detailed reviews for each paper, read the reviews of the other PC members,
and followed the rebuttals that were added to the papers I reviewed. Before the
actual meeting I reread my reviews to page the discussion back into my mind and
prepared a quick summary for the papers were I was the discussion lead.
Overall the discussion of my papers went well. The overall PC meeting went for
roughly 8 hours with only a short lunch break in between. The discussion was
always on a very high level and it was great to see so many great minds running
in lockstep, discussing pro and contra arguments. In the end, there was only
one paper where there was a disagreement in the PC if we should accept or
reject the paper and this conflict was solved by a democratic vote.
Some of the (sometimes obvious) lessons I have learned during this discussion
are:
- PC meetings are a great place to network. For me it was a huge chance to
actually meet with many more senior people in academia and industry. It was
interesting to bring in my expertise and ask them about their thoughts (both
about the places where they work and their decision whether to choose
academia or industry).
- The double blind review process is really double blind. This fact actually
surprised me a little as I thought it would be handled in a more lax
fashion. For most of the papers I reviewed I had no clue about who the
authors could be and Erez and Dan did a great job in keeping the discussion
straight on the technical contributions of each work. I only saw the authors
of the accepted papers a couple of days after the PC meeting after we have all
finished revising our reviews.
- The discussion itself is surprisingly open and honest. Nobody was pushing
their own work and all papers were evaluated in a fair and objective way.
Even though I was one of the academically youngest people, I was taken
serious during the discussion. Somewhat surprisingly I was not just a silent
listener but an active participant that was frequently asked about his opinion.
- I liked the fact that the reviewer with the best evaluation had to summarize
the paper and all other reviews at the beginning of each discussion. This
gave me the opportunity to push the papers I really liked.
All in all, I must say that I really enjoyed my first real PC meeting. I learned
first hand how these things work and Erez and Dan did a great job in leading
the meeting and discussion while Martin did an awesome job at organizing the
meeting (and the following dinner).