Program committee (PC) meetings are a strange thing. As a graduate student you will surely have experienced the outcome from PC meetings: a message telling you your paper got accepted or the dreaded message that it got rejected and you'll have to send it to another conference.

But how does the review process itself work? During my time as graduate student and my time as a post doctoral scholar I've had my fair share of rejects and a good share of accepts and I have always wondered how the decisions were carried out in the background. I got my first experiences into the process as an external reviewer for my adviser where I had to sum up papers and their contributions for a set of conferences. Later in my studies, I got invited to review some articles for journals where I have apparently become an expert.

Only when I started my postdoctoral studies I got invited to my first PCs, starting with a systems conference (ACM SYSTOR'13 - thanks Mary and Dilma) and a workshop (ACM PPREW'14 - thanks Todd). Unfortunately, the PC meetings for both these conferences were only virtual, meaning that there was no real-world meeting where all the papers were discussed but we used an online forum to discuss which papers got accepted and which ones rejected. Earlier this week I finally went to my first real-world PC meeting at IBM Watson in NY to discuss the program for the upcoming ACM VEE'14 conference (thanks Dilma and Dan for thinking of me).

For all these conferences the review process is surprisingly similar: first, shortly after the submission deadline the reviewers read through all the paper titles and abstract and bet for papers they want to review. I tend to group papers into 4 sets, namely I want to review this paper, I could review this paper, I don't care, and I don't want to review this paper. The system then clusters all bets and suggests a selection of papers for each reviewer that the program chairs can manually optimize. Second, the actual review phase lasts between just a couple of days for small workshops to several weeks. In this phase, the reviewers actually review the papers, write their reviews, adjust their scores over all their papers, and submit the reviews into the submission system. This process usually lasts until a couple of days before the PC meeting. After submitting their reviews, the reviewers usually have access to the reviews of the other reviewers. Third, sometimes there is a rebuttal phase where the authors get access to the latest version of the reviews and get a chance to defend their work from the usually anonymous reviewers. The fourth step is the actual PC meeting.

In the meeting the merit of all submitted papers is discussed. In my first PC meeting the chairs ordered the submissions into 3 groups: early accept (papers with only positive reviews), early reject (papers with only negative reviews), and remaining papers. We voted to follow these suggestions and accept the positively reviewed papers and reject the negatively reviewed papers. All remaining papers were discussed in detail. For each paper we had a discussion lead that summarized the paper in 3-4 minutes, mentioning strengths and weaknesses as identified by the individual reviewers. In the open discussion that followed, the actual reviewers weighed in their opinions first and later all PC members were allowed to join the discussion. I enjoyed the open atmosphere and we had some great technical discussions over some of the papers. It was also really interesting to see the nuances how different reviewers judge the scientific contributions of a paper.

So as not to embarrass myself I prepared well, I finished my reviews early (I knew that I would get busy with the reviews for ASIACCS'14 in due time), wrote up detailed reviews for each paper, read the reviews of the other PC members, and followed the rebuttals that were added to the papers I reviewed. Before the actual meeting I reread my reviews to page the discussion back into my mind and prepared a quick summary for the papers were I was the discussion lead.

Overall the discussion of my papers went well. The overall PC meeting went for roughly 8 hours with only a short lunch break in between. The discussion was always on a very high level and it was great to see so many great minds running in lockstep, discussing pro and contra arguments. In the end, there was only one paper where there was a disagreement in the PC if we should accept or reject the paper and this conflict was solved by a democratic vote.

Some of the (sometimes obvious) lessons I have learned during this discussion are:

• PC meetings are a great place to network. For me it was a huge chance to actually meet with many more senior people in academia and industry. It was interesting to bring in my expertise and ask them about their thoughts (both about the places where they work and their decision whether to choose academia or industry).
• The double blind review process is really double blind. This fact actually surprised me a little as I thought it would be handled in a more lax fashion. For most of the papers I reviewed I had no clue about who the authors could be and Erez and Dan did a great job in keeping the discussion straight on the technical contributions of each work. I only saw the authors of the accepted papers a couple of days after the PC meeting after we have all finished revising our reviews.
• The discussion itself is surprisingly open and honest. Nobody was pushing their own work and all papers were evaluated in a fair and objective way. Even though I was one of the academically youngest people, I was taken serious during the discussion. Somewhat surprisingly I was not just a silent listener but an active participant that was frequently asked about his opinion.
• I liked the fact that the reviewer with the best evaluation had to summarize the paper and all other reviews at the beginning of each discussion. This gave me the opportunity to push the papers I really liked.

All in all, I must say that I really enjoyed my first real PC meeting. I learned first hand how these things work and Erez and Dan did a great job in leading the meeting and discussion while Martin did an awesome job at organizing the meeting (and the following dinner).