PC selection

The PC chair (or chairs) is usually selected by the conference steering committee. The chair and the steering committee then come up with a list of potential PC members. To ensure that all topics and areas in the call for papers are well covered, the list of PC members should be balanced according to individual strengths and foci of the different PC members. The call for papers is usually based on last years edition with tweaks regarding topics based on changes in the community to adapt to new topics and changes in existing topics.

After adding enough candidates to the PC pool, the candidates are ranked based on different criteria such as (i) if they have served before, (ii) if they are well known in the community/do they publish at the same conferences, (iii) if they write reasonable reviews, and (iv) conference and committee politics. Depending on the prestige of the conference and how well a chair can bully their friends into serving (and the tone of the invitation email), more or less people will agree to serve on the PC. The declined invitations can be filled with people further down on the list in the same area to ensure good topic coverage.

The selection process continues until enough people have agreed to be on the PC. The chair or steering committee (or web chair) publishes the list of PC members, starts advertising the conference, and asks the PC members to advertise and sometimes to submit papers as well.

Another administrative task is the selection of the submission system. While there are several systems, some are more or less friendly to the reviewers. All have different advantages and disadvantages. My personal favorite is HotCRP which can either be used in its self-hosted open-source version or in the fairly costly hosted version (where the conference pays for each submitted paper). Alternatives are, e.g., EasyChair or OpenConf.

Paper bidding

After the submission deadline has passed and all the papers are in the system, the chairs generally work through all the papers to remove any papers that violate format guidelines or are only half submitted. Afterwards PC members are asked to bid for papers they want to review. In rare cases, the PC chair directly assigns papers to reviewers, but this is less common.

Remember when, as an author, you select from a list of topics to submit a paper. These topics allow a coarse selection based on interests. PC members similarly mark their interests from the same list. On HotCRP, PC members signal from strong interest to strong disinterest on a per-topic basis. These topic-based interests are then used for an initial ranking of the papers. If a PC member forgets to bid for papers, this selection can be used as a crude proxy to make a selection.

As you'll be entering preferences, try to consider how likely you want to review the paper. Read the abstract, title, and keywords. Does this sound like a fun and interesting paper in your area? Then go ahead and mark your interest. PC members are asked to enter a bid between -20 and +20 to show disinterest or interest to rank each individual paper. The bids are then normalized over all reviewers. A special mark (-100) is often used to highlight a potential forgotten conflict.

The long list of papers can be daunting. All papers that have a topic score above 0 should could be interesting and you should at least check the title and abstract. With lower topic scores, the chances are less likely that you will enjoy reviewing the paper. When entering bids, I follow the strategy to sort papers based on the topic ranking from most to least interesting, displaying a detailed view with titles and abstracts. After going through all papers, I always sort the papers based on my bids to check the top papers again. Those are the papers most likely to end up in my review pile. Additionally, I often search for keywords that I know are interesting for me to ensure that I did not forget to bid for a particular paper. Think of this as a whitelist that you want to check for (I don't have a blacklist at the moment).

After the end of the bidding phase, the PC chair assigns papers to individual PC members, often in an automatic or semi-automatic way to reduce the amount of reviewer pain and to maximize expertise for each paper. The reviewing systems often provide several metrics and interfaces to carefully select and balance reviewer workload, interests, and a fair set of reviewers per paper. Not each paper will get a perfect set of reviewers. Sometimes there are papers with no bids and it is up to the discretion of the PC chair to select a lucky PC member to review that paper. Not every PC member only gets their favourite papers to review. The selection process explains some of the low expertise reviews. Two alternate options are that some PC members may not bid for papers or they misjudge their expertise in a paper.

Both as an author and a reviewer it is super important to not underestimate the topic selection and bidding process. As an author, you set the tone of the paper by providing a good set of topics that allows the reviewers to quickly assess the generic area of your paper and the contributions of your paper. As a reviewer, with the topic selection you can decide what papers you want to focus on when making your bid. It's a great idea to spend the majority of your bidding selection time on papers you would want to review. For your bidding, you then need to make sure that these are papers that are interesting to you. You will spend quite some time reading the papers and writing the reviews, so it should also be fun!

Reviewing

Reviewing is an art in itself and has been covered by several blog posts, classes, or even technical reports. I personally follow the strategy to skim the paper from beginning to end to get an overview, then read the paper to keep detailed notes. My review is then often split into summary, list of strengths, list of weaknesses, general comments, writing issues, and questions to the authors. An important aspect is the discussion of novelty and how the paper relates to existing work. If you think something is not novel, always make sure to give references. The tone of the review is also important. Reviews are blind and a one-way communication. Authors cannot respond to reviews and as a reviewer it is your job to provide context and as much information to help the author improve their paper. Aggressive, angry, or insulting reviews are not helpful in the review process. Stay friendly and helpful.

When entering scores for the papers make sure to normalize over the batch of papers that you have received as this will allow some initial assessment of your batch. Note that it is statistically highly unlikely that it's always you who gets the worst batch of papers. Don't try to kill papers, stay positive. Search for novel nuggets and contributions that make a paper worthy of acceptance.

PC meeting

PC meetings are either online or offline. At the PC meeting, the reviewers argue about acceptance or rejection of each submitted paper, orchestrated by the PC chair. Online PC meetings often use the discussion feature of the reviewing system. The reviewers of the paper add comments and discuss pros and cons. Stay active and involved in the papers you reviewed and make sure to follow up on questions from other reviewers and try to defend the papers you want to argue for. Bring factual arguments about the novelty or why you think the paper should be accepted.

If the PC meeting is onsite, be prepared for some extra fun. An onsite PC meeting is usually 1-2 days and heavily orchestrated by the PC chair. The first task for the chair is to decide the discussion order, i.e., in what sequence papers will be discussed. There are several strategies with different advantages and disadvantages. Some of the more common ones are: (i) random order to keep reviewers arguing close to a baseline, (ii) front load good papers to set the tone, with the potential drawback that the discussion turns negative as "too many papers have already been accepted", (iii) front load bad papers, or (iv) minimize conflicts. Conflicted reviewers have to leave the room if a conflicted paper is discussed. This can delay the discussion due to shuffling people around and fetching them back into the room.

Independent of the order, some or all papers will be discussed and each discussed paper receives a slot of a couple of minutes. The discussion lead first summarizes the paper and the highlights in the reviews. The merit of the paper is then discussed by all the reviewers. The PC chair often slightly steers the discussion so that it stays on track and may table the discussion if it takes too long, coming back to the paper later. If the reviewers cannot agree if the paper should be accepted or not, other PC members can chime in and, finally, vote on the merit of the paper.

One aspect that is getting increasingly common is to add a note of the PC discussion to the reviews which allows the authors to better understand what aspects were discussed during the meeting and what the reviewers think should be changed or what they liked.

Especially as a junior person, an important aspect of the onsite PC meeting is networking. PC meetings are an awesome opportunity to discuss research questions with senior colleagues or to get to know other people in the wider field. If you can, take part of the onsite PC meeting even if it involves travel. It's usually more compact than a conference and you get much more face time with your colleagues.

Student PC

Student PCs are a somewhat modern approach to allow senior students to experience a mock PC. The student PC reviews the same (or sometimes a slightly smaller) set of papers in a realistic setting as possible. Such PCs are often a great way to get to know other academics and to network with your peer students. The trade-off is the amount of work and the additional travel that takes some time of your schedule compared to working on your next research project.

Case study: a local dry-run

Student PCs may sometimes be too high a burden with additional travel and especially for junior students the total overhead may not be worth it. To allow my students to experience and understand the PC environment, we have conducted a mock Usenix Security PC last week. After receiving my batch of Usenix Security papers, I played the role of PC chair and assigned my interested students a set of papers to review. Orthogonally, I have reviewed all the papers myself, to act as a discussion partner during the mock PC meeting. We then shared all the reviews to prepare for an onsite discussion. Similar to a real PC meeting, we had lots of caffeinated drinks and sugery food to stay sharp.

During our half-day PC meeting we worked through the set of papers, following a random strategy. The students were the discussion leads as we discussed each paper. When writing reviews, students often go through an interesting development: at the beginning of their research career, they are overly positive (this is the best paper I've ever seen) without considering related work. With some research experience, they become overly negative (everything has been done before), until they start to normalize and become more practiced in their reviewing skills.

For each paper, we discussed strengths and weaknesses and worked on either accepting or rejecting it. In the end we came up with a set of 5 out of 12 papers that were at least slightly positive to advance to the next round and likely 3 that would have been accepted. Not bad for a first PC meeting. After we finished the discussion, I went through all the reviews again and adjusted some of my reviews and scores. This exercise was also worthwhile for me and helped me improve my reviews as well.

A note on conflicts of interest: handling conflicts is hard! I ensured that only my primary advisees were reviewing the papers. As they share my conflicts and my conflicts are a superset of theirs, this mitigates the risk of unhandled conflicts.

Conclusion

PC meetings are somewhat opaque to students. We as a community should allow students who are responsible to carry the main load of most research projects to better understand the review process. With understanding comes deeper insight that allows optimization of the process. Students that understand how a paper is reviewed by a PC member will start to write better papers -- highlighting the strengths and novelties of the paper while clearly discussing the limitations.

I hope this short summary is useful to new students. A tricky question when conducting such mock PCs are conflict of interest. As a PC member, the group leader is responsible for all the reviews and should write the reviews themselves. Orthogonally, students profit drastically from the experience of reviewing papers. Until they are invited into PCs themselves or are senior enough to become (rare) subreviewers, my opinion is that such mock PCs are a great way to train our students.

I always encourage comments and feedback. Let me know what you think, if your opinion on an aspect of the review process differs, what I missed, or what we can improve for the next mock PC!