Program Committee (PC) meetings are this mysterious event where the fate of our
research projects is decided based on a review of our paper submission.
Especially for beginning researchers (i.e., PhD students) it is unclear
how the evaluation and review process actually works. From a student's
perspective, a paper is -- in systems or security in computer science --
generally submitted to a conference after experiments and writing are completed.
How to efficiently focus on design, implementation, evaluation, and writing of a
paper is not straight forward and is worth a couple of other blog posts. While
others have discussed how to review a paper, the discussion
process on how papers are evaluated to build a conference program are somewhat
opaque. This blog post tries to shed some light on this process and the work of
the PC committee with some hints what the PC chair does. The target audience are
PhD students that want to understand the process and the post is written from
the viewpoint of a junior researcher that is selected to be on a conference PC
for the first time.
PC selection
The PC chair (or chairs) is usually selected by the conference steering
committee. The chair and the steering committee then come up with a list of
potential PC members. To ensure that all topics and areas in the call for papers
are well covered, the list of PC members should be balanced according to
individual strengths and foci of the different PC members. The call for papers
is usually based on last years edition with tweaks regarding topics based on
changes in the community to adapt to new topics and changes in existing topics.
After adding enough candidates to the PC pool, the candidates are ranked based
on different criteria such as (i) if they have served before, (ii) if they are
well known in the community/do they publish at the same conferences, (iii) if
they write reasonable reviews, and (iv) conference and committee politics.
Depending on the prestige of the conference and how well a chair can bully their
friends into serving (and the tone of the invitation email), more or less
people will agree to serve on the PC. The declined invitations can be filled
with people further down on the list in the same area to ensure good topic
coverage.
The selection process continues until enough people have agreed to be on the PC.
The chair or steering committee (or web chair) publishes the list of PC members,
starts advertising the conference, and asks the PC members to advertise
and sometimes to submit papers as well.
Another administrative task is the selection of the submission system. While
there are several systems, some are more or less friendly to the reviewers.
All have different advantages and disadvantages. My personal favorite is HotCRP which can either be used in its self-hosted
open-source version or in the fairly costly hosted version (where the conference
pays for each submitted paper). Alternatives are, e.g., EasyChair or OpenConf.
Paper bidding
After the submission deadline has passed and all the papers are in the system,
the chairs generally work through all the papers to remove any papers that
violate format guidelines or are only half submitted. Afterwards PC members are
asked to bid for papers they want to review. In rare cases, the PC chair
directly assigns papers to reviewers, but this is less common.
Remember when, as an author, you select from a list of topics to submit a paper.
These topics allow a coarse selection based on interests. PC members similarly
mark their interests from the same list. On HotCRP, PC members signal from
strong interest to strong disinterest on a per-topic basis. These topic-based
interests are then used for an initial ranking of the papers. If a PC member
forgets to bid for papers, this selection can be used as a crude proxy to make a
selection.
As you'll be entering preferences, try to consider how likely you want to review
the paper. Read the abstract, title, and keywords. Does this sound like a fun
and interesting paper in your area? Then go ahead and mark your interest. PC
members are asked to enter a bid between -20 and +20 to show disinterest or
interest to rank each individual paper. The bids are then normalized over all
reviewers. A special mark (-100) is often used to highlight a potential
forgotten conflict.
The long list of papers can be daunting. All papers that have a topic score
above 0 should could be interesting and you should at least check the title and
abstract. With lower topic scores, the chances are less likely that you will
enjoy reviewing the paper. When entering bids, I follow the strategy to sort
papers based on the topic ranking from most to least interesting, displaying a
detailed view with titles and abstracts. After going through all papers, I
always sort the papers based on my bids to check the top papers again. Those are
the papers most likely to end up in my review pile. Additionally, I often
search for keywords that I know are interesting for me to ensure that I did not
forget to bid for a particular paper. Think of this as a whitelist that you
want to check for (I don't have a blacklist at the moment).
After the end of the bidding phase, the PC chair assigns papers to individual PC
members, often in an automatic or semi-automatic way to reduce the amount of
reviewer pain and to maximize expertise for each paper. The reviewing systems
often provide several metrics and interfaces to carefully select and balance
reviewer workload, interests, and a fair set of reviewers per paper. Not each
paper will get a perfect set of reviewers. Sometimes there are papers with no
bids and it is up to the discretion of the PC chair to select a lucky PC member
to review that paper. Not every PC member only gets their favourite papers to
review. The selection process explains some of the low expertise reviews. Two
alternate options are that some PC members may not bid for papers or they
misjudge their expertise in a paper.
Both as an author and a reviewer it is super important to not underestimate the
topic selection and bidding process. As an author, you set the tone of the paper
by providing a good set of topics that allows the reviewers to quickly assess
the generic area of your paper and the contributions of your paper. As a
reviewer, with the topic selection you can decide what papers you want to focus
on when making your bid. It's a great idea to spend the majority of your bidding
selection time on papers you would want to review. For your bidding, you then
need to make sure that these are papers that are interesting to you. You will
spend quite some time reading the papers and writing the reviews, so it should
also be fun!
Reviewing
Reviewing is an art in itself and has been covered by several blog posts,
classes, or even technical reports. I personally follow the strategy to skim the
paper from beginning to end to get an overview, then read the paper to keep
detailed notes. My review is then often split into summary, list of
strengths, list of weaknesses, general comments, writing issues, and questions
to the authors. An important aspect is the discussion of novelty and how the
paper relates to existing work. If you think something is not novel, always make
sure to give references. The tone of the review is also important. Reviews are
blind and a one-way communication. Authors cannot respond to reviews and as a
reviewer it is your job to provide context and as much information to help the
author improve their paper. Aggressive, angry, or insulting reviews are not
helpful in the review process. Stay friendly and helpful.
When entering scores for the papers make sure to normalize over the batch of
papers that you have received as this will allow some initial assessment of your
batch. Note that it is statistically highly unlikely that it's always you who
gets the worst batch of papers. Don't try to kill papers, stay positive. Search
for novel nuggets and contributions that make a paper worthy of acceptance.
PC meeting
PC meetings are either online or offline. At the PC meeting, the reviewers
argue about acceptance or rejection of each submitted paper, orchestrated by the
PC chair. Online PC meetings often use the discussion feature of the reviewing
system. The reviewers of the paper add comments and discuss pros and cons. Stay
active and involved in the papers you reviewed and make sure to follow up on
questions from other reviewers and try to defend the papers you want to argue
for. Bring factual arguments about the novelty or why you think the paper should
be accepted.
If the PC meeting is onsite, be prepared for some extra fun. An onsite PC
meeting is usually 1-2 days and heavily orchestrated by the PC chair. The first
task for the chair is to decide the discussion order, i.e., in what sequence
papers will be discussed. There are several strategies with different
advantages and disadvantages. Some of the more common ones are: (i) random order
to keep reviewers arguing close to a baseline, (ii) front load good papers to
set the tone, with the potential drawback that the discussion turns negative as
"too many papers have already been accepted", (iii) front load bad papers, or
(iv) minimize conflicts. Conflicted reviewers have to leave the room if a
conflicted paper is discussed. This can delay the discussion due to shuffling
people around and fetching them back into the room.
Independent of the order, some or all papers will be discussed and each
discussed paper receives a slot of a couple of minutes. The discussion
lead first summarizes the paper and the highlights in the reviews. The merit of
the paper is then discussed by all the reviewers. The PC chair often slightly
steers the discussion so that it stays on track and may table the discussion if
it takes too long, coming back to the paper later. If the reviewers cannot agree
if the paper should be accepted or not, other PC members can chime in and,
finally, vote on the merit of the paper.
One aspect that is getting increasingly common is to add a note of the PC
discussion to the reviews which allows the authors to better understand what
aspects were discussed during the meeting and what the reviewers think should be
changed or what they liked.
Especially as a junior person, an important aspect of the onsite PC meeting is
networking. PC meetings are an awesome opportunity to discuss research questions
with senior colleagues or to get to know other people in the wider field. If you
can, take part of the onsite PC meeting even if it involves travel. It's usually
more compact than a conference and you get much more face time with your
colleagues.
Student PC
Student PCs are a somewhat modern approach to allow senior students to
experience a mock PC. The student PC reviews the same (or sometimes a slightly
smaller) set of papers in a realistic setting as possible. Such PCs are often a
great way to get to know other academics and to network with your peer students.
The trade-off is the amount of work and the additional travel that takes some
time of your schedule compared to working on your next research project.
Case study: a local dry-run
Student PCs may sometimes be too high a burden with additional travel and
especially for junior students the total overhead may not be worth it. To allow
my students to experience and understand the PC environment, we have conducted a
mock Usenix Security PC last week. After receiving my batch of Usenix Security papers, I
played the role of PC chair and assigned my interested students a set of papers
to review. Orthogonally, I have reviewed all the papers myself, to act as a
discussion partner during the mock PC meeting. We then shared all the reviews
to prepare for an onsite discussion. Similar to a real PC meeting, we had
lots of caffeinated drinks and sugery food to stay sharp.
During our half-day PC meeting we worked through the set of papers, following
a random strategy. The students were the discussion leads as we discussed each
paper. When writing reviews, students often go through an interesting
development: at the beginning of their research career, they are overly positive
(this is the best paper I've ever seen) without considering related work. With
some research experience, they become overly negative (everything has been done
before), until they start to normalize and become more practiced in their
reviewing skills.
For each paper, we discussed strengths and weaknesses and worked on either
accepting or rejecting it. In the end we came up with a set of 5 out of 12
papers that were at least slightly positive to advance to the next round and
likely 3 that would have been accepted. Not bad for a first PC meeting. After we
finished the discussion, I went through all the reviews again and adjusted some
of my reviews and scores. This exercise was also worthwhile for me and helped me
improve my reviews as well.
A note on conflicts of interest: handling conflicts is hard! I ensured that only
my primary advisees were reviewing the papers. As they share my conflicts and my
conflicts are a superset of theirs, this mitigates the risk of unhandled
conflicts.
Conclusion
PC meetings are somewhat opaque to students. We as a community should allow
students who are responsible to carry the main load of most research projects to
better understand the review process. With understanding comes deeper insight
that allows optimization of the process. Students that understand how a paper is
reviewed by a PC member will start to write better papers -- highlighting the
strengths and novelties of the paper while clearly discussing the limitations.
I hope this short summary is useful to new students. A tricky question when
conducting such mock PCs are conflict of interest. As a PC member, the group
leader is responsible for all the reviews and should write the reviews
themselves. Orthogonally, students profit drastically from the experience of
reviewing papers. Until they are invited into PCs themselves or are senior
enough to become (rare) subreviewers, my opinion is that such mock PCs are a
great way to train our students.
I always encourage comments and feedback. Let me know what you think, if your
opinion on an aspect of the review process differs, what I missed, or what we
can improve for the next mock PC!