How is the program created?
Path #1 – Peer Review
Immediately following the close of the submission site, proposals are sent to our Program Planning Committee (PPC) for formal, blind peer review. The PPC takes great care to review, score, and comment on all submitted proposals. The PPC chair organizes the assignments ensuring balanced distribution. The results are distilled into a comprehensive report that is then reviewed by the program committee chairs. Within this report, average scores indicate only one part of the overall desirability of a session. Standard deviation plays a significant part. When the standard deviation is high, regardless of the average score, there is further discussion on that submissions to identify why reviewers had vastly different opinions on the talk. Sometimes a new topic needs to be on the program because of this high standard deviation.
Path #2 – Community Vote
In parallel to the peer review, we conduct a Community Vote to share the submitted proposals to the community to gauge interest and surface submission trends. Our committee is keen on knowing the desires of the community and what will be of most interest. It is a simple and anonymous process that gleans an interesting set of data from the community. We realize that the Community Vote could be perceived as a popularity contest (with a high vote count possibly being the result of someone rallying to tip the scales), so rest assured we pay close attention to those types of outliers in voting during the final determination.
Without question, the most important part of the review process is the peer review and the resulting average scores and standard deviations. Community Vote is one part of a multi-faceted approach that considers the PPC scores and standard deviation first, then community vote results followed by space constraints, track balancing, representations (organization size, geography and other), and other special circumstances, creating a program that is balanced and fulfills the needs of the community.