Rosewood
Implicit Bias and Conference Paper Reviews: A Double-Blind Experiment (304766)
Andrew Tomkins, Google Inc.Min Zhang, Tsinghua University
*William D Heavlin, Google Inc.
Keywords: double-blind review, effect sizes, gender bias, meta-analysis
WSDM 2017, the 10th International ACM Conference on Web Search and Data Mining, experimented with single- versus double-blind reviewing. Reviewers were placed in two pools, those with access to author names and affiliation, and those with no access. For each paper, two reviewers from each pool were assigned. This information asymmetry persisted through the review steps of bidding for reviews, reviewing assigned papers, and scoring them. Reviewers in the single-blind condition typically bid for 22% fewer papers, and bid preferentially for papers from top institutions. Single-blind reviewers were significantly more likely than their double-blind counterparts to recommend acceptance for papers from famous authors and top institutions: Odds multipliers for famous authors and top universities and companies are 1.63, 1.58, and 2.10, (0.61, 0.57, and 0.92 paper quality standard deviations), respectively. For female authors, the associated odds multiplier of 0.78 (-0.31 quality standard deviations) is not statistically significant. However, a meta-analysis places this value in line with those of similar studies; in this wider context, the gender effect is also statistically significant.