Online Program Home
My Program

Abstract Details

Activity Number: 139 - Competing Effectively: Hosting, Designing, and Participating in Kaggle-Style Competitions
Type: Invited
Date/Time: Monday, July 30, 2018 : 10:30 AM to 12:20 PM
Sponsor: Section on Statistics in Defense and National Security
Abstract #326778 Presentation
Title: Bayesian Design of Experiments with Multiple Priors for Kaggle Competition Design
Author(s): Kevin Randal Quinlan* and Christine M Anderson-Cook
Companies: The Pennsylvania State University and Los Alamos National Laboratory
Keywords: Design of Experiments; Bayesian D-optimal Designs; Logistic Regression; Kaggle Competition

When assessing the performance of multiple algorithms in a competition setting, it is desirable to construct a design that considers a variety of performance levels across the algorithms. We describe a strategy to use Bayesian design of experiments with multiple prior estimates to capture anticipated performance. Our goal is to characterize results from the different algorithms as a function of different explanatory variables and use this to help choose a design about which units to test. We use this approach to develop methodology for the case where there are several potentially non-overlapping priors under consideration. While multiple priors have been used for analysis in the past, they have not been used in a design context. The Weighted Priors method performs well for a broad range of true underlying model parameter choices and is more robust when compared to other reasonable design choices. We illustrate the method as well through multiple scenarios and a motivating example. Additionally, we propose multiple new plots which are useful for evaluating design performance in higher dimensional problems.

Authors who are presenting talks have a * after their name.

Back to the full JSM 2018 program