Model Selection Bias Invalidates Goodness of Fit Tests

• Activity Number: 601 - Recent Advances in Variable Selection for Linear and Nonlinear Models
• Type: Topic Contributed
• Date/Time: Thursday, August 1, 2019 : 8:30 AM to 10:20 AM
• Sponsor: Biometrics Section
• Author(s): Joshua Loftus*
• Companies: New York University
• Keywords: Selective inference; Model selection; Goodness of fit; Unbiased tests

Abstract

We study goodness of fit tests in a variety of model selection settings and find that selection bias generally makes such tests conservative. Since selection methods choose the "best" model, a goodness of fit test will usually fail to reject, even when the incorrect model has been chosen. This is troubling, as it implies these tests in practice do not actually provide evidence in favor of the chosen model. We also explore methods of post selection inference to obtain conditionally unbiased goodness of fit tests and show how these outperform the unadjusted tests.