|
Abstract:
|
In the supervised learning context, black-box learning methods are often viewed as sacrificing interpretability for predictive accuracy. The complex forms of the model estimates preclude many traditional approaches to statistical inference. While heuristic approaches have been developed for tasks like measuring variable importance, these are ad hoc, lack statistical justification, and can produce quite misleading results. In this work, we begin to bridge this gap by developing provably valid hypothesis tests for comparing models trained on different inputs. This allows for testing a model trained on the original covariates against one trained on a randomized subset in order to formally test importance. While closed-form distributional results are newly available for particular models like random forests, practical implementation is limited by exceptionally difficult parameter estimation problems. We thus demonstrate how a permutation test approach circumvents these challenges, producing valid inference with high power with orders of magnitude less computational overhead.
|
