Functional Priors for Bayesian Deep Learning

• Activity Number: 73 - SPEED: Statistical Computing and Statistics in Genomics Part 2
• Type: Contributed
• Date/Time: Sunday, August 7, 2022 : 5:05 PM to 5:50 PM
• Sponsor: Section on Statistical Computing
• Author(s): Ba-Hien Tran* and Simone Rossi and Dimitrios Milios and Pietro Michiardi and Maurizio Filippone
• Companies: EURECOM and EURECOM and EURECOM and EURECOM and EURECOM
• Keywords: Bayesian inference; Gaussian processes; neural networks; Wasserstein distance; prior distribution

Abstract

The Bayesian treatment of neural networks dictates that a prior distribution is specified over their weight and bias parameters. This poses a challenge because modern neural networks are characterized by a large number of parameters and non-linearities. The choice of these priors has an unpredictable effect on the distribution of the functional output which could represent a hugely limiting aspect of Bayesian deep learning models. Differently, Gaussian processes offer a rigorous non-parametric framework to define prior distributions over the space of functions. With this contributed presentation, we aim to introduce a novel and robust framework to impose such functional priors on modern neural networks through minimizing the Wasserstein distance between samples of stochastic processes. We provide extensive experimental evidence that coupling these priors with scalable Markov chain Monte Carlo sampling offers systematically large performance improvements over alternative choices of priors and state-of-the-art approximate Bayesian deep learning approaches.