Some theory on Bayesian neural networks by Julyan Arbel (Mistis, Grenoble)

In this talk, we first present seminal works at the basis of the theory of Bayesian neural networks. These include Radford Neal result in the 90s regarding the connexion between Gaussian processes and wide neural networks, and the recent developments of this result to deep neural networks. In a second part, we focus on understanding priors in Bayesian neural networks at the unit level. More specifically, we investigate deep Bayesian neural networks with Gaussian weight priors and a class of ReLU-like nonlinearities. We establish that the induced prior distribution on the units before and after activation becomes increasingly heavy-tailed with the depth of the layer.