Bayes NNs where only some weights are random and others are fixed. This raises various difficulties — how do you update a fixed parameter? It sounds like a sparse Bayes problem, but whereas in sparse bayes we wish to audition interpretable regressors for inclusion in the model, here we wish to audition uninterpretable, unidentifiable weights for inclusion in the model as random variables, but ultimately all weights are included either as random variates or deterministic parameters.
Moving target alert! No-one agrees what to call them. For now I use the emerging pBNNs, aka “partial Bayesian neural networks” (Zhao et al. 2024) which seems like an acceptable term.
Is this even principled?
It sounds like it should be, but it is a little fiddly in practice. How would we interpret the “posterior” of a fixed parameter? Surely there is some kind of variational argument?
Try Sharma et al. (2022) for a start.
How to update a deterministic parameter?
From the perspective of Bayes inference, parameters we do not update have zero prior variance. And yet we do update them by SGD. What does that mean? How can we make that statistically well-posed?
Via sequential Monte Carlo?
Zhao et al. (2024) is an elegant paper which shows how to train a pBNN using sequential Monte Carlo. This may be the definitive method; it’s the least ad hoc option here.
Probabilistic weight tying
Possibly also in effect a form of pBNN. Rafael Oliveira has referred me to Roth and Pernkopf (2020) for some ideas about this.
References
Daxberger, Nalisnick, Allingham, et al. 2020. “Expressive yet Tractable Bayesian Deep Learning via Subnetwork Inference.” In.
Daxberger, Nalisnick, Allingham, et al. 2021.
“Bayesian Deep Learning via Subnetwork Inference.” In
Proceedings of the 38th International Conference on Machine Learning.
Durasov, Bagautdinov, Baque, et al. 2021.
“Masksembles for Uncertainty Estimation.” In
2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Dusenberry, Jerfel, Wen, et al. 2020.
“Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors.” In
Proceedings of the 37th International Conference on Machine Learning.
Izmailov, Maddox, Kirichenko, et al. 2020.
“Subspace Inference for Bayesian Deep Learning.” In
Proceedings of The 35th Uncertainty in Artificial Intelligence Conference.
Page, Bhattacharya, and Dunson. 2013.
“Classification via Bayesian Nonparametric Learning of Affine Subspaces.” Journal of the American Statistical Association.
Roth, and Pernkopf. 2020.
“Bayesian Neural Networks with Weight Sharing Using Dirichlet Processes.” IEEE Transactions on Pattern Analysis and Machine Intelligence.
Spantini, Cui, Willcox, et al. 2017.
“Goal-Oriented Optimal Approximations of Bayesian Linear Inverse Problems.” SIAM Journal on Scientific Computing.
Spantini, Solonen, Cui, et al. 2015.
“Optimal Low-Rank Approximations of Bayesian Linear Inverse Problems.” SIAM Journal on Scientific Computing.
Thomas, You, Lin, et al. 2022.
“Learning Subspaces of Different Dimensions.” Journal of Computational and Graphical Statistics.
Tran, M.-N., Nguyen, Nott, et al. 2019.
“Bayesian Deep Net GLM and GLMM.” Journal of Computational and Graphical Statistics.
Tran, Ba-Hien, Rossi, Milios, et al. 2022.
“All You Need Is a Good Functional Prior for Bayesian Deep Learning.” Journal of Machine Learning Research.
Zhao, Mair, Schön, et al. 2024.
“On Feynman-Kac Training of Partial Bayesian Neural Networks.” In
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics.
