Bayesian model selection
August 20, 2017 — November 19, 2024
Frequentist model selection is not the only type, although you can miss learning about this in introductory Bayesian statistics. What is model selection in a Bayesian context? Surely you don’t ever get some models with zero posterior probability? In my intro Bayesian classes, I learned that one simply keeps all the models weighted by posterior likelihood when making predictions. But sometimes we wish to get rid of some models. When does this work, and when not?
🏗
1 Sparsity
Setting: in the phenomenon we observe, we suspect that of the many regressors, most “don’t matter” in some sense. This is an interesting special case: See Bayesian sparsity.
2 Cross-validation and Bayes
There is a relation between cross-validation and Bayes evidence, a.k.a. marginal likelihood — see (Claeskens and Hjort 2008; Fong and Holmes 2020).
3 Evidence/marginal likelihood/type II maximum likelihood/Bayes factor
A classic; Worth its own notebook. See model selection by model evidence maximisation.
4 Incoming
John Mount on applied variable selection (Mount 2020)
We have also always felt a bit exposed in this, as feature selection seems unjustified in standard explanations of regression. One feels that if a coefficient were meant to be zero, the fitting procedure would have set it to zero. Under this misapprehension, stepping in and removing some variables feels unjustified.
Regardless of intuition or feelings, it is a fair question: is variable selection a natural justifiable part of modelling? Or is it something that is already done (therefore redundant)? Or is it something that is not done for important reasons (such as avoiding damaging bias)?
In this note we will show that feature selection is in fact an obvious justified step when using a sufficiently sophisticated model of regression. This note is long, as it defines so many tiny elementary steps. However this note ends with a big point: variable selection is justified. It naturally appears in the right variation of Bayesian Regression. You should select variables, using your preferred methodology. And you shouldn’t feel bad about selecting variables.