Energy based models

Inference with kinda-tractable un-normalized densities

June 7, 2021 — June 7, 2021

approximation

Bayes

generative

Monte Carlo

optimization

probabilistic algorithms

probability

statistics

statmech

I don’t actually know what fits under this heading, but it sounds like it is simply inference for undirected graphical models. Or is there something distinct going on?

Figure 1: Descending the local energy gradient to a more probable configuration

1 References

Bardes, Ponce, and LeCun. 2022. “VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning.”

Che, Zhang, Sohl-Dickstein, et al. 2020. “Your GAN Is Secretly an Energy-Based Model and You Should Use Discriminator Driven Latent Sampling.” arXiv:2003.06060 [Cs, Stat].

Clifford. 1990. “Markov random fields in statistics.” In Disorder in Physical Systems: A Volume in Honour of John Hammersley.

Hinton. 2010. “A Practical Guide to Training Restricted Boltzmann Machines.” In Neural Networks: Tricks of the Trade. Lecture Notes in Computer Science 7700.

LeCun, Chopra, Hadsell, et al. 2006. “A Tutorial on Energy-Based Learning.” In Predicting Structured Data.

Montavon, Müller, and Cuturi. 2016. “Wasserstein Training of Restricted Boltzmann Machines.” In Advances in Neural Information Processing Systems 29.

Pollard. 2004. “Hammersley-Clifford Theorem for Markov Random Fields.”

Salakhutdinov. 2015. “Learning Deep Generative Models.” Annual Review of Statistics and Its Application.