Score matching

November 11, 2021 — April 2, 2024

approximation
Bayes
Bregman
generative
Monte Carlo
neural nets
optimization
probabilistic algorithms
probability
score function
statistics

A way of learning a score function from data. Very famous now thanks to neural diffusions.

Figure 1

1 Basic

TBC

2 Sliced

Sliced Score Matching (Y. Song et al. 2019).

3 Noise Conditional

Yang Song, Generative Modeling by Estimating Gradients of the Data Distribution

4 Data assimilation

francois-rozet/sda: Official implementation of Score-based Data Assimilation implements (Rozet and Louppe 2023b) which I should read.

5 Incoming

Suggestive connection to thermodynamics (Sohl-Dickstein et al. 2015), score estimators in gradient, and Bregman divergences (Gutmann and Hirayama 2011).

6 References

Bao, Chipilski, Liang, et al. 2024. “Nonlinear Ensemble Filtering with Diffusion Models: Application to the Surface Quasi-Geostrophic Dynamics.”
Bao, Zhang, and Zhang. 2023. “A Score-Based Nonlinear Filter for Data Assimilation.”
———. 2024. “An Ensemble Score Filter for Tracking High-Dimensional Nonlinear Dynamical Systems.”
Dockhorn, Vahdat, and Kreis. 2022. “GENIE: Higher-Order Denoising Diffusion Solvers.” In.
Gutmann, and Hirayama. 2011. “Bregman Divergence as General Framework to Estimate Unnormalized Statistical Models.” In Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence. UAI’11.
Holzschuh, Vegetti, and Thuerey. 2022. “Score Matching via Differentiable Physics.”
Hyvärinen. 2005. “Estimation of Non-Normalized Statistical Models by Score Matching.” The Journal of Machine Learning Research.
Lim, Kovachki, Baptista, et al. 2023. “Score-Based Diffusion Models in Function Space.”
McAllester. 2023. “On the Mathematics of Diffusion Models.”
Rozet, and Louppe. 2023a. “Score-Based Data Assimilation.”
———. 2023b. “Score-Based Data Assimilation for a Two-Layer Quasi-Geostrophic Model.”
Schröder, Ou, Lim, et al. 2023. “Energy Discrepancies: A Score-Independent Loss for Energy-Based Models.”
Sharrock, Simons, Liu, et al. 2022. “Sequential Neural Score Estimation: Likelihood-Free Inference with Conditional Score Based Diffusion Models.”
Sohl-Dickstein, Weiss, Maheswaranathan, et al. 2015. “Deep Unsupervised Learning Using Nonequilibrium Thermodynamics.”
Song, Yang, and Ermon. 2020a. “Generative Modeling by Estimating Gradients of the Data Distribution.” In Advances In Neural Information Processing Systems.
———. 2020b. “Improved Techniques for Training Score-Based Generative Models.” In Advances In Neural Information Processing Systems.
Song, Yang, Garg, Shi, et al. 2019. “Sliced Score Matching: A Scalable Approach to Density and Score Estimation.”
Song, Jiaming, Meng, and Ermon. 2021. “Denoising Diffusion Implicit Models.” arXiv:2010.02502 [Cs].
Song, Yang, Sohl-Dickstein, Kingma, et al. 2022. “Score-Based Generative Modeling Through Stochastic Differential Equations.” In.
Swersky, Ranzato, Buchman, et al. 2011. “On Autoencoders and Score Matching for Energy Based Models.” In Proceedings of the 28th International Conference on Machine Learning (ICML-11).
Tran, Rossi, Milios, et al. 2021. “Model Selection for Bayesian Autoencoders.” In Advances in Neural Information Processing Systems.
Vincent. 2011. “A connection between score matching and denoising autoencoders.” Neural Computation.
Zhuang, Abnar, Gu, et al. 2022. “Diffusion Probabilistic Fields.” In.