Conformal prediction

December 26, 2016 — May 5, 2023

Bayes
statistics
stochastic processes
surrogate
uncertainty
Figure 1

Predicting with competence: the best machine learning idea you never heard of from renowned passive-aggressive grumpy bastard Scott Locklin (Sorry Scott, but you are so reliably objectionable that I am always going to need to put a disclaimer on links to you, why do you refer to female scientists as “this woman”?):

The essential idea is that a “conformity function” exists. Effectively you are constructing a sort of multivariate cumulative distribution function for your machine learning gizmo using the conformity function. Such CDFs exist for classical stuff like ARIMA and linear regression under the correct circumstances; CP brings the idea to machine learning in general, and to models like ARIMA when the standard parametric confidence intervals won’t work.

Cosma Shalizi recommends Samii’s Conformal Inference Tutorial and Lei et al. (2017), because he felt Vovk, Gammerman, and Shafer (2005) was badly written. Maybe Shafer’s tutorial is good? (Shafer and Vovk 2008). Modern takes in Alvarsson et al. (2021);Zeni, Fontana, and Vantini (2020) and A Tutorial on Conformal Prediction plus accompanying video (Angelopoulos and Bates 2022).

Emmanuel Candés’ Neurips keynote on Conformal Prediction in 2022 was good.

Question: how does conformal prediction work under dataset shift (Tibshirani et al. 2019; Barber et al. 2023)?

1 Incoming

  • time: Lin, Trivedi, and Sun (2022)

2 References

Alvarsson, Arvidsson McShane, Norinder, et al. 2021. “Predicting With Confidence: Using Conformal Prediction in Drug Discovery.” Journal of Pharmaceutical Sciences.
Angelopoulos, and Bates. 2022. “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification.”
———. 2023. “Conformal Prediction: A Gentle Introduction.” Foundations and Trends® in Machine Learning.
Barber, Candes, Ramdas, et al. 2023. “Conformal Prediction Beyond Exchangeability.”
Barber, Candès, Ramdas, et al. 2021. “Predictive Inference with the Jackknife+.” The Annals of Statistics.
Bastani, Gupta, Jung, et al. 2022. “Practical Adversarial Multivalid Conformal Prediction.”
Card, Zhang, and Smith. 2019. “Deep Weighted Averaging Classifiers.” In Proceedings of the Conference on Fairness, Accountability, and Transparency.
Efron. 2021. “Resampling Plans and the Estimation of Prediction Error.” Stats.
Fontana, Zeni, and Vantini. 2023. “Conformal Prediction: A Unified Review of Theory and New Challenges.” Bernoulli.
Gibbs, and Candès. 2022. “Conformal Inference for Online Prediction with Arbitrary Distribution Shifts.”
Hu, Musielewicz, Ulissi, et al. 2022. “Robust and Scalable Uncertainty Estimation with Conformal Prediction for Machine-Learned Interatomic Potentials.” Machine Learning: Science and Technology.
Lei, G’Sell, Rinaldo, et al. 2017. “Distribution-Free Predictive Inference For Regression.”
Lin, Trivedi, and Sun. 2022. “Conformal Prediction Intervals with Temporal Dependence.”
Norinder, Carlsson, Boyer, et al. 2014. “Introducing Conformal Prediction in Predictive Modeling. A Transparent and Flexible Alternative to Applicability Domain Determination.” Journal of Chemical Information and Modeling.
Romano, Patterson, and Candes. 2019. “Conformalized Quantile Regression.” In Advances in Neural Information Processing Systems.
Shafer, and Vovk. 2008. “A Tutorial on Conformal Prediction.” Journal of Machine Learning Research.
Tibshirani, Foygel Barber, Candes, et al. 2019. “Conformal Prediction Under Covariate Shift.” In Advances in Neural Information Processing Systems.
Vovk, Gammerman, and Shafer. 2005. Algorithmic Learning in a Random World.
Vovk, Nouretdinov, and Gammerman. 2009. “On-Line Predictive Linear Regression.” The Annals of Statistics.
Zeni, Fontana, and Vantini. 2020. “Conformal Prediction: A Unified Review of Theory and New Challenges.” arXiv:2005.07972 [Cs, Econ, Stat].