Neural PDE operator learning on domains with interesting geometry

Can ML solve PDEs on non-Euclidean domains?

October 15, 2019 — March 7, 2025

calculus
dynamical systems
geometry
Hilbert space
how do science
Lévy processes
machine learning
neural nets
PDEs
physics
regression
sciml
SDEs
signal processing
statistics
statmech
stochastic processes
surrogate
time series
uncertainty

Suspiciously similar content

Figure 1

Placeholder for a discussion of how to apply machine learning to partial differential equations on interesting weirdly-shaped domains, by which I mean things that are not toruses, spheres, or hyper-rectangles.

Why does this even need saying?, you might ask. Surely most PDEs are solved on interestingly-shaped objects? That would indicate that you come from a PDE background. On the ML side, the ascendancy of neural nets means that per default everything is on non-challenging geometries. The field has seen early success on rasterised grids (and spheres and toruses) which are easy to analysis and convenient to compute efficiently. The next step is to generalise to more interesting geometries and consider the implications of these generalisations. That is what this notebook is about.

1 Mapping Euclidean methods to non-Euclidean domains

This seems to be mostly the implicit representation to learn diffeomorphisms to map between a euclidean grid and a non-euclidean domain.

2 Masking

TODO

3 Transformers

TODO

4 Graph neural operators

TODO

5 DeepONets

TODO

6 References

Abueidda, Pantidis, and Mobasher. 2024. “DeepOKAN: Deep Operator Network Based on Kolmogorov Arnold Networks for Mechanics Problems.”
Alkin, Fürst, Schmid, et al. 2024. “Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operators.”
Bastek, Sun, and Kochmann. 2024. “Physics-Informed Diffusion Models.”
Batlle, Darcy, Hosseini, et al. 2023. “Kernel Methods Are Competitive for Operator Learning.” SSRN Scholarly Paper.
Blechschmidt, and Ernst. 2021. “Three Ways to Solve Partial Differential Equations with Neural Networks — A Review.” GAMM-Mitteilungen.
Bodnar, Bruinsma, Lucic, et al. 2024. “Aurora: A Foundation Model of the Atmosphere.”
Bonev, Kurth, Hundt, et al. 2023. “Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere.” In Proceedings of the 40th International Conference on Machine Learning. ICML’23.
Boullé, Halikias, and Townsend. 2023. “Elliptic PDE Learning Is Provably Data-Efficient.” Proceedings of the National Academy of Sciences.
Brandstetter, Berg, Welling, et al. 2022. “Clifford Neural Layers for PDE Modeling.” In.
Brandstetter, Worrall, and Welling. 2022. “Message Passing Neural PDE Solvers.” In International Conference on Learning Representations.
Brunton, and Kutz. 2024. “Promising Directions of Machine Learning for Partial Differential Equations.” Nature Computational Science.
Cao, Shuhao. 2021. “Choose a Transformer: Fourier or Galerkin.” In Advances in Neural Information Processing Systems.
Cao, Qianying, Goswami, and Karniadakis. 2023. “LNO: Laplace Neural Operator for Solving Differential Equations.”
Chen, Li, Long, et al. 2024. “Pseudo Physics-Informed Neural Operators.”
Cockayne, Oates, Sullivan, et al. 2016. “Probabilistic Numerical Methods for Partial Differential Equations and Bayesian Inverse Problems.”
———, et al. 2017. “Probabilistic Numerical Methods for PDE-Constrained Bayesian Inverse Problems.” In AIP Conference Proceedings.
Faroughi, Pawar, Fernandes, et al. 2023. “Physics-Guided, Physics-Informed, and Physics-Encoded Neural Networks in Scientific Computing.”
Frerix, Kochkov, Smith, et al. 2021. “Variational Data Assimilation with a Learned Inverse Observation Operator.” In.
Gajjar, Hegde, and Musco. 2022. “Provable Active Learning of Neural Networks for Parametric PDEs.” In.
George, Zhao, Kossaifi, et al. 2024. “Incremental Spatial and Spectral Learning of Neural Operators for Solving Large-Scale PDEs.”
Ghattas, and Willcox. 2021. “Learning Physics-Based Models from Data: Perspectives from Inverse Problems and Model Reduction.” Acta Numerica.
Gilpin. 2023. “Model Scale Versus Domain Knowledge in Statistical Forecasting of Chaotic Systems.” Physical Review Research.
Goswami, Bora, Yu, et al. 2022. “Physics-Informed Deep Neural Operator Networks.”
Grohs, and Herrmann. 2022. “Deep Neural Network Approximation for High-Dimensional Elliptic PDEs with Boundary Conditions.” IMA Journal of Numerical Analysis.
Guibas, Mardani, Li, et al. 2021. “Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers.”
Haitsiukevich, Poyraz, Marttinen, et al. 2024. “Diffusion Models as Probabilistic Neural Operators for Recovering Unobserved States of Dynamical Systems.”
Herde, Raonić, Rohner, et al. 2024. “Poseidon: Efficient Foundation Models for PDEs.”
Holzschuh, Vegetti, and Thuerey. 2022. “Score Matching via Differentiable Physics.”
Howard, Jacob, Murphy, et al. 2024. “Finite Basis Kolmogorov-Arnold Networks: Domain Decomposition for Data-Driven and Physics-Informed Problems.”
Huang, Shi, Gao, et al. 2024. “LordNet: An Efficient Neural Network for Learning to Solve Parametric Partial Differential Equations Without Simulated Data.” Neural Networks.
Jin, Zhang, and Espinosa. 2023. “Recent Advances and Applications of Machine Learning in Experimental Solid Mechanics: A Review.”
Jouvet, and Cordonnier. 2023. “Ice-Flow Model Emulator Based on Physics-Informed Deep Learning.” Journal of Glaciology.
Kadri, Duflos, Preux, et al. 2016. “Operator-Valued Kernels for Learning from Functional Response Data.” The Journal of Machine Learning Research.
Karlbauer, Praditia, Otte, et al. 2022. “Composing Partial Differential Equations with Physics-Aware Neural Networks.”
Kidger. 2022. “On Neural Differential Equations.”
Kita, Dubiński, Rokita, et al. 2024. “Generative Diffusion Models for Fast Simulations of Particle Collisions at CERN.”
Kochkov, Sanchez-Gonzalez, Smith, et al. 2020. “Learning Latent FIeld Dynamics of PDEs.” In Machine Learning and the Physical Sciences Workshop at the 33rd Conference on Neural Information Processing Systems (NeurIPS).
Koenig, Kim, and Deng. 2024. “KAN-ODEs: Kolmogorov–Arnold Network Ordinary Differential Equations for Learning Dynamical Systems and Hidden Physics.” Computer Methods in Applied Mechanics and Engineering.
Kossaifi, Kovachki, Azizzadenesheli, et al. 2023. “Multi-Grid Tensorized Fourier Neural Operator for High-Resolution PDEs.”
Kovachki, Lanthaler, and Mishra. 2021. “On Universal Approximation and Error Bounds for Fourier Neural Operators.” arXiv:2107.07562 [Cs, Math].
Kovachki, Li, Liu, et al. 2021. “Neural Operator: Learning Maps Between Function Spaces.” In arXiv:2108.08481 [Cs, Math].
———, et al. 2023. “Neural Operator: Learning Maps Between Function Spaces With Applications to PDEs.” Journal of Machine Learning Research.
Krämer, Schmidt, and Hennig. 2022. “Probabilistic Numerical Method of Lines for Time-Dependent Partial Differential Equations.” In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics.
Lam, Sanchez-Gonzalez, Willson, et al. 2023a. “GraphCast: Learning Skillful Medium-Range Global Weather Forecasting.”
———, et al. 2023b. “Learning Skillful Medium-Range Global Weather Forecasting.” Science.
Lanthaler, and Stuart. 2023. “The Parametric Complexity of Operator Learning.”
Li, Huang, Liu, et al. 2022. “Fourier Neural Operator with Learned Deformations for PDEs on General Geometries.”
Li, Kovachki, Azizzadenesheli, Liu, Stuart, et al. 2020. “Multipole Graph Neural Operator for Parametric Partial Differential Equations.” In Advances in Neural Information Processing Systems.
Li, Kovachki, Azizzadenesheli, Liu, Bhattacharya, et al. 2020a. “Neural Operator: Graph Kernel Network for Partial Differential Equations.” In.
———, et al. 2020b. “Fourier Neural Operator for Parametric Partial Differential Equations.” arXiv:2010.08895 [Cs, Math].
Liu, Ziming, Ma, Wang, et al. 2024. “KAN 2.0: Kolmogorov-Arnold Networks Meet Science.”
Liu, Xiao, Yeo, and Lu. 2020. “Statistical Modeling for Spatio-Temporal Data From Stochastic Convection-Diffusion Processes.” Journal of the American Statistical Association.
Li, Zheng, Kovachki, et al. 2021. “Physics-Informed Neural Operator for Learning Partial Differential Equations.”
Long, Lu, Ma, et al. 2018. “PDE-Net: Learning PDEs from Data.” In Proceedings of the 35th International Conference on Machine Learning.
Lu. 2020. “Theory, Algorithms, and Software for Physics-Informed Deep Learning.”
Lu, Jin, and Karniadakis. 2020. “DeepONet: Learning Nonlinear Operators for Identifying Differential Equations Based on the Universal Approximation Theorem of Operators.” arXiv:1910.03193 [Cs, Stat].
Lu, Meng, Cai, et al. 2022. “A Comprehensive and Fair Comparison of Two Neural Operators (with Practical Extensions) Based on FAIR Data.” Computer Methods in Applied Mechanics and Engineering.
Lu, Meng, Mao, et al. 2021. “DeepXDE: A Deep Learning Library for Solving Differential Equations.” SIAM Review.
Luo, O’Leary-Roseberry, Chen, et al. 2023. “Efficient PDE-Constrained Optimization Under High-Dimensional Uncertainty Using Derivative-Informed Neural Operators.”
MacKinlay, Pagendam, Kuhnert, et al. 2021. “Model Inversion for Spatio-Temporal Processes Using the Fourier Neural Operator.” In.
Magnani, Krämer, Eschenhagen, et al. 2022. “Approximate Bayesian Neural Operators: Uncertainty Quantification for Parametric PDEs.”
Magnani, Pförtner, Weber, et al. 2024. “Linearization Turns Neural Operators into Function-Valued Gaussian Processes.”
Mahesh, Collins, Bonev, et al. 2024. “Huge Ensembles Part I: Design of Ensemble Weather Forecasts Using Spherical Fourier Neural Operators.”
Meuris, Qadeer, and Stinis. 2023. “Machine-Learning-Based Spectral Methods for Partial Differential Equations.” Scientific Reports.
Mialon, Garrido, Lawrence, et al. 2024. “Self-Supervised Learning with Lie Symmetries for Partial Differential Equations.”
Mora, Yousefpour, Hosseinmardi, and Bostanabad. 2024. “Neural Networks with Kernel-Weighted Corrective Residuals for Solving Partial Differential Equations.” arXiv Preprint arXiv:2401.03492.
Mora, Yousefpour, Hosseinmardi, Owhadi, et al. 2024. “Operator Learning with Gaussian Processes.”
Nüsken, and Richter. 2021. “Solving High-Dimensional Hamilton–Jacobi–Bellman PDEs Using Neural Networks: Perspectives from the Theory of Controlled Diffusions and Measures on Path Space.” Partial Differential Equations and Applications.
Oates, Cockayne, Aykroyd, et al. 2019. “Bayesian Probabilistic Numerical Methods in Time-Dependent State Estimation for Industrial Hydrocyclone Equipment.” Journal of the American Statistical Association.
Opschoor, Petersen, and Schwab. 2020. “Deep ReLU Networks and High-Order Finite Element Methods.” Analysis and Applications.
Pathak, Subramanian, Harrington, et al. 2022. “Fourcastnet: A Global Data-Driven High-Resolution Weather Model Using Adaptive Fourier Neural Operators.”
Pestourie, Mroueh, Rackauckas, et al. 2022. “Physics-Enhanced Deep Surrogates for PDEs.”
Poli, Massaroli, Berto, et al. 2022. “Transform Once: Efficient Operator Learning in Frequency Domain.” In Advances in Neural Information Processing Systems.
Psaros, Meng, Zou, et al. 2023. “Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons.” Journal of Computational Physics.
Ronneberger, Fischer, and Brox. 2015. “U-Net: Convolutional Networks for Biomedical Image Segmentation.” Edited by Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Lecture Notes in Computer Science.
Ruhe, Gupta, de Keninck, et al. 2023. “Geometric Clifford Algebra Networks.” In arXiv Preprint arXiv:2302.06594.
Saha, and Balamurugan. 2020. “Learning with Operator-Valued Kernels in Reproducing Kernel Krein Spaces.” In Advances in Neural Information Processing Systems.
Takamoto, Praditia, Leiteritz, et al. 2022. “PDEBench: An Extensive Benchmark for Scientific Machine Learning.” In.
Tran, Mathews, Xie, et al. 2022. “Factorized Fourier Neural Operators.”
Wang, J., Cockayne, and Oates. 2018. “On the Bayesian Solution of Differential Equations.” In 38th International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering.
Wang, Rui, Kashinath, Mustafa, et al. 2020. “Towards Physics-Informed Deep Learning for Turbulent Flow Prediction.” In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. KDD ’20.
Wang, Yizheng, Sun, Bai, et al. 2024. “Kolmogorov Arnold Informed Neural Network: A Physics-Informed Deep Learning Framework for Solving Forward and Inverse Problems Based on Kolmogorov Arnold Networks.”
Wen, Li, Azizzadenesheli, et al. 2022. “U-FNO—An Enhanced Fourier Neural Operator-Based Deep-Learning Model for Multiphase Flow.” Advances in Water Resources.
Witman, New, Alkendry, et al. 2022. “Neural Basis Functions for Accelerating Solutions to High Mach Euler Equations.”
Xu, Kunpeng, Chen, and Wang. 2024. “Kolmogorov-Arnold Networks for Time Series: Bridging Predictive Power and Interpretability.”
Xu, Kailai, and Darve. 2020. “ADCME: Learning Spatially-Varying Physical Fields Using Deep Neural Networks.” In arXiv:2011.11955 [Cs, Math].
Yin, Yuan, Kirchmeyer, Franceschi, et al. 2023. “Continuous PDE Dynamics Forecasting with Implicit Neural Representations.”
Yin, Minglang, Zhang, Yu, et al. 2022. “Interfacing Finite Elements with Deep Neural Operators for Fast Multiscale Modeling of Mechanics Problems.” Computer Methods in Applied Mechanics and Engineering, A Special Issue in Honor of the Lifetime Achievements of J. Tinsley Oden,.
Zhou, Anthony, Lorsung, Hemmasian, et al. 2024. “Strategies for Pretraining Neural Operators.”
Zhou, Hang, Ma, Wu, et al. 2024. “Unisolver: PDE-Conditional Transformers Are Universal PDE Solvers.”