Speaker
Description
Natural Gradient Descent (NGD) has recently gained attention as an effective optimization approach for deep-learning-based solvers of partial differential equations (PDEs), particularly Physics-Informed Neural Networks (PINNs). By leveraging the geometric structure of the neural network parameter manifold, NGD can achieve substantially faster convergence in terms of iteration count compared to standard first-order optimization methods. However, widespread practical use has been hindered by the high computational cost of constructing and inverting the Gramian matrix, which scales cubically with the number of network parameters.
In this talk, we introduce a computationally efficient NGD framework for neural PDE solvers that addresses these challenges through a combination of matrix-free formulations and low-rank preconditioning techniques. We generalize matrix-free NGD to a wide class of neural PDE models, including PINNs, Variational PINNs, Finite Element Interpolated Neural Networks, and Robust VPINNs, as well as to general choices of underlying metrics. Exploiting the empirically observed low-rank structure of the Gramian matrix, we design preconditioners based on randomized numerical linear algebra methods, including Nyström approximations and partial pivoted Cholesky factorizations. These approaches significantly accelerate convergence of the inner iterative solvers while maintaining manageable memory and computational requirements.
We provide a systematic comparison of multiple NGD variants—explicit inversion, unpreconditioned matrix-free schemes, and several preconditioned strategies—analyzing both theoretical complexity and practical performance to identify regimes where each method is most effective. We further discuss efficient implementations based on automatic differentiation and offer practical guidelines for integrating NGD into existing optimization and autodiff frameworks. Finally, we benchmark the proposed methods against state-of-the-art optimizers across a range of PDE problems, demonstrating notable reductions in training time alongside improved accuracy and robustness. Overall, the results establish low-rank preconditioned NGD as a scalable and competitive optimization paradigm for modern neural PDE solvers.