Research at the Interface of Applied Mathematics and Machine Learning

Monday, 12/08/2025

Tuesday, 12/09/2025

Wednesday, 12/10/2025

Thursday, 12/11/2025

This lecture sets the main notation for this conference and discusses different forms of ML that are relevant to this workshop. For simplicity, we define learning as designing a function or operator $f_\theta$ and learning its weights $\theta$ to accomplish a given task. In this workshop, we focus on $f_\theta$ as a neural network. This lecture overviews terminology and different learning tasks in applied mathematics. The remaining lectures in this module discuss the two crucial aspects of learning in more detail. To give participants a broad context, we provide motivating examples from unsupervised, semi-supervised, supervised learning, generative modeling, reinforcement learning, and operator learning. Where possible, we link the examples to corresponding problems in applied mathematics. For example, supervised learning corresponds to data fitting, generative modeling has links to optimal transport, reinforcement learning is tied to optimal control, and operator learning arises in solving PDEs and inverse problems.

This lecture is devoted to different ways to design the neural network architecture that defines $f_\theta$. This is a crucial step in solving practical problems. We review classical multilayer perceptron architectures, where $f_{\theta}$ is defined by concatenating affine transformations and pointwise nonlinear activation functions. While these architectures are universal function approximators and can be effective in many applications, they have difficulties approximating simple functions, like the identity mapping, and their training is challenging with increasing depth. Adding skip connections, as done in residual networks, can overcome this disadvantage and be trained with hundreds or thousands of layers. The latter architectures provide links to optimal control and will be revisited in the next module. We also present graph neural networks, which are crucial to handling unstructured data, and give a mathematical description of transformers, including their attention mechanism.

This lecture introduces the loss functions that can be used to train the neural network architectures for a given task. For supervised learning, we discuss regression and cross-entropy losses. We discuss maximum likelihood training and variational inference with the empirical lower bound (ELBO) for generative modeling. As an example of unsupervised learning, we discuss PDE losses in PINNs. We illustrate the difference between minimizing the loss function and learning, which requires generalization to unseen data, using examples from polynomial data fitting that most participants will recall. We then provide further insights by discussing the integration error of Monte Carlo approximation of the loss.

Having defined the ML model and loss functions, this lecture discusses optimization algorithms that can be used to identify weights. Since the loss functions usually involve high-dimensional integrals, we approximate them using Monte Carlo integration. This naturally leads to stochastic optimization algorithms such as stochastic gradient descent and its variants. In this lecture, we discuss convergence properties, theoretical and empirical results that show convergence to global minimizers for highly nonconvex functions, and their ability to regularize the problem.

This lecture investigates the relation between generalization and regularization in more depth. Building upon advances in applied mathematics, we discuss iterative regularization, direct regularization, and hybrid approaches in the context of ill-posed inverse problems. From this perspective, we show new insights into the double-descent phenomenon arising in modern ML. We illustrate this using the random feature models and demonstrate that adequate regularization can help those models generalize to unseen data. We also review recent results on the benefits and challenges of adding regularization theory into stochastic optimization schemes.

This lecture surveys neural network architectures whose depth corresponds to artificial time and whose forward propagation is given by differential equations. As a starting point, we view the infinite-depth limit residual networks as forward Euler discretizations of a nonstationary, nonlinear initial value problem. We discuss the theoretical implications for supervised learning and the opportunities in generative modeling. We present extensions to PDEs when the features correspond to image, speech, or video data and the benefits of generalizing the framework to stochastic dynamics. The latter allows us to discuss image-generation algorithms like Dalle-2 and other techniques based on score-based diffusion.

This lecture demonstrates the use cases and challenges of using ML in scientific contexts. Unlike traditional big-data applications, specific challenges arise from scarce datasets, the complex nature of observations, the lack of similar experience, which makes finding hyperparameters difficult, and so on. We also overview important research directions not covered in the remaining lectures, such as PINNs, neural operator learning, and reinforcement learning.

This lecture introduces several high-dimensional PDE problems for which ML techniques provide promising avenues. While many PDE problems are phrased in physical coordinates and are thus limited to three dimensions plus time, there are several areas in which the dimensions become considerably higher, and the curse of dimensionality limits traditional numerical techniques. We use examples including the Black Scholes equations in finance and Hamilton Jacobi Bellman equations in optimal control. In those cases, the curse of dimensionality can be mitigated (but not entirely overcome) by appropriate numerical integration, adequate neural network models, and training.

Due to their ill-posedness and (in some cases) abundance of training data, applying ML techniques to improve the solution of inverse problems has been a promising research direction. In this lecture, we will first demonstrate that approximating the inverse map with a neural network in a supervised way in ill-posed inverse problems leads to unstable approximations. We then consider Bayesian formulations of the inverse problem as they rely on modeling and analyzing complex, high-dimensional probability distributions. This provides links to generative modeling, and we show how those techniques can overcome the limitations of the traditional Bayesian setting and improve computational efficiency.

This lecture highlights recent trends in mathematics that employ large language models ({\bf LLM}s) to discover and reason about mathematics. For example, combining proof assistants with LLMs assists in formalizing mathematical theorems, proofs, and even sketch proofs. Another example is the use of reinforcement learning to discover counter-examples in combinatorics. We discuss funsearch and its ability to develop interpretable solutions to challenging discrete math problems. In the context of computational mathematics, reinforcement learning is noteworthy for discovering more efficient implementations of matrix-matrix products.

Panel (Academia)

TBA

Panel (Industry)

TBA