Neural network-based techniques for solving differential equations can
be traced at least to the 1990s. The remarkable success of deep
learning in the last decade has stimulated a significant amount of
efforts in the development of deep neural network (DNN) based PDE
solvers. DNN-based PDE solvers have witnessed a robust and dramatic
growth in the past few years, with several successful ones emerging.
Can neural network-based methods out-compete traditional numerical
methods in computational performance for solving PDEs? This is a
question hanging in the air ever since the early studies of neural
networks for differential equations and intriguing both computational
mathematicians and machine learning practitioners. Here by
"out-compete" we mean that one method achieves a better
accuracy under the same computational budget/cost or incurs a lower
computational cost to achieve the same accuracy. While their
computational performance is promising, the existing DNN-based PDE
solvers suffer from several limitations, which make them numerically
less than satisfactory and computationally uncompetitive. The most
prominent include the limited accuracy, a general lack of convergence
with a certain convergence rate, and extremely high computational cost
(very long time to train). Due to these limitations, these solvers
seem to fall short, at least in their current state, and cannot
compete with traditional numerical methods, except perhaps for certain
problems such as high-dimensional problems.
In this talk we discuss a neural network-based method (termed local
extreme learning machines, or locELM) for solving linear and nonlinear
PDEs that exhibits a disparate computational performance from the
above DNN-based PDE solvers and in a sense overcomes the above
limitations. This method combines the ideas of extreme learning
machines, domain decomposition and local neural networks. The field
solution on each sub-domain is represented by a local feed-forward
neural network, and \(C^k\) continuity is imposed on the sub-domain
boundaries. Each local neural network consists of a small number of
hidden layers, whose coefficients are pre-set to random values and
fixed through the computation, and the trainable parameters consist of
the output-layer coefficients. The overall neural network is trained
by a linear or nonlinear least squares computation, not by the
back-propagation (or gradient descent) type algorithms.
The presented method exhibits a clear sense of convergence with
respect to the degrees of freedom in the system. For smooth solutions
its numerical errors decrease exponentially as the number of training
parameters or the number of training data points increases, much like
the traditional spectral or spectral element type methods. We compare
the current locELM method with the state-of-the-art DNN-based PDE
solvers, such as the physics-informed neural network (PINN) and the
deep Galerkin method (DGM), and with the classical and high-order
finite element methods (FEM). The numerical errors and network
training time of locELM are considerably smaller, typically by orders
of magnitude, than those of PINN and DGM. We show evidence that the
current method far outperforms the classical 2nd-order FEM. The
computational performance of the presented method is comparable to
that of the high-order FEM for smaller problem sizes, and for larger
problem sizes it markedly outperforms the high-order FEM. A number of
numerical benchmarks will be presented to demonstrate these points.
|