Neural operators represent an extension of traditional artificial neural networks, marking a departure from the typical focus on learning mappings between finite-dimensional Euclidean spaces or finite sets.
[1] The primary application of neural operators is in learning surrogate maps for the solution operators of partial differential equations (PDEs),[1] which are critical tools in modeling the natural environment.
[2][3] Standard PDE solvers can be time-consuming and computationally intensive, especially for complex systems.
Neural operators have demonstrated improved performance in solving PDEs[4] compared to existing machine learning methodologies while being significantly faster than numerical solvers.
[5][6][7] Neural operators have also been applied to various scientific and engineering disciplines such as turbulent flow modeling, computational mechanics, graph-structured data,[8] and the geosciences.
[9] In particular, they have been applied to learning stress-strain fields in materials, classifying complex data like spatial transcriptomics, predicting multiphase flow in porous media,[10] and carbon dioxide migration simulations.
Finally, the operator learning paradigm allows learning maps between function spaces, and is different from parallel ideas of learning maps from finite-dimensional spaces to function spaces,[11][12] and subsumes these settings when limited to fixed input resolution.
Understanding and mapping relationships between function spaces has many applications in engineering and the sciences.
In particular, one can cast the problem of solving partial differential equations as identifying a map between function spaces, such as from an initial condition to a time-evolved state.
Using traditional machine learning methods, addressing this problem would involve discretizing the infinite-dimensional input and output function spaces into finite-dimensional grids and applying standard learning models, such as neural networks.
The primary properties of neural operators that differentiate them from traditional neural networks is discretization invariance and discretization convergence.
[1] Unlike conventional neural networks, which are fixed on the discretization of training data, neural operators can adapt to various discretizations without re-training.
This property improves the robustness and applicability of neural operators in different scenarios, providing consistent performance across different resolutions and grids.
Architecturally, neural operators are similar to feed-forward neural networks in the sense that they are composed of alternating linear maps and non-linearities.
are the lifting (lifting the codomain of the input function to a higher dimensional space) and projection (projecting the codomain of the intermediate function to the output codimension) operators, respectively.
These operators act pointwise on functions and are typically parametrized as multilayer perceptrons.
(usually parameterized by a pointwise neural network), a kernel integral operator
-th hidden layer, a kernel integral operator
is a learnable implicit neural network, parametrized by
In practice, one is often given the input function to the neural operator at a specific resolution.
is the sub-area volume or quadrature weight associated to the point
[13] There have been various parameterizations of neural operators for different applications.
The most popular instantiation is the Fourier neural operator (FNO).
and by applying the convolution theorem, arrives at the following parameterization of the kernel integral operator:
represents the Fourier transform of some periodic function
When the grid at which the input function is presented is uniform, the Fourier transform can be approximated using the discrete Fourier transform (DFT) with frequencies below some specified threshold.
Neural operators can be trained directly using backpropagation and gradient descent-based methods.
Another training paradigm is associated with physics-informed machine learning.
In particular, physics-informed neural networks (PINNs) use complete physics laws to fit neural networks to solutions of PDEs.
Extensions of this paradigm to operator learning are broadly called physics-informed neural operators (PINO),[14] where loss functions can include full physics equations or partial physical laws.