---
title: Operator Neural Networks (ONNs) Overview
url: https://www.emergentmind.com/topics/operator-neural-networks-onns
type: topic
---

# Operator Neural Networks (ONNs) Overview

Operational Neural Networks (ONNs) are a class of neural architectures designed to generalize and surpass traditional convolutional and fully connected networks by introducing flexible, heterogeneous, and often learnable nonlinear operators at the neuron or connection level. ONNs enable the direct modeling and learning of complex local or function-to-function mappings ("operators") relevant in both scientific computing and pattern analysis. Several ONN paradigms have been developed, ranging from patch-wise Taylor-expansion–based generative neurons (Self-ONNs) and library-based heterogeneous neurons (classic ONNs), to function-space operator networks such as DeepONet, RBON, and their domain-specific and ensemble extensions.

## 1. Operator Neural Networks: Definitions and Generalization

ONNs extend conventional neural architectures by replacing the standard linear weight/activation pipeline with a more general triple: nodal operator (nonlinear function of the weight and input), pool operator (aggregation), and activation. In ONNs, the pre-activation of a neuron is typically
$$
x_i^{l+1} = b_i^{l+1} + P_i^{l+1}\{Y_i^{l+1}(w_{ij}^{l+1}, y_j^{l}) : j=1,\ldots,N_l\},
$$
where \( Y \) is a (potentially nonlinear) nodal operator, \( P \) is a pool (e.g., sum, median), and \( f \) is the activation function [1902.11106]. This generalization enables heterogeneous operator assignment at the level of each neuron or even each connection, vastly increasing representational capacity compared to the homogeneous, linear convolutional neurons found in CNNs. ONNs subsume CNNs as the case \( Y(w, y) = w y,\, P = \sum,\, f = \mathrm{ReLU} \).

Broadly, ONNs fall into two main families:
- **Patch-wise ONNs:** Operate on grid-structured data (images, time series); generalize convolutional layers via flexible nodal/pool/activation operators, either chosen from a discrete "operator library" [2009.00612] or learned "on-the-fly" via Taylor expansions (Self-ONN) [2004.11778, 2109.14873].
- **Operator learning ONNs:** Learn maps between function spaces, as in DeepONet or RBON, central in scientific machine learning for approximating PDE solution operators [2301.06701, 2410.04639, 2209.12871, 2207.05748].

## 2. Self-Organized ONNs (Self-ONNs) and Generative Neurons

Self-ONNs eliminate the operator library and associated search by equipping each connection with a learnable, locally nonlinear function parameterized by a Taylor (Maclaurin) series:
$$
u(w, y) = \sum_{q=0}^{Q-1} w^{(q)}\, y^{q}
$$
where \( w^{(q)} \) are learned coefficients and \( Q \) is the expansion order [2004.11778, 2109.14873]. Each operational or convolutional kernel is thus a tensor of shape (input channels, output channels, spatial dims, Q), allowing each connection to synthesize its optimal local operator during training.

Key properties:
- **Maximum heterogeneity:** Every connection, kernel element, and order has independent learnable parameters.
- **Self-organization:** No pre-specified function library; all operator coefficients are updated by standard backpropagation.
- **Computational tractability:** Forward propagation can be decomposed into Q parallel standard convolutions, making Self-ONNs efficiently vectorizable and compatible with existing GEMM/BLAS/conv backends [2103.03060, 2103.03070].

Empirically, Self-ONNs yield significant performance improvements over both classic ONNs and CNNs—up to 8% higher F₁ in severe bearing fault diagnosis [2109.14873], 1–3 dB better PSNR in restoration/denoising [2008.12894, 2103.03070], and state-of-the-art compact biomedical classifiers [2109.13604].

## 3. Operator Learning Architectures: DeepONet, RBON, and Extensions

Operator learning ONNs aim to learn mappings between function spaces (G: 𝒰 → 𝒱). The seminal DeepONet [2301.06701, 2207.05748, 2309.01020] approximates
$$
G(u)(y) \approx \sum_{i=1}^p b_i(u) t_i(y)
$$
where \( b_i(u) \) (branch net) encodes the input function (e.g., sampled at m points), and \( t_i(y) \) (trunk net) encodes the output location. Universal approximation theorems guarantee convergence for continuous G as \( m,\,p \rightarrow \infty \) [2301.06701, 2405.11907, 2309.01020].

Recent extensions include:

- **Ensemble and Mixture-of-Experts DeepONet:** Multiple trunks (global, local, or data-driven) are combined, allowing the network to simultaneously model global modes and localized features. Mixture-of-Experts variants use partitions of unity to blend local trunks, introducing spatial sparsity and improving steep-gradient resolution [2405.11907].

- **RBON/NRBON/F-RBON:** The Radial Basis Operator Network replaces the branch and trunk nets by RBF layers:
$$
G(u)(x) = \sum_{i,k} \xi_i^k\, g(\|u^m - \mu_{ik}^m\|)\,g(\|x - x_k\|),
$$
with K-means–clustered centers and spreads. Exact linear algebraic fitting gives near machine-precision accuracy and better out-of-distribution robustness compared to DeepONet and FNO [2410.04639].

- **Variationally Mimetic Operator Networks (VarMiON):** Architectures reflecting the variational/Galerkin structure of the underlying PDE, splitting the network into basis-construction (trunk) and coefficient-assembly (branch) modules, which enhances data efficiency and interpretability [2209.12871].

## 4. Training, Optimization, and Operator Search Strategies

Methods to assign or learn operator sets in ONNs include:

- **Greedy Iterative Search (GIS):** For classic ONNs using a finite operator library, GIS trains multiple times per layer to select the best operator set. This approach is computationally demanding and restricts heterogeneity to the layer level [1902.11106].

- **Synaptic Plasticity Monitoring (SPM):** SPM evaluates the "plasticity" (change in weight variance) of each operator set assigned to neurons in random or biased runs, ranking operator sets based on their dynamic contribution to learning. This enables the construction of "elite" ONNs with high intra-layer heterogeneity and has been shown to yield further gains over GIS ONNs and CNNs on restoration, synthesis, and transformation tasks [2009.08934, 2009.00612].

- **Self-organization (Self-ONN):** Backpropagation directly tunes all Taylor coefficients per connection; no outer search is required. Empirically, Self-ONN converges an order of magnitude faster than CNN or ONN [2004.11778].

- **Two-step training for DeepONet:** Decouples trunk (basis) and branch (coefficient) training, orthonormalizing the trunk basis for improved generalization and optimization stability [2309.01020].

## 5. Performance Benchmarks and Applications

ONNs unlock improved expressivity, computational efficiency, and adaptability across domains:

- **Image and signal processing:** Self-ONNs and SPM-ONNs consistently outperform same-sized CNNs and even deeper state-of-the-art networks (DnCNN, BM3D), especially in compact, shallow settings and under severe noise models. For high-noise AWGN, 2-layer Self-ONNs approach or surpass BM3D with up to 1.3 dB PSNR gain [2103.03060]. In biomedical contexts, compact Self-ONNs achieve 100% F₁ in glaucoma detection (ESOGU) and 99.1% F₁ in ECG peak detection, with orders of magnitude lower parameter count and real-time inference [2109.13604, 2110.02381].

- **Operator learning for scientific computing:** DeepONet and its variants (ensemble, PoU, RBON, VarMiON) demonstrate rapid convergence and superior generalization on PDE surrogate tasks (Darcy flow, lid-driven cavity, beam equation, Burgers equation, Allen-Cahn). Ensembles can yield 4× lower errors over standard DeepONet, and RBON achieves $10^{-7}$ L² error in/out-of-distribution [2405.11907, 2410.04639, 2301.06701, 2309.01020, 2209.12871].

- **Physics–informed learning:** Energy-dissipative DeepONet and variationally mimetic ONNs incorporate PDE structure (energy dissipation, weak formulation) into training for stronger inductive bias, guaranteed stability, and better OOD and limited-data performance [2306.06281, 2209.12871].

- **Computational cost:** Despite increased complexity per neuron (e.g., Q polynomials in Self-ONN), most implementations execute Q parallel convolutions, yielding practical inference speeds competitive with CNN baselines—often with dramatic parameter and data efficiency [2109.14873, 2109.13604, FastONN in 2006.02267].

## 6. Theoretical Guarantees, Expressivity, and Limitations

- **Universal approximation:** DeepONet, RBON (incl. normalized and frequency versions), and classical ONNs are all provably universal approximators for nonlinear operators between function spaces, given sufficient network width/depth or RBF basis size [2410.04639, 2301.06701, 1902.11106].
- **Expressivity:** ONNs embed nonlinear transformations at the kernel, connection, or neuron level. Self-ONNs realize a continuous, data-driven operator family, while classic ONNs rely on operator choice (potentially suboptimal if the library is not rich enough). Empirical ablations support that higher-order Taylor expansion (Q ≳ 5) offers the best tradeoff before overfitting or diminishing returns [2008.12894].
- **Robustness and generalization:** DeepONet and RBON demonstrate strong zero-shot generalization; ensemble/moe DeepONets and RBONs maintain low error on OOD tasks, while basic DeepONet can overfit [2410.04639, 2405.11907].
- **Limitations:** The main drawbacks are parameter growth with Q/order, potential for overfitting, and, in some ONNs, reliance on operator libraries or empirical operator search (in GIS/ONN). Adaptive order selection and further operator-space regularization remain active topics. Taylor-based generative neurons may poorly approximate non-polynomial operator classes in some settings [2004.11778].
- **Computation and implementation:** With appropriate vectorization and GPU backends, Self-ONNs and FastONN achieve efficient batch inference (see 2006.02267).

## 7. Outlook and Algorithmic Variants

Active research directions include:
- **Operator enrichment:** Stacked or hybrid trunk and branch networks (POD, RBF, tails, interpretable modes) to better match multiscale or localized operator structure [2405.11907, 2410.04639].
- **Adaptive/sparse operator design:** Partition-of-unity and mixture-of-experts methods for local attention, adaptivity, and scalability without a global parameter increase [2405.11907].
- **Physics-informed and mimetic architectures:** VarMiON, EDE-DeepONet, and PINOs for improved stability, interpretability, and accuracy under physical constraints [2306.06281, 2209.12871, 2207.05748].
- **Hyperparameter adaptation:** Selection of Taylor order Q per layer or neuron, regularization for overfitting control, and reinforcement of synaptic plasticity assignment during learning [2009.08934, 2004.11778].
- **Software frameworks:** GPU-efficient implementations (FastONN) and modifiable operator-set libraries for rapid prototyping and experimentation [2006.02267].

ONNs thus represent a comprehensive and flexible neural modeling paradigm that spans from locally nonlinear signal-processing networks to universal operator learners, combining theoretical rigor, empirical superiority, broad applicability, and extensible implementation [1902.11106, 2004.11778, 2109.14873, 2410.04639, 2405.11907, 2301.06701, 2209.12871].

Source: https://www.emergentmind.com/topics/operator-neural-networks-onns