Integral Transform Network (ITNet)

• Integral Transform Network (ITNet) is a neural architecture that employs a learnable integral transform with a kernel dependent on both positions and features, recovering convolution, attention, and recurrence as special cases.
• The network uses a two-layer MLP kernel enhanced with random Fourier features and positional cues to capture complex spatial and feature interactions over continuous domains.
• Variants such as GDNN, GIT-Net, and IAE-Net extend ITNet by replacing conventional activations or enabling operator learning for PDEs and multimodal tasks, highlighting its flexibility and efficiency.

Integral Transform Network (ITNet) most explicitly denotes a neural architecture built around a learnable integral transform whose kernel depends jointly on positions and features, so that convolution, self-attention, and autoregressive recurrence arise as special cases under appropriate parameterizations (Dhor et al., 17 Jun 2026). In adjacent literature, the same label is also used informally for standard feedforward networks in which conventional activations are replaced by Integral Activation Transform layers, and it is substantively related to integral-transform neural operators such as GIT-Net and discretization-invariant encoder–decoder models such as IAE-Net (Zhang et al., 2023, Wang et al., 2023, Ong et al., 2022). By contrast, the similarly written “IT-Net” in 3D point-cloud processing refers to the Iterative Transformer Network, a geometric rigid-alignment module rather than an integral-transform architecture (Yuan et al., 2018).

1. Terminology and naming

The literature contains several distinct uses of closely related names. The most explicit use of the exact name appears in “ITNet: A Learnable Integral Transform That Subsumes Convolution, Attention, and Recurrence,” where ITNet is the title of the architecture itself (Dhor et al., 17 Jun 2026). In “Improving the Expressive Power of Deep Neural Networks through Integral Activation Transform,” the paper does not use the name “Integral Transform Network”; in that source, “ITNet” is an explanatory label for a standard DNN in which some or all activations are replaced by Integral Activation Transform layers (Zhang et al., 2023). “GIT-Net: Generalized Integral Transform for Operator Learning” likewise does not explicitly use the name “ITNet,” but it is presented as substantively an integral-transform-based neural operator (Wang et al., 2023). “IAE-Net: Integral Autoencoders for Discretization-Invariant Learning” is a deep architecture built from learned integral transforms in encoder and decoder blocks, and is presented as a concrete realization of an integral transform network in the operator-learning sense (Ong et al., 2022).

Source	Name in source	Relation to ITNet
(Dhor et al., 17 Jun 2026)	ITNet	Explicit learnable integral-transform architecture
(Zhang et al., 2023)	GDNN / IAT	“ITNet” used as DNN + IAT layers
(Wang et al., 2023)	GIT-Net	Generalized integral-transform neural operator
(Ong et al., 2022)	IAE-Net	Deep stack of learned integral encoders/decoders
(Yuan et al., 2018)	IT-Net	Iterative Transformer Network, not integral-transform

The distinction from the 3D point-cloud paper is essential. That method is the Iterative Transformer Network, not “Integral Transform Network,” and “Transformer” there refers to a geometric transformer that predicts and applies rigid $SE(3)$ transforms rather than an attention-based Transformer or an integral operator (Yuan et al., 2018). A common misconception is therefore purely terminological: the string “IT-Net” does not uniquely identify a single research program.

2. Explicit ITNet as a learnable integral operator

In the explicit 2026 formulation, ITNet replaces convolutions, softmax attention, and recurrent/state-space propagation with a single learnable integral transform whose kernel depends jointly on positions and features at both endpoints of an interaction (Dhor et al., 17 Jun 2026). Let $\Omega \subset \mathbb{R}^s$ be the input domain with finite measure $\mu$, and let $f:\Omega \to \mathbb{R}^C$ be the input feature field. The general operator is written as

$$(\mathcal{T}_\theta f)(x) = \int_\Omega K_\theta(x, y, f(x), f(y)) V_\theta(f(y))\, d\mu(y).$$

The practical specialization absorbs the value projection into the kernel and adds a residual term:

$$(K[u])(x) = \int_\Omega \kappa_\theta(x, y, u(x), u(y))\, u(y)\, d\mu(y) + W_\theta u(x).$$

Here $\kappa_\theta \in \mathbb{R}^{C \times C}$ is a learnable matrix-valued kernel and $W_\theta \in \mathbb{R}^{C \times C}$ is a learnable residual. The residual is stated to guarantee that the operator can represent the identity at initialization and to stabilize deep stacks (Dhor et al., 17 Jun 2026).

For discrete domains with $N$ positions $\{x_1,\dots,x_N\}$ and uniform atomic measure $\Omega \subset \mathbb{R}^s$0, the layer becomes

$\Omega \subset \mathbb{R}^s$1

optionally with masking and normalization. Multi-head ITNet splits channels into $\Omega \subset \mathbb{R}^s$2 heads of size $\Omega \subset \mathbb{R}^s$3, applies independent kernels $\Omega \subset \mathbb{R}^s$4, and recombines with an output projection $\Omega \subset \mathbb{R}^s$5 (Dhor et al., 17 Jun 2026).

The kernel itself is implemented as a 2-layer MLP of width $\Omega \subset \mathbb{R}^s$6 with GELU, receiving

$\Omega \subset \mathbb{R}^s$7

where $\Omega \subset \mathbb{R}^s$8 is a random Fourier feature map of positions with $\Omega \subset \mathbb{R}^s$9 frequencies. This gives the kernel direct access to absolute position, relative displacement, Euclidean distance, endpoint features, and elementwise feature interactions (Dhor et al., 17 Jun 2026). Modality-specific encoders then map raw signals into a common feature space $\mu$0: images use $\mu$1 patches with 2-D coordinates, text uses token embeddings and normalized indices, point clouds use raw 3-D coordinates and optional attributes, and multimodal inputs use a union domain $\mu$2 with distinct modality embeddings and a shared operator (Dhor et al., 17 Jun 2026).

The stacked network uses a pre-norm residual layout,

$\mu$3

with 2-layer FFNs, GELU, LayerNorm, and residuals (Dhor et al., 17 Jun 2026). The kernel MLP is initialized near zero output with $\mu$4, $\mu$5, and $\mu$6, so each layer begins near identity (Dhor et al., 17 Jun 2026).

3. Special-case reductions and theoretical status

A central claim of the explicit ITNet formulation is exact subsumption: convolution, self-attention, and autoregressive recurrence are obtained by imposing appropriate structure on the same operator class (Dhor et al., 17 Jun 2026). Convolution is recovered when the kernel depends only on relative position and is scalar times identity,

$\mu$7

with $\mu$8, yielding

$\mu$9

Local receptive fields, dilation, stride, and depthwise or grouped variants arise through support restrictions, dilated grids, strided query sets, and diagonal or block-diagonal channel structure (Dhor et al., 17 Jun 2026).

Self-attention is recovered by choosing a softmax-normalized bilinear form. With $f:\Omega \to \mathbb{R}^C$0, $f:\Omega \to \mathbb{R}^C$1, and $f:\Omega \to \mathbb{R}^C$2, the attention kernel

$f:\Omega \to \mathbb{R}^C$3

gives

$f:\Omega \to \mathbb{R}^C$4

Within ITNet this is obtained by setting

$f:\Omega \to \mathbb{R}^C$5

where $f:\Omega \to \mathbb{R}^C$6 and positional biases are absorbed by directly supplying $f:\Omega \to \mathbb{R}^C$7, $f:\Omega \to \mathbb{R}^C$8, $f:\Omega \to \mathbb{R}^C$9, and $$(\mathcal{T}_\theta f)(x) = \int_\Omega K_\theta(x, y, f(x), f(y)) V_\theta(f(y))\, d\mu(y).$$0 to the kernel (Dhor et al., 17 Jun 2026).

Causal recurrence is recovered by triangular masking, $$(\mathcal{T}_\theta f)(x) = \int_\Omega K_\theta(x, y, f(x), f(y)) V_\theta(f(y))\, d\mu(y).$$1 for $$(\mathcal{T}_\theta f)(x) = \int_\Omega K_\theta(x, y, f(x), f(y)) V_\theta(f(y))\, d\mu(y).$$2. The paper gives explicit causal-kernel forms for linear time-invariant state-space models, discrete-time RNNs, LSTM, GRU, and Mamba. For example, an S4-like operator with $$(\mathcal{T}_\theta f)(x) = \int_\Omega K_\theta(x, y, f(x), f(y)) V_\theta(f(y))\, d\mu(y).$$3 and $$(\mathcal{T}_\theta f)(x) = \int_\Omega K_\theta(x, y, f(x), f(y)) V_\theta(f(y))\, d\mu(y).$$4 becomes

$$(\mathcal{T}_\theta f)(x) = \int_\Omega K_\theta(x, y, f(x), f(y)) V_\theta(f(y))\, d\mu(y).$$5

which is ITNet with kernel $$(\mathcal{T}_\theta f)(x) = \int_\Omega K_\theta(x, y, f(x), f(y)) V_\theta(f(y))\, d\mu(y).$$6 and residual $$(\mathcal{T}_\theta f)(x) = \int_\Omega K_\theta(x, y, f(x), f(y)) V_\theta(f(y))\, d\mu(y).$$7 (Dhor et al., 17 Jun 2026).

The same source states a universality theorem: under compact $$(\mathcal{T}_\theta f)(x) = \int_\Omega K_\theta(x, y, f(x), f(y)) V_\theta(f(y))\, d\mu(y).$$8 and a compact input set $$(\mathcal{T}_\theta f)(x) = \int_\Omega K_\theta(x, y, f(x), f(y)) V_\theta(f(y))\, d\mu(y).$$9, ITNet uniformly approximates any continuous operator $$(K[u])(x) = \int_\Omega \kappa_\theta(x, y, u(x), u(y))\, u(y)\, d\mu(y) + W_\theta u(x).$$0 (Dhor et al., 17 Jun 2026). The proof outline proceeds by quadrature discretization, MLP approximation of the continuous target kernel, and composition of these approximations. The paper also frames ITNet’s inductive biases as learned rather than hard-coded: locality is induced by support masks, sequential memory by causality, and content-dependent pairwise interaction by conditioning on $$(K[u])(x) = \int_\Omega \kappa_\theta(x, y, u(x), u(y))\, u(y)\, d\mu(y) + W_\theta u(x).$$1, $$(K[u])(x) = \int_\Omega \kappa_\theta(x, y, u(x), u(y))\, u(y)\, d\mu(y) + W_\theta u(x).$$2, and $$(K[u])(x) = \int_\Omega \kappa_\theta(x, y, u(x), u(y))\, u(y)\, d\mu(y) + W_\theta u(x).$$3 (Dhor et al., 17 Jun 2026). This suggests a unification at the operator level rather than a mere empirical resemblance between architectural families.

4. Continuous width and Integral Activation Transform layers

A different but related line of work develops integral transforms at the activation stage rather than the interaction stage. In the GDNN/IAT formulation, the traditional notion of neurons in each layer is replaced by a continuous state function indexed by a continuous variable $$(K[u])(x) = \int_\Omega \kappa_\theta(x, y, u(x), u(y))\, u(y)\, d\mu(y) + W_\theta u(x).$$4, yielding a Generalized Deep Neural Network whose forward propagation uses integral transforms (Zhang et al., 2023). Under a finite-rank kernel parameterization, this continuous-width GDNN is exactly equivalent to a standard DNN in which the usual elementwise activation is replaced by an Integral Activation Transform layer (Zhang et al., 2023).

Given basis collections $$(K[u])(x) = \int_\Omega \kappa_\theta(x, y, u(x), u(y))\, u(y)\, d\mu(y) + W_\theta u(x).$$5 and $$(K[u])(x) = \int_\Omega \kappa_\theta(x, y, u(x), u(y))\, u(y)\, d\mu(y) + W_\theta u(x).$$6 on $$(K[u])(x) = \int_\Omega \kappa_\theta(x, y, u(x), u(y))\, u(y)\, d\mu(y) + W_\theta u(x).$$7, the IAT maps $$(K[u])(x) = \int_\Omega \kappa_\theta(x, y, u(x), u(y))\, u(y)\, d\mu(y) + W_\theta u(x).$$8 to

$$(K[u])(x) = \int_\Omega \kappa_\theta(x, y, u(x), u(y))\, u(y)\, d\mu(y) + W_\theta u(x).$$9

Operationally, the input vector is lifted to a function $\kappa_\theta \in \mathbb{R}^{C \times C}$0, nonlinearity is applied in function space, and the result is projected back to $\kappa_\theta \in \mathbb{R}^{C \times C}$1 through integration against $\kappa_\theta \in \mathbb{R}^{C \times C}$2 (Zhang et al., 2023). With $\kappa_\theta \in \mathbb{R}^{C \times C}$3, the paper defines

$\kappa_\theta \in \mathbb{R}^{C \times C}$4

the activation pattern

$\kappa_\theta \in \mathbb{R}^{C \times C}$5

and the activation matrix

$\kappa_\theta \in \mathbb{R}^{C \times C}$6

The forward map then becomes

$\kappa_\theta \in \mathbb{R}^{C \times C}$7

with gradient

$\kappa_\theta \in \mathbb{R}^{C \times C}$8

When $\kappa_\theta \in \mathbb{R}^{C \times C}$9 is continuous, $W_\theta \in \mathbb{R}^{C \times C}$0 varies continuously in $W_\theta \in \mathbb{R}^{C \times C}$1, and the layer is described as smooth in the usual training sense, with continuous gradient almost everywhere (Zhang et al., 2023).

The paper also derives a global Lipschitz bound,

$W_\theta \in \mathbb{R}^{C \times C}$2

and notes that rectangular piecewise-constant bases reduce IAT-ReLU to the ordinary componentwise ReLU (Zhang et al., 2023). In midpoint quadrature form,

$W_\theta \in \mathbb{R}^{C \times C}$3

with $W_\theta \in \mathbb{R}^{C \times C}$4 and $W_\theta \in \mathbb{R}^{C \times C}$5, the per-layer cost is $W_\theta \in \mathbb{R}^{C \times C}$6 (Zhang et al., 2023).

The reported numerical behavior is strongly tied to trainability. In 2D function fitting with $W_\theta \in \mathbb{R}^{C \times C}$7, standard ReLU achieves MSE $W_\theta \in \mathbb{R}^{C \times C}$8, while the best IAT-ReLU combinations achieve $W_\theta \in \mathbb{R}^{C \times C}$9 to $N$0; in random label memorization with $N$1, ReLU gives $N$2 accuracy and the best IAT-ReLU reaches up to $N$3 (Zhang et al., 2023). Basis selection is part of the design: global, zero-mean input bases such as Fourier or wavelets paired with local output bases such as piecewise quadratic functions performed best on average in the reported experiments (Zhang et al., 2023).

5. Neural-operator variants: GIT-Net and IAE-Net

In operator learning, the integral-transform viewpoint is developed at the level of mappings between function spaces. GIT-Net formulates operator learning as regression between infinite-dimensional function spaces $N$4 and $N$5, with training pairs $N$6 satisfying $N$7 and expected risk

$N$8

approximated empirically over $N$9 samples (Wang et al., 2023). Its motivating observation is that many PDE operators are parsimonious in specialized functional bases. The canonical integral form

$\{x_1,\dots,x_N\}$0

is written, when diagonalizable in an appropriate basis, as

$\{x_1,\dots,x_N\}$1

GIT-Net implements a generalized version of this idea using PCA bases, learnable changes of basis $\{x_1,\dots,x_N\}$2, a learned frequency-wise channel mixer $\{x_1,\dots,x_N\}$3, and a channel mixer $\{x_1,\dots,x_N\}$4 (Wang et al., 2023).

For coefficient tensor $\{x_1,\dots,x_N\}$5, the learned generalized nonlocal operator is

$\{x_1,\dots,x_N\}$6

and the nonlinear GIT layer is

$\{x_1,\dots,x_N\}$7

With orthonormal basis functions, the corresponding physical-space kernel is written explicitly as

$\{x_1,\dots,x_N\}$8

An $\{x_1,\dots,x_N\}$9-layer GIT-Net then has the form

$\Omega \subset \mathbb{R}^s$00

with GELU used in all but the last layer (Wang et al., 2023). The reported evaluation complexity is

$\Omega \subset \mathbb{R}^s$01

and a layer uses on the order of $\Omega \subset \mathbb{R}^s$02 parameters rather than $\Omega \subset \mathbb{R}^s$03 for a dense linear map (Wang et al., 2023).

IAE-Net takes a different route to integral-transform operator learning by using an integral encoder, a fixed-size bottleneck, and an integral decoder. For a function $\Omega \subset \mathbb{R}^s$04 on $\Omega \subset \mathbb{R}^s$05, the encoder maps from an arbitrary input grid $\Omega \subset \mathbb{R}^s$06 to a fixed latent grid $\Omega \subset \mathbb{R}^s$07 through

$\Omega \subset \mathbb{R}^s$08

the bottleneck applies a standard FNN $\Omega \subset \mathbb{R}^s$09, and the decoder reconstructs through

$\Omega \subset \mathbb{R}^s$10

Numerically, the integral transform

$\Omega \subset \mathbb{R}^s$11

is implemented on samples as

$\Omega \subset \mathbb{R}^s$12

or, for nonlinear kernels,

$\Omega \subset \mathbb{R}^s$13

with appropriate quadrature weights on nonuniform grids (Ong et al., 2022).

IAE-Net composes such blocks densely. Its multi-channel block applies several IAEs in parallel, including an identity channel and transformed channels such as Fourier,

$\Omega \subset \mathbb{R}^s$14

and a deep network concatenates all previous outputs in a DenseNet-like fashion before another IAE block (Ong et al., 2022). Discretization invariance is pursued through operator design and a randomized discretization-augmentation objective,

$\Omega \subset \mathbb{R}^s$15

with $\Omega \subset \mathbb{R}^s$16 by default and cubic or bicubic interpolation for resampling (Ong et al., 2022). The supervised loss is the relative $\Omega \subset \mathbb{R}^s$17 error,

$\Omega \subset \mathbb{R}^s$18

Empirically, GIT-Net is evaluated on Navier–Stokes, Helmholtz, structural mechanics, 1D advection, and Poisson problems, using $\Omega \subset \mathbb{R}^s$19 GIT layers, GELU, PCA truncation preserving at least $\Omega \subset \mathbb{R}^s$20 energy capped at $\Omega \subset \mathbb{R}^s$21, channel counts $\Omega \subset \mathbb{R}^s$22, frequencies $\Omega \subset \mathbb{R}^s$23, and training sizes $\Omega \subset \mathbb{R}^s$24 (Wang et al., 2023). In the large-data regime $\Omega \subset \mathbb{R}^s$25, it is reported to achieve the smallest test error across all PDE problems, and on complex geometries it is described as substantially more accurate than FNO because the latter suffered from interpolation artifacts (Wang et al., 2023). IAE-Net is evaluated on 1D Burgers, 2D Darcy flow, Darcy on a general domain, forward and inverse scattering, CT image denoising, and signal source separation, with four IAE blocks, channel width $\Omega \subset \mathbb{R}^s$26, a 2-channel identity-plus-Fourier design, Adam at learning rate $\Omega \subset \mathbb{R}^s$27, 500 epochs, and a half-on-plateau scheduler (Ong et al., 2022). Representative results include Burgers with $\Omega \subset \mathbb{R}^s$28, where IAE-Net reports $\Omega \subset \mathbb{R}^s$29 relative error versus FNO $\Omega \subset \mathbb{R}^s$30, and CT denoising at $\Omega \subset \mathbb{R}^s$31, where IAE-Net reports $\Omega \subset \mathbb{R}^s$32 versus FNO $\Omega \subset \mathbb{R}^s$33 (Ong et al., 2022).

6. Empirical profile, computational trade-offs, and scope

The explicit 2026 ITNet is evaluated as a single shared operator across images, text, point clouds, and multimodal reasoning, with only lightweight modality-specific encoders changing across tasks (Dhor et al., 17 Jun 2026). Reported headline results are ImageNet-1K top-1 of $\Omega \subset \mathbb{R}^s$34 for ITNet-S (22M), $\Omega \subset \mathbb{R}^s$35 for ITNet-B (86M), and $\Omega \subset \mathbb{R}^s$36 for ITNet-L (307M); GLUE dev-set averages of $\Omega \subset \mathbb{R}^s$37 for ITNet-B and $\Omega \subset \mathbb{R}^s$38 for ITNet-L; ModelNet40 overall accuracy of $\Omega \subset \mathbb{R}^s$39 for ITNet-PC (3.1M) with local $\Omega \subset \mathbb{R}^s$40 and $\Omega \subset \mathbb{R}^s$41 for ITNet-B; and multimodal scores of $\Omega \subset \mathbb{R}^s$42 on VQA v2 and $\Omega \subset \mathbb{R}^s$43 on NLVR2 for ITNet-B, rising to $\Omega \subset \mathbb{R}^s$44 and $\Omega \subset \mathbb{R}^s$45 for ITNet-L (Dhor et al., 17 Jun 2026). The same source states that tiled exact ITNet is within approximately $\Omega \subset \mathbb{R}^s$46 the throughput of FlashAttention-2 at moderate sequence length, citing DeiT-S throughput of $\Omega \subset \mathbb{R}^s$47 img/s versus ITNet-S at $\Omega \subset \mathbb{R}^s$48 img/s for $\Omega \subset \mathbb{R}^s$49 patches, while low-rank ITNet yields $\Omega \subset \mathbb{R}^s$50–$\Omega \subset \mathbb{R}^s$51 higher throughput with less than $\Omega \subset \mathbb{R}^s$52–$\Omega \subset \mathbb{R}^s$53 accuracy loss at ranks $\Omega \subset \mathbb{R}^s$54–$\Omega \subset \mathbb{R}^s$55 on ImageNet-1K (Dhor et al., 17 Jun 2026).

The main computational issue is that naïve exact evaluation materializes an $\Omega \subset \mathbb{R}^s$56 matrix-valued kernel at cost $\Omega \subset \mathbb{R}^s$57 time and memory (Dhor et al., 17 Jun 2026). Three remedies are given: tiled kernel fusion, importance-weighted Monte Carlo integration, and learned low-rank factorization. Tiled fusion avoids materializing the global kernel and reduces peak memory to $\Omega \subset \mathbb{R}^s$58; Monte Carlo reduces complexity to $\Omega \subset \mathbb{R}^s$59 for $\Omega \subset \mathbb{R}^s$60 with an unbiased estimator; and low-rank factorization reduces the operator to $\Omega \subset \mathbb{R}^s$61, with the bound

$\Omega \subset \mathbb{R}^s$62

and reported relative error below $\Omega \subset \mathbb{R}^s$63 at $\Omega \subset \mathbb{R}^s$64 in ImageNet experiments (Dhor et al., 17 Jun 2026). By comparison, IAT layers incur $\Omega \subset \mathbb{R}^s$65 cost per layer under midpoint quadrature, GIT-Net layers evaluate in $\Omega \subset \mathbb{R}^s$66, and each IAE encoder or decoder application is a matrix–vector product with cost $\Omega \subset \mathbb{R}^s$67 (Zhang et al., 2023, Wang et al., 2023, Ong et al., 2022).

The limitations are architecture-specific rather than uniform across the entire integral-transform family. For the explicit ITNet, exact computation is quadratic without approximation, very long contexts require low-rank or sampling approximations, training stability depends on careful initialization and normalization, strict token-by-token autoregressive decoding still benefits from scanning-friendly parameterizations, and interpretability of the learned kernel $\Omega \subset \mathbb{R}^s$68 is challenging (Dhor et al., 17 Jun 2026). For IAT-based models, computational overhead grows with discretization $\Omega \subset \mathbb{R}^s$69, performance is sensitive to basis choice, discretization introduces piecewise linearity, and higher-order derivatives remain problematic in the discrete IAT-ReLU setting (Zhang et al., 2023). For GIT-Net, the basis-induced bias is that operators are assumed near-diagonal in an appropriate basis, explicit translation equivariance is lost when Fourier bases are replaced by PCA, and the paper does not present new formal approximation theorems (Wang et al., 2023). For IAE-Net, assembling and applying full kernel matrices becomes expensive at very high resolutions, latent size and kernel-network capacity require tuning, and improved quadrature may be needed for highly irregular sampling (Ong et al., 2022).

Across these variants, the shared structural motif is the replacement of fixed local kernels, scalar dot-product attention weights, or discretization-bound matrices by learned integral operators acting either over interactions, activations, or function spaces. A plausible implication is that “Integral Transform Network” now names a family of architectures rather than a single immutable blueprint. What remains non-negotiable in the literature is the distinction from the 2018 point-cloud IT-Net, whose subject is iterative rigid $\Omega \subset \mathbb{R}^s$70 canonicalization rather than integral transforms (Yuan et al., 2018).