---
title: Integral Transform Network (ITNet)
url: https://www.emergentmind.com/topics/integral-transform-network-itnet
type: topic
---

# Integral Transform Network (ITNet)

Integral Transform Network (ITNet) most explicitly denotes a neural architecture built around a learnable integral transform whose kernel depends jointly on positions and features, so that convolution, self-attention, and autoregressive recurrence arise as special cases under appropriate parameterizations [2606.19538]. In adjacent literature, the same label is also used informally for standard feedforward networks in which conventional activations are replaced by Integral Activation Transform layers, and it is substantively related to integral-transform neural operators such as GIT-Net and discretization-invariant encoder–decoder models such as IAE-Net [2312.12578][2312.02450][2203.05142]. By contrast, the similarly written “IT-Net” in 3D point-cloud processing refers to the Iterative Transformer Network, a geometric rigid-alignment module rather than an integral-transform architecture [1811.11209].

## 1. Terminology and naming

The literature contains several distinct uses of closely related names. The most explicit use of the exact name appears in “ITNet: A Learnable Integral Transform That Subsumes Convolution, Attention, and Recurrence,” where ITNet is the title of the architecture itself [2606.19538]. In “Improving the Expressive Power of Deep Neural Networks through Integral Activation Transform,” the paper does not use the name “Integral Transform Network”; in that source, “ITNet” is an explanatory label for a standard DNN in which some or all activations are replaced by Integral Activation Transform layers [2312.12578]. “GIT-Net: Generalized Integral Transform for Operator Learning” likewise does not explicitly use the name “ITNet,” but it is presented as substantively an integral-transform-based neural operator [2312.02450]. “IAE-Net: Integral Autoencoders for Discretization-Invariant Learning” is a deep architecture built from learned integral transforms in encoder and decoder blocks, and is presented as a concrete realization of an integral transform network in the operator-learning sense [2203.05142].

| Source | Name in source | Relation to ITNet |
|---|---|---|
| [2606.19538] | ITNet | Explicit learnable integral-transform architecture |
| [2312.12578] | GDNN / IAT | “ITNet” used as DNN + IAT layers |
| [2312.02450] | GIT-Net | Generalized integral-transform neural operator |
| [2203.05142] | IAE-Net | Deep stack of learned integral encoders/decoders |
| [1811.11209] | IT-Net | Iterative Transformer Network, not integral-transform |

The distinction from the 3D point-cloud paper is essential. That method is the Iterative Transformer Network, not “Integral Transform Network,” and “Transformer” there refers to a geometric transformer that predicts and applies rigid $SE(3)$ transforms rather than an attention-based Transformer or an integral operator [1811.11209]. A common misconception is therefore purely terminological: the string “IT-Net” does not uniquely identify a single research program.

## 2. Explicit ITNet as a learnable integral operator

In the explicit 2026 formulation, ITNet replaces convolutions, softmax attention, and recurrent/state-space propagation with a single learnable integral transform whose kernel depends jointly on positions and features at both endpoints of an interaction [2606.19538]. Let $\Omega \subset \mathbb{R}^s$ be the input domain with finite measure $\mu$, and let $f:\Omega \to \mathbb{R}^C$ be the input feature field. The general operator is written as
$$
(\mathcal{T}_\theta f)(x) = \int_\Omega K_\theta(x, y, f(x), f(y)) V_\theta(f(y))\, d\mu(y).
$$
The practical specialization absorbs the value projection into the kernel and adds a residual term:
$$
(K[u])(x) = \int_\Omega \kappa_\theta(x, y, u(x), u(y))\, u(y)\, d\mu(y) + W_\theta u(x).
$$
Here $\kappa_\theta \in \mathbb{R}^{C \times C}$ is a learnable matrix-valued kernel and $W_\theta \in \mathbb{R}^{C \times C}$ is a learnable residual. The residual is stated to guarantee that the operator can represent the identity at initialization and to stabilize deep stacks [2606.19538].

For discrete domains with $N$ positions $\{x_1,\dots,x_N\}$ and uniform atomic measure $\mu(\{x_j\})=1/N$, the layer becomes
$$
y_i = \sum_j \frac{1}{N}\,\kappa_\theta(x_i, x_j, u_i, u_j)\, u_j + W_\theta u_i,
$$
optionally with masking and normalization. Multi-head ITNet splits channels into $H$ heads of size $d_h=C/H$, applies independent kernels $\kappa^{(h)}$, and recombines with an output projection $W^O$ [2606.19538].

The kernel itself is implemented as a 2-layer MLP of width $128$ with GELU, receiving
$$
z_{xy} = [\gamma(x);\gamma(y);\gamma(x-y);\|x-y\|_2;u(x);u(y);u(x)\odot u(y)],
$$
where $\gamma$ is a random Fourier feature map of positions with $L=64$ frequencies. This gives the kernel direct access to absolute position, relative displacement, Euclidean distance, endpoint features, and elementwise feature interactions [2606.19538]. Modality-specific encoders then map raw signals into a common feature space $u:\Omega \to \mathbb{R}^C$: images use $P \times P$ patches with 2-D coordinates, text uses token embeddings and normalized indices, point clouds use raw 3-D coordinates and optional attributes, and multimodal inputs use a union domain $\Omega_{\text{img}} \cup \Omega_{\text{txt}}$ with distinct modality embeddings and a shared operator [2606.19538].

The stacked network uses a pre-norm residual layout,
$$
z^{(\ell)} = K^{(\ell)}[\mathrm{LN}(u^{(\ell-1)})] + u^{(\ell-1)}, \qquad
u^{(\ell)} = \mathrm{FFN}^{(\ell)}(\mathrm{LN}(z^{(\ell)})) + z^{(\ell)},
$$
with 2-layer FFNs, GELU, LayerNorm, and residuals [2606.19538]. The kernel MLP is initialized near zero output with $W_2=\epsilon I$, $\epsilon=10^{-3}$, and $W_\theta=I$, so each layer begins near identity [2606.19538].

## 3. Special-case reductions and theoretical status

A central claim of the explicit ITNet formulation is exact subsumption: convolution, self-attention, and autoregressive recurrence are obtained by imposing appropriate structure on the same operator class [2606.19538]. Convolution is recovered when the kernel depends only on relative position and is scalar times identity,
$$
\kappa_\theta(i,j,x_i,x_j) = w_\phi(x_i-x_j)I,
$$
with $W_\theta=0$, yielding
$$
(K[u])(x)=\int_\Omega w_\phi(x-y)u(y)\,d\mu(y)=(w_\phi * u)(x).
$$
Local receptive fields, dilation, stride, and depthwise or grouped variants arise through support restrictions, dilated grids, strided query sets, and diagonal or block-diagonal channel structure [2606.19538].

Self-attention is recovered by choosing a softmax-normalized bilinear form. With $q_i=W_qx_i$, $k_j=W_kx_j$, and $v_j=W_vx_j$, the attention kernel
$$
K_\theta(i,j,x_i,x_j)=\mathrm{softmax}_j\!\left(\frac{q_i^\top k_j}{\sqrt{d_k}}+b_{\text{pos}}(i,j)\right)
$$
gives
$$
y_i = \sum_j K_\theta(i,j,\cdot)\,v_j.
$$
Within ITNet this is obtained by setting
$$
\kappa_\theta(x,y,u(x),u(y))
=
\frac{\exp(Q(x)^\top K(y)/\sqrt{d_k})}{Z(x)} \cdot W_v,
$$
where $Z(x)=\int_\Omega \exp(Q(x)^\top K(z)/\sqrt{d_k})\,d\mu(z)$ and positional biases are absorbed by directly supplying $x$, $y$, $x-y$, and $\|x-y\|_2$ to the kernel [2606.19538].

Causal recurrence is recovered by triangular masking, $\kappa_\theta(i,j,\cdot)=0$ for $j>i$. The paper gives explicit causal-kernel forms for linear time-invariant state-space models, discrete-time RNNs, LSTM, GRU, and Mamba. For example, an S4-like operator with $\dot h(t)=Ah(t)+Bu(t)$ and $y(t)=Ch(t)+Du(t)$ becomes
$$
y(t)=\int_0^t Ce^{A(t-s)}B\,u(s)\,ds + Du(t),
$$
which is ITNet with kernel $\kappa_\theta(t,s,\cdot)=\mathbf{1}_{s\le t}Ce^{A(t-s)}B$ and residual $W_\theta=D$ [2606.19538].

The same source states a universality theorem: under compact $\Omega$ and a compact input set $U_c \subset C(\Omega,\mathbb{R}^C)$, ITNet uniformly approximates any continuous operator $F:U_c \to C(\Omega,\mathbb{R}^C)$ [2606.19538]. The proof outline proceeds by quadrature discretization, MLP approximation of the continuous target kernel, and composition of these approximations. The paper also frames ITNet’s inductive biases as learned rather than hard-coded: locality is induced by support masks, sequential memory by causality, and content-dependent pairwise interaction by conditioning on $u(x)$, $u(y)$, and $u(x)\odot u(y)$ [2606.19538]. This suggests a unification at the operator level rather than a mere empirical resemblance between architectural families.

## 4. Continuous width and Integral Activation Transform layers

A different but related line of work develops integral transforms at the activation stage rather than the interaction stage. In the GDNN/IAT formulation, the traditional notion of neurons in each layer is replaced by a continuous state function indexed by a continuous variable $s \in [-1,1]$, yielding a Generalized Deep Neural Network whose forward propagation uses integral transforms [2312.12578]. Under a finite-rank kernel parameterization, this continuous-width GDNN is exactly equivalent to a standard DNN in which the usual elementwise activation is replaced by an Integral Activation Transform layer [2312.12578].

Given basis collections $p(\cdot)\in\mathbb{R}^{d_1}$ and $q(\cdot)\in\mathbb{R}^{d_2}$ on $\mathcal{S}=[-1,1]$, the IAT maps $z\in\mathbb{R}^{d_1}$ to
$$
u = I^\sigma_{p,q}(z) = \int_{-1}^{1} q(s)\,\sigma(z^\top p(s))\,ds.
$$
Operationally, the input vector is lifted to a function $s \mapsto z^\top p(s)$, nonlinearity is applied in function space, and the result is projected back to $\mathbb{R}^{d_2}$ through integration against $q(s)$ [2312.12578]. With $\sigma(t)=\mathrm{ReLU}(t)$, the paper defines
$$
I^{\mathrm{ReLU}}_{p,q}(z)=\int_{-1}^{1} q(s)\,\mathrm{ReLU}(z^\top p(s))\,ds,
$$
the activation pattern
$$
D(z)=\{s\in[-1,1]:p(s)^\top z>0\},
$$
and the activation matrix
$$
S(D(z))=\int_{D(z)} q(s)p(s)^\top ds.
$$
The forward map then becomes
$$
I^{\mathrm{ReLU}}_{p,q}(z)=S(D(z))z,
$$
with gradient
$$
\nabla_z I^{\mathrm{ReLU}}_{p,q}(z)=S(D(z)).
$$
When $p(\cdot)$ is continuous, $D(z)$ varies continuously in $z$, and the layer is described as smooth in the usual training sense, with continuous gradient almost everywhere [2312.12578].

The paper also derives a global Lipschitz bound,
$$
\|I^{\mathrm{ReLU}}_{p,q}(z)-I^{\mathrm{ReLU}}_{p,q}(z')\|
\le
\left(\int_{-1}^{1}\|q(s)\|\,\|p(s)\|\,ds\right)\|z-z'\|,
$$
and notes that rectangular piecewise-constant bases reduce IAT-ReLU to the ordinary componentwise ReLU [2312.12578]. In midpoint quadrature form,
$$
\hat u = \frac{2}{M}Q\,\sigma(P^\top z),
$$
with $P\in\mathbb{R}^{d\times M}$ and $Q\in\mathbb{R}^{m\times M}$, the per-layer cost is $O((d+m)M)$ [2312.12578].

The reported numerical behavior is strongly tied to trainability. In 2D function fitting with $M=500$, standard ReLU achieves MSE $\approx 1.1\mathrm{e}{-3}$, while the best IAT-ReLU combinations achieve $\approx 6.7\mathrm{e}{-5}$ to $1.0\mathrm{e}{-4}$; in random label memorization with $M=500$, ReLU gives $\approx 90.4\%$ accuracy and the best IAT-ReLU reaches up to $\approx 98.3\%$ [2312.12578]. Basis selection is part of the design: global, zero-mean input bases such as Fourier or wavelets paired with local output bases such as piecewise quadratic functions performed best on average in the reported experiments [2312.12578].

## 5. Neural-operator variants: GIT-Net and IAE-Net

In operator learning, the integral-transform viewpoint is developed at the level of mappings between function spaces. GIT-Net formulates operator learning as regression between infinite-dimensional function spaces $U$ and $V$, with training pairs $(u_i,y_i)$ satisfying $y_i=F(u_i)$ and expected risk
$$
\mathbb{E}_{u\sim\mu}\big[\|F(u)-N_\theta(u)\|_V^2\big],
$$
approximated empirically over $N_{\text{train}}$ samples [2312.02450]. Its motivating observation is that many PDE operators are parsimonious in specialized functional bases. The canonical integral form
$$
(\mathcal{T}_\theta u)(x)=\int_{\Omega_u}K_\theta(x,\xi)\,u(\xi)\,d\xi
$$
is written, when diagonalizable in an appropriate basis, as
$$
\mathcal{T}_\theta = \Phi^{-1} M_\theta \Phi.
$$
GIT-Net implements a generalized version of this idea using PCA bases, learnable changes of basis $P,Q\in\mathbb{R}^{K\times K}$, a learned frequency-wise channel mixer $D\in\mathbb{R}^{C\times C\times K}$, and a channel mixer $T\in\mathbb{R}^{C\times C}$ [2312.02450].

For coefficient tensor $\alpha\in\mathbb{R}^{C\times K}$, the learned generalized nonlocal operator is
$$
K(\alpha)=\big((\alpha \mathbf{P})\otimes \mathbf{D}\big)\mathbf{Q},
$$
and the nonlinear GIT layer is
$$
G(\alpha)=\sigma(\mathbf{T}\alpha + K(\alpha)).
$$
With orthonormal basis functions, the corresponding physical-space kernel is written explicitly as
$$
K_{\theta;c,d}(x,\xi)
=
\sum_{l=1}^{K} b_l(\xi)\,
\sum_{k=1}^{K}\mathbf{P}_{l,k}\mathbf{D}_{d,c,k}\,
\sum_{i=1}^{K}\mathbf{Q}_{k,i}\widehat b_i(x).
$$
An $L$-layer GIT-Net then has the form
$$
N = P_v \circ \Phi^\downarrow \circ G^{(L)} \circ \cdots \circ G^{(1)} \circ \Phi^\uparrow \circ A_u,
$$
with GELU used in all but the last layer [2312.02450]. The reported evaluation complexity is
$$
\mathcal{O}(N_p + C^2K + CK^2)=\mathcal{O}(N_p + CK(C+K)),
$$
and a layer uses on the order of $2K^2 + KC^2 + C^2$ parameters rather than $(CK)^2$ for a dense linear map [2312.02450].

IAE-Net takes a different route to integral-transform operator learning by using an integral encoder, a fixed-size bottleneck, and an integral decoder. For a function $a$ on $\Omega$, the encoder maps from an arbitrary input grid $S\subset\Omega$ to a fixed latent grid $S_z\subset\Omega_z$ through
$$
v(z)=\int_\Omega \phi_1(a(x),x,z;\theta_{\phi_1})\,a(x)\,dx,
$$
the bottleneck applies a standard FNN $\phi_0$, and the decoder reconstructs through
$$
b(x)=\int_{\Omega_z}\phi_2(u(z),x,z;\theta_{\phi_2})\,u(z)\,dz.
$$
Numerically, the integral transform
$$
(Tf)(z)=\int_{\Omega_x}K(x,z)f(x)\,dx
$$
is implemented on samples as
$$
(Tf)(z_j)=\frac{1}{s}\sum_{i=1}^s K(x_i,z_j)f(x_i),
$$
or, for nonlinear kernels,
$$
(\bar T f)(z_j)=\frac{1}{s}\sum_{i=1}^s \bar K(f(x_i),x_i,z_j)f(x_i),
$$
with appropriate quadrature weights on nonuniform grids [2203.05142].

IAE-Net composes such blocks densely. Its multi-channel block applies several IAEs in parallel, including an identity channel and transformed channels such as Fourier,
$$
\mathrm{IAE}(a;\theta_{\mathrm{IAE}})
:=
M\big([\Phi_1(a),\mathcal{F}_1^{-1}\Phi_2(\mathcal{F}_1 a),\dots]\big),
$$
and a deep network concatenates all previous outputs in a DenseNet-like fashion before another IAE block [2203.05142]. Discretization invariance is pursued through operator design and a randomized discretization-augmentation objective,
$$
\min_{\theta_{\Psi_n}}
\mathbb{E}_{(\bar f,\bar g)\sim p_{\text{data}}}
\mathbb{E}_{IT\sim\mathcal{I}}
\big[
L(\Psi_n(\bar f),\bar g)
+
\lambda L(\Psi_n(IT(\bar f)),IT(\bar g))
\big],
$$
with $\lambda=1$ by default and cubic or bicubic interpolation for resampling [2203.05142]. The supervised loss is the relative $L_2$ error,
$$
\mathcal{L}(x,y)=\frac{\|x-y\|_2}{\|y\|_2}.
$$

Empirically, GIT-Net is evaluated on Navier–Stokes, Helmholtz, structural mechanics, 1D advection, and Poisson problems, using $L=3$ GIT layers, GELU, PCA truncation preserving at least $99.999\%$ energy capped at $P=200$, channel counts $C\in\{2,4,8,16,32\}$, frequencies $K\in\{16,64,128,256,512\}$, and training sizes $N_{\text{train}}\in\{2500,5000,10000,20000\}$ [2312.02450]. In the large-data regime $N_{\text{train}}=20{,}000$, it is reported to achieve the smallest test error across all PDE problems, and on complex geometries it is described as substantially more accurate than FNO because the latter suffered from interpolation artifacts [2312.02450]. IAE-Net is evaluated on 1D Burgers, 2D Darcy flow, Darcy on a general domain, forward and inverse scattering, CT image denoising, and signal source separation, with four IAE blocks, channel width $w=64$, a 2-channel identity-plus-Fourier design, Adam at learning rate $10^{-3}$, 500 epochs, and a half-on-plateau scheduler [2203.05142]. Representative results include Burgers with $\nu=10^{-1}$, where IAE-Net reports $\approx 2.75\mathrm{e}{-4}$ relative error versus FNO $\approx 7.24\mathrm{e}{-4}$, and CT denoising at $s=128$, where IAE-Net reports $\approx 5.99\%$ versus FNO $\approx 17.8\%$ [2203.05142].

## 6. Empirical profile, computational trade-offs, and scope

The explicit 2026 ITNet is evaluated as a single shared operator across images, text, point clouds, and multimodal reasoning, with only lightweight modality-specific encoders changing across tasks [2606.19538]. Reported headline results are ImageNet-1K top-1 of $81.4\pm0.2\%$ for ITNet-S (22M), $83.9\pm0.1\%$ for ITNet-B (86M), and $85.8\pm0.1\%$ for ITNet-L (307M); GLUE dev-set averages of $81.5\pm0.2$ for ITNet-B and $83.1\pm0.2$ for ITNet-L; ModelNet40 overall accuracy of $93.5\pm0.2$ for ITNet-PC (3.1M) with local $K=16$ and $94.6\pm0.1$ for ITNet-B; and multimodal scores of $78.4\pm0.2$ on VQA v2 and $82.1\pm0.2$ on NLVR2 for ITNet-B, rising to $83.6\pm0.2$ and $84.1\pm0.2$ for ITNet-L [2606.19538]. The same source states that tiled exact ITNet is within approximately $0.8\times$ the throughput of FlashAttention-2 at moderate sequence length, citing DeiT-S throughput of $1820$ img/s versus ITNet-S at $1480$ img/s for $196$ patches, while low-rank ITNet yields $2.1$–$3.9\times$ higher throughput with less than $0.5$–$0.1\%$ accuracy loss at ranks $r=96$–$192$ on ImageNet-1K [2606.19538].

The main computational issue is that naïve exact evaluation materializes an $N\times N$ matrix-valued kernel at cost $O(N^2C^2)$ time and memory [2606.19538]. Three remedies are given: tiled kernel fusion, importance-weighted Monte Carlo integration, and learned low-rank factorization. Tiled fusion avoids materializing the global kernel and reduces peak memory to $O(NC)+O(B_qB_kC^2)$; Monte Carlo reduces complexity to $O(NMC^2)$ for $M\ll N$ with an unbiased estimator; and low-rank factorization reduces the operator to $O(NRC)$, with the bound
$$
\|\kappa-\kappa^{(R)}\|_F \le \frac{\|\kappa\|_*}{\sqrt{R}}
$$
and reported relative error below $1\%$ at $R=32$ in ImageNet experiments [2606.19538]. By comparison, IAT layers incur $O((d+m)M)$ cost per layer under midpoint quadrature, GIT-Net layers evaluate in $\mathcal{O}(N_p + CK(C+K))$, and each IAE encoder or decoder application is a matrix–vector product with cost $O(|S_z|\cdot|S|)$ [2312.12578][2312.02450][2203.05142].

The limitations are architecture-specific rather than uniform across the entire integral-transform family. For the explicit ITNet, exact computation is quadratic without approximation, very long contexts require low-rank or sampling approximations, training stability depends on careful initialization and normalization, strict token-by-token autoregressive decoding still benefits from scanning-friendly parameterizations, and interpretability of the learned kernel $\kappa_\theta$ is challenging [2606.19538]. For IAT-based models, computational overhead grows with discretization $M$, performance is sensitive to basis choice, discretization introduces piecewise linearity, and higher-order derivatives remain problematic in the discrete IAT-ReLU setting [2312.12578]. For GIT-Net, the basis-induced bias is that operators are assumed near-diagonal in an appropriate basis, explicit translation equivariance is lost when Fourier bases are replaced by PCA, and the paper does not present new formal approximation theorems [2312.02450]. For IAE-Net, assembling and applying full kernel matrices becomes expensive at very high resolutions, latent size and kernel-network capacity require tuning, and improved quadrature may be needed for highly irregular sampling [2203.05142].

Across these variants, the shared structural motif is the replacement of fixed local kernels, scalar dot-product attention weights, or discretization-bound matrices by learned integral operators acting either over interactions, activations, or function spaces. A plausible implication is that “Integral Transform Network” now names a family of architectures rather than a single immutable blueprint. What remains non-negotiable in the literature is the distinction from the 2018 point-cloud IT-Net, whose subject is iterative rigid $SE(3)$ canonicalization rather than integral transforms [1811.11209].

Source: https://www.emergentmind.com/topics/integral-transform-network-itnet