ITNet: A Learnable Integral Transform That Subsumes Convolution, Attention, and Recurrence

Abstract: Convolutional networks, recurrent networks, and transformers each encode different inductive biases -- locality, sequential memory, and content-dependent pairwise interaction -- and have remained mathematically distinct since their inception. We show that this fragmentation reflects not a fundamental diversity in how signals should be processed, but rather incomplete views of a single underlying mathematical object: a learnable integral transform. We introduce the Integral Transform Network (ITNet), a unified architecture built around a learnable kernel that depends jointly on positions and features. This kernel is implemented as a small neural network, specifically an MLP, that models pairwise interactions, enabling the model to adapt its behavior from data. We show that convolution, self-attention (including multi-head), and autoregressive recurrence (including LSTM, GRU, S4, and Mamba) arise as special cases under appropriate parameterizations, and that ITNet is a universal approximator of continuous operators. To make this practical, we develop tiled kernel fusion, importance-weighted Monte Carlo integration, and learned low-rank factorization, enabling efficient and scalable computation. A single ITNet architecture with a shared operator and lightweight modality-specific encoders matches or exceeds specialized baselines on ImageNet-1K , GLUE, ModelNet40, VQA\,v2 and NLVR2. The results demonstrate that a single learned interaction mechanism can recover the behavior of all three architectural families from data.

• The paper introduces ITNet, a unified integral transform that subsumes convolution, attention, and recurrence by learning content and position-dependent interactions.
• It employs a matrix-valued kernel along with techniques like tiled kernel fusion, Monte Carlo integration, and low-rank factorization to ensure efficient computation.
• Empirical results across vision, language, and 3D modalities demonstrate ITNet's universal approximation capabilities and superior performance over specialized architectures.

Integral Transform Network (ITNet): Unified Operator Framework for Deep Learning Architectures

Introduction and Motivation

The paper "ITNet: A Learnable Integral Transform That Subsumes Convolution, Attention, and Recurrence" (2606.19538) addresses architectural fragmentation in deep learning by proposing a single operator—the Integral Transform Network (ITNet)—that generalizes convolution, attention, and recurrence under a unified learnable integral transform parameterized by a matrix-valued neural kernel. Unlike existing approaches with hard-coded inductive biases and architecture-specific domains, ITNet learns content and position-dependent pairwise interactions directly from data. This operator-centric view subsumes CNNs, Transformers, and RNN-based models as exact special cases, thereby challenging the premise that domain-specific architectures are necessary for processing structured signals.

ITNet Operator: Mathematical Formulation and Architectural Structure

ITNet operates on a domain $\Omega$ with features $u:\Omega \to \mathbb{R}^d$. The core operator is:

$$(K[u])(x) = \int_{\Omega} \kappa_\theta(x, y, u(x), u(y))\, u(y)\, d\mu(y) + W_\theta u(x),$$

where $\kappa_\theta$ is a learnable matrix-valued kernel implemented by a shallow MLP mapping positions and features to $d \times d$ matrices, and $W_\theta$ is a residual connection. The kernel input encompasses absolute and relative position encodings (Fourier-lifted), Euclidean distance, endpoint features, and their Hadamard product.

The architecture stacks $L$ such operator blocks with pre-norm layer normalization and per-position FFNs, following the Transformer meta-structure but extending its operator space.

Figure 1: ITNet's architecture: Stacked pre-norm layers, each with (i) layer normalization, (ii) the integral operator with position and content-dependent kernel, and (iii) a position-wise FFN; residual connections enhance stability. The kernel generalizes CNNs, attention, and recurrence within a single unified stack.

Theoretical Foundations: Unification and Expressiveness

Subsumption Theorems

ITNet demonstrates, via constructive kernel parameterizations, that classical architectures are embedded as special cases:

• Convolution: For $\kappa_\theta(x, y, u(x), u(y)) = w_\theta(x-y) I$, ITNet reduces to convolution across $\Omega$, covering discrete and continuous, grouped, and depthwise variants.
• Self-Attention: Setting $\kappa_\theta(x, y, u(x), u(y))$ to softmax-normalized bilinear forms reproduces Transformer-style attention with optional multi-head structure.
• Recurrence: A causal kernel ($u:\Omega \to \mathbb{R}^d$0 for $u:\Omega \to \mathbb{R}^d$1) and appropriate parameterization recovers RNNs, LSTMs, SSM/SSMs (S4/Mamba), and discrete autoregressive recurrence.

Universal Operator Approximation

ITNet is proven to be a universal approximator of continuous operators from compact subsets of $u:\Omega \to \mathbb{R}^d$2 to themselves. Any map that can be realized continuously with respect to the sup-norm can be $u:\Omega \to \mathbb{R}^d$3-approximated by stacking ITNet layers with sufficiently wide MLP kernels. This strictly generalizes the operator classes expressible by CNNs, attention, and RNNs:

$u:\Omega \to \mathbb{R}^d$4

Explicit counterexamples (e.g., non-causal or non-equivarient operators) illustrate strict inclusions.

Data Symmetry and Kernel Recovery

Under translation-invariant data distributions, gradient flow dynamics in ITNet ensure convergence toward translation-invariant kernels, thereby automatically recovering convolutional behavior when appropriate, without explicit architectural constraints.

Scalable Implementation and Efficient Computation

To mitigate the $u:\Omega \to \mathbb{R}^d$5 cost of the general operator, several mechanisms are introduced:

• Tiled Kernel Fusion: Inspired by FlashAttention, only $u:\Omega \to \mathbb{R}^d$6 tile blocks are ever materialized in SRAM, jointly evaluating the kernel MLP and integral in hardware-efficient kernels.
• Monte Carlo Integration: Reduces complexity to $u:\Omega \to \mathbb{R}^d$7 by sampling $u:\Omega \to \mathbb{R}^d$8 keys per query using a learnable proposal, with a principled cross-entropy loss ensuring unbiasedness and variance minimization.
• Low-Rank Kernel Factorization: The kernel is factorized as $u:\Omega \to \mathbb{R}^d$9, reducing complexity to $$(K[u])(x) = \int_{\Omega} \kappa_\theta(x, y, u(x), u(y))\, u(y)\, d\mu(y) + W_\theta u(x),$$0 for rank $$(K[u])(x) = \int_{\Omega} \kappa_\theta(x, y, u(x), u(y))\, u(y)\, d\mu(y) + W_\theta u(x),$$1, with tight error control via nuclear norm bounds.

These allow practical deployment on long sequences and high-dimensional data within modern accelerator memory budgets.

Empirical Results Across Modalities

Strong numerical results are reported for three model scales (S/B/L: 22M/86M/307M params) across standard benchmarks:

• ImageNet-1K: ITNet-L achieves 85.8% top-1, exceeding parameter-matched ConvNeXt-V2-B (84.2%), BiFormer-B (84.4%), and EfficientVMamba-B (84.0%), with ITNet-B matching or surpassing DeiT-B, Swin-B, and ConvNeXt-B.
• GLUE: ITNet-(B/L) closely matches BERT-large and RoBERTa-base (when normalized for pretraining data), with best results on syntactic tasks, indicating enhanced modeling of long-range structural dependencies.
• ModelNet40: ITNet-B attains 94.6% OA, matching state-of-the-art point cloud methods, with similar patterns in modality-specific ablations.
• Multimodal (VQA v2, NLVR2): ITNet-L achieves 83.6% and 84.1% respectively, outperforming BLIP (ViT-B), ALBEF, and other fusion-centric transformer baselines, with a single joint-domain kernel.

Ablations reveal that both content and position conditioning are essential for optimal performance, and the kernel adapts its inductive bias depending on input modality; geometry dominates in vision, content in language and 3D recognition. Removing the Hadamard interaction or relative position terms consistently degrades results.

Implications and Potential for Future Architectures

ITNet's results support the thesis that the diversity of modern neural architectures can be reduced to learnable parametrizations of a generalized joint interaction kernel, with inductive biases (locality, content-dependence, causality) emerging from the dataset. Practically, this enables deployment of a single operator (with lightweight modality-specific encoders) across vision, language, multi-modal, and geometric domains, reducing engineering overhead and facilitating cross-domain generalization.

Theoretically, ITNet sets the stage for further research on universal operator learning, modular kernel architectures, and efficient high-rank parameterizations. The path to autoregressive modeling (for long-context language tasks) via causal kernel constraints is outlined as a next step. In addition, the ability to learn cross-modal kernel functions naturally addresses multimodal fusion without explicit cross-attention or bottlenecked latent arrays, suggesting a unification avenue for multi-foundation models.

Conclusion

ITNet provides a mathematically principled and empirically validated unification of convolution, attention, and recurrence within a single learnable integral operator framework (2606.19538). With universal approximation guarantees, efficient implementations, and strong performance across multiple modalities, ITNet demonstrates that the apparent need for task-specific neural architectures can often be replaced by a data-driven kernel learning paradigm. This advances both the practical frontiers of modality-agnostic AI systems and the theoretical understanding of the inductive biases underpinning deep learning operators.