KVNN: Learnable Multi-Kernel Volterra Neural Networks

Abstract: Higher-order learning is fundamentally rooted in exploiting compositional features. It clearly hinges on enriching the representation by more elaborate interactions of the data which, in turn, tends to increase the model complexity of conventional large-scale deep learning models. In this paper, a kernelized Volterra Neural Network (kVNN) is proposed. The key to the achieved efficiency lies in using a learnable multi-kernel representation, where different interaction orders are modeled by distinct polynomial-kernel components with compact, learnable centers, yielding an order-adaptive parameterization. Features are learned by the composition of layers, each of which consists of parallel branches of different polynomial orders, enabling kVNN filters to directly replace standard convolutional kernels within existing architectures. The theoretical results are substantiated by experiments on two representative tasks: video action recognition and image denoising. The results demonstrate favorable performance-efficiency trade-offs: kVNN consistently yields reduced model (parameters) and computational (GFLOPs) complexity with competitive and often improved performance. These results are maintained even when trained from scratch without large-scale pretraining. In summary, we substantiate that structured kernelized higher-order layers offer a practical path to balancing expressivity and computational cost in modern deep networks.

• The paper introduces kVNN, a learnable multi-kernel Volterra layer that models first-, second-, and third-order feature interactions with compact, order-specific centers and integrates directly into CNNs.
• The paper shows that kVNN-B with third-order filters reaches 92.67% UCF101 accuracy from scratch using 30M parameters and 35.18 GFLOPs, outperforming larger or pretrained baselines.
• The paper demonstrates practical efficiency gains in denoising, including roughly one-third fewer parameters and FLOPs while maintaining or improving PSNR on synthetic and real-world noise benchmarks.

Motivation and problem statement

The core computational unit of most modern deep networks remains a linear operator—convolution or linear projection—with nonlinearity supplied only by pointwise activations and depth. Higher-order, multiplicative interactions among features are therefore synthesized implicitly across layers at the cost of increased model complexity. Explicit higher-order filtering via truncated Volterra series captures such couplings directly, but the number of free coefficients in an order-$r$ term scales as $\mathcal{O}(d^r)$ in input dimension $d$, making direct deployment in large architectures impractical. Classical kernel methods offer a principled alternative: a degree-$r$ homogeneous polynomial kernel $K_r(x,x') = (x^\top x')^r$ induces an RKHS whose elements are exactly degree-$r$ homogeneous polynomials, so Volterra-type nonlinearities become linear functionals in feature space. However, the representer-theorem expansion $f(x)=\sum_{j=1}^{N}\gamma_j K(x,x_j)$ scales with the training-set size $N$, which is untenable at scale, and existing scalable approximations (Nyström, random features) do not integrate cleanly into end-to-end deep pipelines under fixed parameter budgets.

The paper proposes kVNN, a kernelized Volterra layer that resolves this tension through a learnable multi-kernel representation: each interaction order is modeled by its own degree-$r$ kernel component with a compact set of learnable centers, decoupling the basis from training samples. The result is a plug-and-play module that replaces standard convolutional kernels within existing CNNs and trains end-to-end.

Learnable multi-kernel representation

A $p$-th order truncated Volterra mapping decomposes into homogeneous components $\mathcal{O}(d^r)$0, where each $\mathcal{O}(d^r)$1 collects all $\mathcal{O}(d^r)$2-way monomial interactions. The central design decision is to approximate each component with its own kernel expansion,

$\mathcal{O}(d^r)$3

with order-specific learnable centers $\mathcal{O}(d^r)$4 and coefficients $\mathcal{O}(d^r)$5, where $\mathcal{O}(d^r)$6. The full model is then $\mathcal{O}(d^r)$7, interpretable as living in the direct sum of RKHSs $\mathcal{O}(d^r)$8 induced by the kernels $\mathcal{O}(d^r)$9. This order-wise separation avoids redundant cross terms and lets each order capture distinct manifold structure; the authors motivate the local, center-based approximation by the geometry of the data manifold (reach and curvature), contrasting it with the global representer expansion.

Two theoretical results support the construction. First, a multi-kernel validity theorem shows that any weighted combination $d$0 with $d$1 is positive semidefinite and that every function in its RKHS admits an exact order-wise decomposition into the constituent spaces. Second, a finite atomic representation theorem establishes that the atoms $d$2 span the entire finite-dimensional space of degree-$d$3 homogeneous polynomials—identifying these atoms with rank-one symmetric tensors $d$4, which span the full symmetric tensor space. Consequently, any truncated Volterra mapping admits an exact representation of the proposed form with finitely many centers. This guarantee is what licenses treating the centers as freely learnable parameters rather than fixed or sample-derived anchors: the parameterization is expressive enough to recover any target in the class exactly.

Layer construction

Each kVNN filter instantiates one output channel as parallel branches summed across orders. An order-1 branch uses a single linear atom $d$5; an order-2 branch aggregates $d$6 quadratic atoms $d$7; an optional order-3 branch aggregates $d$8 cubic atoms. Centers and coefficients for different orders are learned independently, avoiding the expressivity loss of shared centers. The atom counts $d$9 and $r$0 are structural hyperparameters tuned to data complexity—in the experiments, $r$1 for image tasks and $r$2 for video tasks. Because each filter maps to one convolutional kernel position, layers of such filters drop directly into standard CNN topologies, and all parameters train end-to-end under the task loss. The authors note the approach extends to arbitrary order with manageable growth, though experiments cover only $r$3.

Video action recognition results

Evaluation uses a two-stream pipeline (RGB plus optical flow encoders, concatenation fusion, linear classifier) on UCF101 and HMDB51, with two capacity variants (kVNN-S, 4 blocks per stream; kVNN-B, 5 blocks) crossed with filter orders 2 and 3. All kVNN models are trained from scratch—no Kinetics pretraining—whereas several compared baselines rely on large-scale pretraining.

Model	Params	GFLOPs	UCF101 acc
kVNN-S (2nd order)	7.7M	15.96	86.51%
kVNN-S (3rd order)	12.29M	28.14	90.02%
kVNN-B (2nd order)	17M	19.86	91.17%
kVNN-B (3rd order)	30M	35.18	92.67%
I3D	56M	63.34	88.8%
Video-FocalNet tiny*	49M	63	90.1%
DVFL-Net*	22M	27	88.4%

(* denotes Kinetics pretraining.)

The headline claim is strong: kVNN-B with third-order filters reaches 92.67% on UCF101—the best among all compared methods, including pretrained ones—at roughly half the parameters and compute of I3D. Notably, kVNN-B (2nd order) attains 91.17% with only 19.86 GFLOPs, undercutting even the lightweight DVFL-Net's compute while exceeding its accuracy by nearly three points. On HMDB51, kVNN-B (3rd order) achieves 67.12%, close to the best pretrained results (IMR-Net* at 67.8%), whereas IMR-Net trained from scratch drops to 55.9%—a contrast that underscores the claimed benefit of structured higher-order representations when pretraining is unavailable. The implication is that explicit second/third-order interaction modeling can substitute for both architectural scale and pretraining on these benchmarks.

Image denoising results

Three denoising protocols test whether the gains transfer to low-level vision under matched training conditions, replacing Conv blocks with kVNN blocks in DnCNN, FFDNet, DCANet, and U-Net backbones.

Fixed-$r$4 AWGN (BSD68): a reduced-depth kVNN variant of DnCNN cuts parameters by ~33% (557K → 373K) and GFLOPs by ~34% (2.285 → 1.517) while matching or slightly improving PSNR at $r$5 (e.g., 31.73 → 31.76 dB at $r$6).

Random-$r$7 AWGN (Set12, single model, $r$8): kVNN counterparts improve PSNR across all three backbones at lower cost—DnCNN +0.14 dB (30.31 → 30.45), FFDNet +0.14 dB, DCANet +0.09 dB—with ~33%/34% reductions in params/GFLOPs for the first two. Visual comparisons show sharper boundaries and better texture recovery for the kVNN-based DCANet.

Real-world denoising (SIDD Medium, self-supervised spatially adaptive framework): replacing convolutions in the final U-Net denoiser improves PSNR from 37.29 to 37.47 dB at same depth (1.2M vs. 1.0M params); a reduced-depth kVNN variant reaches 37.35 dB with only 0.7M parameters. This demonstrates compatibility with a CNN-tailored self-supervised recipe without bespoke modifications.

Across both tasks, the consistent pattern—equal or better accuracy at reduced depth, parameters, and FLOPs, without task-specific modules—supports the paper's claim that the improvement stems from the operator itself rather than from architecture tuning.

Limitations and open questions

Several caveats qualify the results. The theoretical expressivity guarantee is exact but non-constructive regarding how many atoms suffice in practice; the chosen values ($r$9 for images, $K_r(x,x') = (x^\top x')^r$0 for video) are empirical hyperparameters rather than derived from manifold geometry, despite the geometric motivation offered. Experiments cover only orders $K_r(x,x') = (x^\top x')^r$1; behavior at higher orders, where the compactness argument matters most, is asserted but not demonstrated. The video evaluation uses a two-stream pipeline requiring precomputed optical flow, so end-to-end efficiency claims do not include flow extraction cost, and comparisons against pretrained baselines conflate operator quality with training regime in ways the from-scratch protocol mitigates but does not fully eliminate. Finally, integration into transformer-based architectures—specifically whether kVNN-style higher-order interactions benefit token-mixing modules—is proposed as future work and left untested.

Conclusion

kVNN translates truncated Volterra filtering into a practical deep-learning operator via a learnable multi-kernel parameterization with order-specific centers, backed by a finite atomic representation theorem guaranteeing exactness of the function class. Empirically, the layer functions as a drop-in convolutional replacement: it achieves state-of-the-art UCF101 accuracy (92.67%) from scratch at roughly half the cost of comparable models, and consistently reduces parameters and GFLOPs by about a third in denoising while matching or improving PSNR across synthetic and real-noise settings. The main open questions concern scaling to higher interaction orders, principled selection of atom counts, and extension beyond convolutional backbones.