KVarN: Neural Volterra Networks & KV Quantization

• KVarN is a dual-method framework that integrates learnable multi-kernel Volterra neural network layers for higher-order interactions in CNNs with a variance-normalized 2-bit KV-cache quantization scheme for Transformers.
• It achieves significant efficiency gains by reducing parameters and FLOPs in convolutional architectures while simultaneously boosting accuracy in video action recognition and image denoising benchmarks.
• The method employs structured representations, including parallel polynomial kernel branches and head-wise Hadamard rotations with dual-axis normalization, to mitigate quantization errors and maintain robust performance during long-context autoregressive decoding.

KVarN refers to two distinct, state-of-the-art methods within the context of deep learning and LLMs: (1) a learnable multi-kernel Volterra neural network layer for higher-order compositional learning in convolutional architectures, and (2) a variance-normalized quantization scheme for Transformer KV-cache compression in long-context generative models. Both approaches establish new trade-offs in expressivity, computational efficiency, and resource constraints by leveraging structured representations (multi-kernel expansions and dual-axis normalization) for their respective domains (Yun et al., 16 Apr 2026, Muller et al., 2 Jun 2026).

1. Kernelized Volterra Neural Networks (kVNN / KVarN) for Higher-Order Representation

The KVarN framework generalizes standard convolutional layers by parameterizing each filter as a learnable sum of polynomial kernel atoms of increasing order. The construction is rooted in the discrete Volterra series, where a function of an input vector $x\in\mathbb{R}^d$ is decomposed as

$$f(x) = \sum_{r=1}^p f_r(x),\quad \text{with } f_r(x) = \sum_{i_1,\dots,i_r=1}^d h_r(i_1,\dots,i_r)\,x_{i_1}\cdots x_{i_r},$$

where $h_r$ encodes $r$-way interactions and is intractable to learn directly for moderate $d, r$ due to exponential parameter scaling.

By exploiting that all homogeneous polynomials of degree $r$ reside within the RKHS of a polynomial kernel $k_r(x, c) = (x^\top c)^r$, KVarN uses compact, learnable centers $w_{r,i}\in\mathbb{R}^d$ and weights $\gamma_{r,i}\in\mathbb{R}$ to approximate each $f_r$ as a sum of $$f(x) = \sum_{r=1}^p f_r(x),\quad \text{with } f_r(x) = \sum_{i_1,\dots,i_r=1}^d h_r(i_1,\dots,i_r)\,x_{i_1}\cdots x_{i_r},$$0 kernel atoms: $$f(x) = \sum_{r=1}^p f_r(x),\quad \text{with } f_r(x) = \sum_{i_1,\dots,i_r=1}^d h_r(i_1,\dots,i_r)\,x_{i_1}\cdots x_{i_r},$$1 Aggregating over orders up to $$f(x) = \sum_{r=1}^p f_r(x),\quad \text{with } f_r(x) = \sum_{i_1,\dots,i_r=1}^d h_r(i_1,\dots,i_r)\,x_{i_1}\cdots x_{i_r},$$2 (commonly $$f(x) = \sum_{r=1}^p f_r(x),\quad \text{with } f_r(x) = \sum_{i_1,\dots,i_r=1}^d h_r(i_1,\dots,i_r)\,x_{i_1}\cdots x_{i_r},$$3), the filter output becomes

$$f(x) = \sum_{r=1}^p f_r(x),\quad \text{with } f_r(x) = \sum_{i_1,\dots,i_r=1}^d h_r(i_1,\dots,i_r)\,x_{i_1}\cdots x_{i_r},$$4

where $$f(x) = \sum_{r=1}^p f_r(x),\quad \text{with } f_r(x) = \sum_{i_1,\dots,i_r=1}^d h_r(i_1,\dots,i_r)\,x_{i_1}\cdots x_{i_r},$$5 represents the bias and $$f(x) = \sum_{r=1}^p f_r(x),\quad \text{with } f_r(x) = \sum_{i_1,\dots,i_r=1}^d h_r(i_1,\dots,i_r)\,x_{i_1}\cdots x_{i_r},$$6 are polynomial kernels. Each order $$f(x) = \sum_{r=1}^p f_r(x),\quad \text{with } f_r(x) = \sum_{i_1,\dots,i_r=1}^d h_r(i_1,\dots,i_r)\,x_{i_1}\cdots x_{i_r},$$7 operates in a separate parallel branch. This parameterization subsumes standard convolution ($$f(x) = \sum_{r=1}^p f_r(x),\quad \text{with } f_r(x) = \sum_{i_1,\dots,i_r=1}^d h_r(i_1,\dots,i_r)\,x_{i_1}\cdots x_{i_r},$$8), while higher-order branches capture richer factorized multi-way interactions.

2. Architectural Formulation, Complexity, and Integration

For a convolutional layer of spatial size $$f(x) = \sum_{r=1}^p f_r(x),\quad \text{with } f_r(x) = \sum_{i_1,\dots,i_r=1}^d h_r(i_1,\dots,i_r)\,x_{i_1}\cdots x_{i_r},$$9 and $h_r$0 input channels, a KVarN layer operates directly on vectorized local patches ($h_r$1). Each filter is a sum over $h_r$2 branches, with configurable atom counts $h_r$3 per order. The parameters per filter are $h_r$4, as each atom contains a center and a scalar weight.

The output for all patches can be implemented via batched $h_r$5 convolutions for each $h_r$6, followed by per-branch elementwise power and reduction (dot with $h_r$7), then summed across $h_r$8. Default configurations use $h_r$9 with very low $r$0, ensuring that $r$1.

Parameter and FLOP complexity for standard convolution ($r$2 weights, $r$3 FLOPs/output position) is exceeded only for very high $r$4 or $r$5. Empirically, KVarN achieves a net reduction in parameters and operations while significantly increasing representational power (Yun et al., 16 Apr 2026).

KVarN filters can directly replace convolutional kernels in “Conv–BN–ReLU” blocks, making the approach compatible with any modern CNN backbone. No large-scale pretraining is required, and models are trained from scratch with standard optimizers (SGD/Adam), using mild $r$6 regularization, often modulated across layers.

3. Empirical Outcomes and Efficiency Trade-offs

KVarN achieves substantial improvements in expressivity–efficiency trade-offs. On video action recognition benchmarks (UCF101, ResNet-style architectures), third-order KVarN with $r$7 delivers higher accuracy than I3D at 46% the parameter count and 44% the GFLOPs (Yun et al., 16 Apr 2026). For image denoising (BSD68, DnCNN backbone), KVarN matches or surpasses baseline PSNR with $r$833% fewer parameters and $r$934% fewer FLOPs.

Model	Params	GFLOPs	Accuracy/PSNR
kVNN-S (3rd ord)	12.3M	28.14	90.02% (UCF101)
I3D Baseline	56M	63.34	88.8% (UCF101)
DnCNN (CNN)	557,057	2.285	29.23dB (BSD68)
DnCNN (kVNN)	372,545	1.517	29.27dB (BSD68)

These results underscore that structured kernelized higher-order layers offer a viable practical route for balancing expressivity and computational footprint in deep architectures.

4. KVarN for KV-Cache Quantization in Transformers

A separate method, also titled KVarN, is a variance-normalized scheme for 2-bit quantization of the key-value (KV) cache in Transformers during long-horizon autoregressive decoding (Muller et al., 2 Jun 2026). Transformer decoding relies on a growing KV-cache: in each layer $d, r$0 and each new token $d, r$1, $d, r$2 and $d, r$3 vectors are appended; this history is needed for attention, but, as $d, r$4 increases, unquantized caches impose prohibitive memory costs.

The KVarN quantizer improves over one-axis quantization by applying (i) a head-wise Hadamard rotation, decorrelating channels and normalizing outliers, followed by (ii) a dual-scaling variance normalization across both token and channel axes. After these linear transforms, a uniform 2-bit quantizer ($d, r$5) is applied with per-channel (for $d, r$6) or per-token (for $d, r$7) groupings in blocks of $d, r$8 elements.

Normalization is defined for a $d, r$9 block $r$0 as: $r$1 and then

$r$2

This ensures per-token and per-channel variances are controlled, mitigating scale errors that would otherwise multiply across timesteps during decoding.

5. Error Accumulation Mitigation and Benchmark Outcomes

Conventional one-axis quantization suffers from scale errors: reconstructed tokens may be scaled by a factor $r$3, and, due to autoregressive token dependencies, downstream activations accumulate $r$4, causing exponential drift or vanishing. KVarN’s dual normalization enforces $r$5 at each step, eliminating such accumulated distortion.

Empirically, KVarN establishes new state-of-the-art results for 2-bit KV-cache quantization on:

• AIME24 (Qwen3-4B): 60.0% accuracy (vs. 55.5% for KIVI, 56.7% for QuaRot)
• MATH500 (Qwen3-4B): 79.2% (vs. 77.8% for KIVI, 78.9% for QuaRot)
• HumanEval (Qwen3-4B): 88.4% (vs. 86.4% for KIVI, 86.3% for QuaRot) Additionally, attention-output MAE under accumulation is halved relative to KIVI at 30k-token contexts.

6. Implementation, Overheads, and Applicability

KVarN quantization is designed for vLLM-based LLM serving: every 128-token block triggers (1) Hadamard rotation (fused within the pipeline), (2) dual-scaling normalization (via 8–10 Sinkhorn iterations in log-domain), (3) 2-bit quantization. Dequantization uses a single fast Triton kernel given codes, a zero-point (FP16), and two FP8 scales per block. The overhead is minimal, constituting ≈0.18% of decode time for quantization and <1% for dequantization.

Total memory overhead is 2 bits per element plus auxiliary scale and zero-point storage, yielding ≈2.25 bpe effective memory. The method is specific to Transformers with explicit KV caches and does not generalize to state-space models. Wider adoption will require LLM stacks to natively support 2-bit compressed caches.

7. Limitations and Future Research Directions

KVarN’s direct dependence on group size $r$6 and the number of Sinkhorn iterations exposes potential stability issues for very small or large quantization groups. The approach is not applicable to architectures lacking explicit KV caches. Several future research avenues are suggested:

• Integration with adaptive codebook or mixed-precision quantizers,
• Joint layer-wise optimization of rotation and normalization matrices,
• Extension of dual-scaling normalization to $r$7-cache or attention score matrices,
• Dynamic, model-specific calibration to minimize reconstruction error.

A plausible implication is that dual-dimension normalization paired with incoherence-inducing rotations provides a general template for low-bit quantization in other sequential-memory–intensive architectures.

---

References:

• Learnable Multi-Kernel Volterra Neural Networks (kVNN, “KVarN”) (Yun et al., 16 Apr 2026)
• KVarN: Variance-Normalized KV-Cache Quantization (Muller et al., 2 Jun 2026)