Dual-Dimension Variance Normalization

• Dual-dimension variance normalization is a method that rescales data along two axes simultaneously, enforcing prescribed variance constraints for balanced, scale-invariant analysis.
• It employs iterative algorithms like Sinkhorn–Knopp to compute multiplicative scaling factors, ensuring symmetry and preserving ratio-scale properties.
• The technique is applied in fields such as neural network training, spectral clustering, and quantized inference, thereby improving noise robustness and convergence.

Dual-dimension variance normalization is a class of normalization techniques that enforce prescribed scale or variance constraints simultaneously along two matrix or tensor axes, typically rows and columns, or other structurally meaningful axes such as batch and channel or feature and sample. This normalization is inherently multiplicative and symmetric, in contrast to classic single-axis approaches such as z-scoring, and is foundational in domains such as matrix normalization, neural network training, kernel-based data analysis, and quantization for efficient inference.

1. Theoretical Foundations and Mathematical Formulation

The core objective of dual-dimension variance normalization is to transform a matrix $A\in\mathbb{R}^{m\times n}$ into a canonical, “scale-invariant” form $S$ by finding positive scaling vectors $\alpha\in\mathbb{R}^m$, $\beta\in\mathbb{R}^n$ (often absorbed into diagonal matrices $D_r$, $D_c$), such that

$A = D_r S D_c\ ,$

with

$$D_r = \operatorname{diag}(\alpha),\quad D_c = \operatorname{diag}(\beta),$$

and $S$ satisfies prescribed variance/allocation along both axes. In projective decomposition, the constraints are typically

$$\left(\frac{1}{n} \sum_{j=1}^n S_{ij}^2\right)^{1/2} = 1\quad (\text{for all rows }i), \quad \left(\frac{1}{m} \sum_{i=1}^m S_{ij}^2\right)^{1/2} = 1\quad (\text{for all columns }j).$$

This process multiplicatively rebalances both axes to a common scale, ensuring symmetry and preserving all relative ratios—properties that underpin invariance for ratio-scale data and enable joint analysis of row and column structures (Robinson, 2019).

2. Algorithmic Realizations and Computation

The Sinkhorn–Knopp algorithm and its variants are central for computing the dual-axis scaling factors in both classical and modern settings. For a general matrix, the following iterative scheme is deployed:

• Initialize all scales to 1.
• Alternate between:
  • Row normalization: divide each row by its root-mean-square (RMS) and update the corresponding scaling factor.
  • Column normalization: divide each column by its RMS and update the corresponding scaling factor.
• Iterate to convergence, i.e., until all row and column RMSs are sufficiently close to the prescribed value (usually 1).

This method converges under broad conditions, producing the canonical doubly RMS-normalized matrix and unique scaling factors up to a global scalar (Robinson, 2019). The same iterative proportional scaling underpins doubly-stochastic kernel normalization, where the scaling guarantees that both row-sums and column-sums of the affinity matrix are unity (Landa et al., 2020), as well as recent quantization pipelines in deep learning (Muller et al., 2 Jun 2026).

3. Varieties and Domain-specific Implementations

3.1 Projective Decomposition in Data Normalization

Projective decomposition (also called dual-dimension variance normalization) applies to real-valued data matrices, especially those representing ratio-scale measurements. Unlike z-transformation, which only acts on columns (or rows) via subtraction and division, projective decomposition rescales both rows and columns multiplicatively, ensuring all axes are variance-normalized and relative ratios are preserved globally. This approach is especially relevant when the ratio between matrix elements conveys the scientific signal, as in gene expression or contingency tables (Robinson, 2019).

3.2 Doubly-stochastic Normalization of Affinity Matrices

A key use case arises in spectral and kernel methods, where pairwise similarities are encoded in an affinity matrix (e.g., by applying a Gaussian kernel to pairs of vectors). The doubly-stochastic normalization seeks positive vectors $S$0 such that

$S$1

has row and column sums equal to one. The Sinkhorn–Knopp algorithm efficiently computes the scaling factors, and the resulting normalization provably corrects biases induced by heteroskedastic noise in high-dimensional regimes, yielding robust spectral embeddings and clusterings, particularly in single-cell RNA-seq and related settings (Landa et al., 2020).

3.3 Dual-Dimension Normalization in Neural Networks

Batch Channel Normalization (BCN) is a neural variant that implements dual-dimension variance normalization across batch (sample) and channel axes in CNN or Transformer activation tensors $S$2:

• Per-channel (across batch/spatial): batch normalization along $S$3.
• Per-sample (across channel/spatial): layer normalization along $S$4. A per-channel learnable parameter mixes the two normalized paths, and an affine transform is applied. BCN adaptively leverages both axes' statistics and has demonstrated faster convergence, improved test accuracy, and batch-size robustness compared to BN and LN alone (Khaled et al., 2023).

3.4 Dual-dimension Normalization in Quantized Inference

KVarN (Variance-Normalized KV-Cache Quantization) introduces a dual-dimension normalization step in quantizing the keys and values of transformer KV caches. After an orthogonal (Hadamard) rotation, per-row and per-column variances are computed; scales are assigned along both axes so that the normalized block achieves unit variance along each. Simple round-to-nearest quantization is then applied. This approach is crucial for controlling token-wise magnitude errors that accumulate during long-horizon autoregressive decoding, resulting in improved performance over purely per-channel methods (Muller et al., 2 Jun 2026).

4. Theoretical Properties and Invariance

Dual-dimension variance normalization methods possess several critical invariance and structure-preserving properties:

• Symmetry: Rows and columns are treated on an equal footing, generalizing single-axis normalization.
• Ratio-scale invariance: All relative ratios within and between rows and columns are preserved. For any indices $S$5,

$S$6

ensuring that groupwise fold-changes are unaffected.

• Equivalence up to scale: All matrices obtained by positive diagonal scaling from a common normalized matrix (e.g., the output of the decomposition) fall into the same equivalence class, with the scale-invariant member unique up to a global scalar (Robinson, 2019).
• Robustness to noise artifacts: In high-dimensional settings with heteroskedastic noise, doubly-stochastic normalization corrects for pointwise variance-induced bias in affinity matrices, achieving convergence rate $S$7 in Frobenius norm, unlike one-sided normalization schemes (Landa et al., 2020).

5. Algorithmic Table: Dual-Dimension Normalization Methods

Methodology	Key Axes (& Dimension)	Normalization Mechanism
Projective Decomposition (Robinson, 2019)	Rows and columns ($S$8)	RMS scaling via iterative proportional scaling
Doubly-stochastic Kernel Normalization (Landa et al., 2020)	All point pairs ($S$9)	Sinkhorn scaling for row/col sums = 1
Batch Channel Normalization (BCN) (Khaled et al., 2023)	Batch-channel ($\alpha\in\mathbb{R}^m$0)	Adaptive weighted BN+LN mix
KVarN quantization (Muller et al., 2 Jun 2026)	Row-column (channel, token)	Inverse SD scales post-Hadamard rotation

Each approach applies the same conceptual strategy—enforcing prescribed scale or variance constraints along both axes—to settings ranging from classical data analysis to deep learning and inference.

6. Empirical Properties and Benchmarks

Significant empirical findings for dual-dimension normalization include:

• Robustness to heteroskedastic noise: Doubly-stochastic normalization yields order-of-magnitude smaller errors (Frobenius norm) in noisy affinity matrices compared to row-normalization, with high-dimensional rate $\alpha\in\mathbb{R}^m$1 (Landa et al., 2020).
• Neural networks: BCN attains higher accuracy and increased stability on image classification benchmarks (e.g., ResNet on CIFAR-100: 79.09% test accuracy with BCN vs. 74.50% for BN and 68.61% for LN) and remains robust for small batch sizes, where single-axis schemes degrade (Khaled et al., 2023).
• Quantization: KVarN enables 2-bit quantization with accuracy much closer to full-precision, e.g., for Qwen3-4B on AIME24, KVarN achieves 60.0% (FP16: 61.1%), while prior KIVI only attains 55.5% (Muller et al., 2 Jun 2026).

7. Applications and Significance Across Domains

Dual-dimension variance normalization is essential for:

• Normalizing ratio-scale data in genomics, mass spectrometry, and contingency tables to ensure invariance in statistical modeling and signal extraction (Robinson, 2019).
• Robust spectral clustering and manifold learning for single-cell RNA-seq and high-dimensional exploratory analysis, where heteroskedastic measurement noise is unavoidable (Landa et al., 2020).
• Neural network architectures (CNNs, Vision Transformers), where per-axis normalization yields superior convergence, generalization, and resilience to batch size variation (Khaled et al., 2023).
• Efficient quantized inference in LLMs, where dual-axis variance normalization suppresses the minority of large quantization errors that dominate long-horizon reasoning performance (Muller et al., 2 Jun 2026).

Dual-dimension variance normalization thus serves as a unifying concept with rigorous, domain-spanning theoretical justification and substantial practical benefits in noise robustness, statistical invariance, and computational efficiency.