Scale-Invariant Neural Networks

• Scale-Invariant Neural Networks are architectures that maintain consistent outputs despite rescaling of inputs, features, or parameters.
• They employ strategies such as deep+wide decomposition, multi-column designs, SI-Conv layers, and scale-steerable filters to achieve invariance in diverse applications.
• Robust training methods including spherical optimization and weight decay improve generalization and simplify hyperparameter tuning while preserving scale invariance.

A scale-invariant neural network is a neural architecture for which the network output or specific learned representations (e.g., logits, feature maps, rank scores) remain invariant under input, feature, or parameter rescalings. Such invariance is relevant in diverse domains including vision, learning-to-rank, time series, and large-model optimization, and can be engineered through architectural design, training procedures, or explicit invariance-enforcing layers.

1. Mathematical Foundations and Taxonomy

Scale invariance in neural networks manifests in several distinct but related forms, depending on whether it is defined over inputs, features, parameters, or outputs.

• Input-scale invariance: $f(cx) = f(x)$ for all $c > 0$ and input $x$.
• Feature-scale invariance: Model output is unchanged when specific features are rescaled by a positive scalar; formally, score or probability differences are preserved under $x^{S} \to c x^{S}$ for selected features.
• Parameter-scale invariance: For weights $w$, $N(\lambda w, x) = N(w, x)$ for all $\lambda>0$, i.e., the output does not depend on overall parameter norm.
• Local vs. global scale invariance: "Local" refers to invariance to transformations within spatial neighborhoods (e.g., image patches), "global" to uniform rescaling of entire features or parameter groups.

Core mathematical structures include 0-homogeneous functions (output invariant to parameter scaling), projection onto spheres for parameters (optimization on $S^{n-1}$), and explicit algebraic cancellation of scale effects via transformations such as $\log(x)$ or normalization.

2. Scale-Invariant Neural Network Architectures

Multiple architectural strategies can guarantee or promote scale invariance:

A. Deep + Wide Decomposition for Feature-scale Invariant LTR

In learning-to-rank (LTR) settings, Sommeregger et al. propose a summation of a deep neural path operating on scale-stable features with a wide path operating on potentially arbitrarily rescaled features using a log transformation and Kronecker-product interaction: $$f(\mathbf{x}_{ij}) = f_d(\mathbf{x}^Q_i,\,\mathbf{x}^{I,F}_{ij}) + f_w(\mathbf{x}^Q_i,\,\mathbf{x}^{I,S}_{ij})$$ where

$$f_w(\mathbf{x}_i^Q, \mathbf{x}_{ij}^{I,S}) = \langle \mathbf{w},\, \mathbf{f}_s(\mathbf{x}_i^Q) \otimes_{\mathrm{kron}} \log(\mathbf{x}_{ij}^{I,S}) \rangle$$

The log transform converts multiplicative feature rescaling into additive shifts, which are canceled out when rank-differences are formed, rendering the scores—and hence predicted orderings—invariant to rescaling of $\mathbf{x}^{I,S}$ (Petrozziello et al., 2 Oct 2024).

B. Multi-Column Architectures with Scale-Tied Filters

SiCNN and related models employ parallel columns, each specialized to a particular input scale but all sharing a canonical set of filters via algebraic transformation (e.g., bicubic interpolation, pseudoinverse). For input $I$ and canonical filter $f_c$: $f_s = T_S(f_c)$ with $T_S$ the linear filter rescaling operator. All column outputs are concatenated, and the classifier operates on the fused representation, yielding robust scale-invariant features (Xu et al., 2014).

C. Locally Scale-Invariant Convolutional Layers

SI-ConvNets implement each convolution layer as a max-pool over convolution outputs from a set of rescaled input versions, sharing the same weights: \begin{align*} \hat z_i & = W * \mathcal{T}i(x) + b\ z_i & = \mathcal{T}_i^{-1}(\hat z_i)\ h & = \sigma \left( \max{1 \leq i \leq n} z_i \right) \end{align*} This produces true local scale-invariance without increasing the parameter count; pool-over-scales leads to robustness against unseen input scale variations (Kanazawa et al., 2014).

D. Scale-Steerable Filters

Scale-steerable convolutional layers decompose spatial filters into a log-radial harmonic basis in polar coordinates. Proper basis expansion allows filters to be "steered" to any scale analytically, yielding variationally optimal responses to scaled patterns and efficient max-pooling over scale-transformed versions (Ghosh et al., 2019).

E. Riesz Networks

Riesz networks replace spatial convolutions with the Riesz transform, a nonlocal linear operator that is provably scale-equivariant: $$\mathcal R_j f(x) = C_d \lim_{\varepsilon \to 0} \int_{|y|>\varepsilon} \frac{y_j}{|y|^{d+1}} f(x-y) dy,$$ Satisfying $\mathcal R_j(f(x/a)) = (\mathcal R_j f)(x/a)$ for all $a>0$ (Barisin et al., 2023).

3. Training and Optimization Under Scale Invariance

Optimization dynamics in scale-invariant spaces exhibit unique properties:

• Homogeneous parameter-space and path-invariant SGD: For ReLU networks, positive node-wise rescaling leaves function unchanged; optimization in the quotient space (e.g., $\mathcal{G}$-SGD) eliminates flat scaling directions, accelerates convergence, and improves generalization by focusing on path-value coordinates rather than raw weights (Meng et al., 2018).
• Spherical optimization and effective learning rates: With norm-invariant parameterizations (e.g., post-normalization in each layer), gradient descent operates on the sphere $S^{n-1}$. The dynamics depend on the effective learning rate $\tilde\eta = \eta/\|\theta\|^2$, giving rise to distinct training regimes (convergence, equilibrium, divergence) depending on ELR value and leading to clearer loss landscape analysis (Kodryan et al., 2022).
• Robust scale-invariant training with SGD + weight decay: For architectures designed to be 0-homogeneous in their parameter groups, coupling SGD with weight decay ensures the parameter norm stabilizes, eliminating vanishing effective learning rates and limiting sensitivity to initialization or loss scaling. Global gradient norm clipping further adds stability (Li et al., 2022).

4. Scale-Invariance in Statistical Learning and Generalization

Scale invariance affects not only optimization, but also generalization, uncertainty, and Bayesian inference:

• Flatness–generalization paradox resolution: Classical flatness metrics (e.g., Hessian trace) can be arbitrarily rescaled under function-preserving parameter transformations (e.g., in BatchNorm’d nets), rendering them ambiguous. Decomposition into scale and connectivity coordinates yields invariance to rescaling in both the connectivity tangent kernel and the data-dependent PAC-Bayes generalization bound (Kim et al., 2022).
• Scale-invariant Laplace approximations: Standard Laplace posteriors become overconfident under scale reparameterizations; the scale-invariant posterior by reparametrization of uncertainty to relative connectivity directions produces variance estimates that remain well-calibrated under rescalings, particularly in models with batch norm and weight decay.
• Scale-statistics–aware normalization for fast convergence: Applying both per-sample normalization (ensuring feature scale invariance) and per-batch whitening (basis invariance) to layer inputs provides rapid, robust training, even for large LR and without learning rate warmup (Ye et al., 2021).

5. Applications Across Domains

Scale-invariant neural networks have been developed and evaluated in numerous application scenarios:

Application Area	Scale-Invariance Strategy	Notable Empirical Results
Learning-to-rank (LTR)	Deep+Wide split, log-Kron	NDCG@k stays invariant under feature scale perturbations (Petrozziello et al., 2 Oct 2024)
Image classification	Multi-column, SI-Conv, Steerable	Ensemble of 4-scale CNN: MCA=82.1%, single-scale worst=62.0% (Noord et al., 2016)
Semantic segmentation	Riesz transforms	Dice >94% on unseen scales, 1-pass (Barisin et al., 2023)
Inverse problems (compressed sensing)	Bias-free homogeneous ReLU nets	2-layer depth suffices for stable, scale-invariant recovery (Bamberger et al., 2023)
Robotics/sensor data	Feature-scale invariance (log)	Robust to reporting units and sensor scale in regression/classification (Petrozziello et al., 2 Oct 2024, Ye et al., 2021)
Time-series	SITHCon (log-space pooling, max)	Generalization over >100x input time-scale variability (Jacques et al., 2021)
LLMs	SIBERT (SI-transformer)	SGD+WD approaches Adam in BERT performance (Li et al., 2022)

Empirical evidence consistently shows that architectures explicitly designed for scale-invariance provide stable performance under drastic input, feature, or parameter rescalings without expensive data augmentation, retraining, or normalization at inference.

6. Theoretical and Structural Significance

• Bridging artificial and biological representations: Dimensional stability and structural self-similarity across scales in deep embeddings strongly predict alignment with fMRI data from human visual cortex. AI models with maximally scale-invariant representations converge on geometric structures more "brain-like" than those lacking such stability, especially when pretrained with large, multimodal datasets (Yu et al., 13 Jun 2025).
• Intrinsic explainability via fractal diagnostics: Scale-invariant diagnostics such as fractal dimension, roughness, and scale-invariant spectral gaps provide intrinsic, architecture-agnostic explainability of DNN dynamics. These measures remain stable under feature, parameter, or architectural rescalings and provide phase-space and graph-based signatures of network convergence and modularity (Moharil et al., 12 Jul 2024).
• Thermodynamic analogy: Training of scale-invariant networks under SGD+weight decay is mathematically analogous to an ideal gas, with explicit mappings from learning rate, weight decay, and noise to temperature, pressure, and volume. This leads to “equations of state” for the parameter distribution, facilitating principled hyperparameter tuning (Sadrtdinov et al., 10 Nov 2025).

7. Limitations and Open Directions

• Computational cost: True local scale invariance (e.g., SI-ConvNets, scale-steerable filters) often increases computational and memory burden due to explicit multi-scale convolutions or large basis expansions.
• Loss of scale-ordering information: Max-pooling over scales (SI-Conv, SITHCon) discards ordinal scale information, which may be necessary for some tasks (e.g., 3D structure).
• Restriction to pre-defined transforms: Most current methods handle only affine/positive scaling; extension to fully learnable or non-homogeneous transforms remains an active area.
• Sensitivity outside designed scale range: While architectures like Riesz networks offer continuous scale equivariance, their performance can degrade for pathological edge cases (e.g., 1px cracks), or when structural parameters (basis, receptive fields) are poorly tuned for the task.

Potential future directions involve combining scale invariance with rotation/affine group equivariances, continuous parameterizations for arbitrary dilation groups, learnable multi-scale pooling layers, and the integration of spectral/structural invariance as auxiliary losses or constraints for unsupervised and self-supervised learning.