Deep Weight Factorization

• Deep Weight Factorization (DWF) is a family of techniques that decomposes neural network weights into multiple factors, enabling improvements in quantization, regularization, and sparsity.
• It employs layer-wise scaling and parametric factorizations to mitigate quantization errors, achieve high sparsity (up to 90% on benchmarks), and enable efficient multilingual adaptation.
• DWF integrates customized initialization and optimization strategies to balance factors during training, reducing accuracy loss in quantized and pruned networks.

Deep Weight Factorization (DWF) refers broadly to a family of techniques that decompose the parameter matrices (or tensors) of deep neural networks into multiple interacting factors, with objectives ranging from quantization robustness and regularization to sparsity and parameter efficiency. DWF exploits the inherent symmetries and redundancy in overparameterized neural architectures by factorizing weights, either analytically for transformation invariance or parametrically for training-time regularization, compression, or adaptation.

1. Functional Weight Factorization for Quantization Robustness

The conceptually simplest form of DWF targets quantization error mitigation by leveraging the degree of freedom in the scaling of intermediate representations when the nonlinearities between layers are positively homogeneous (e.g., ReLU, PReLU, or linear activations) (Meller et al., 2019). For any two consecutive linear layers $L: x \mapsto W^{(l)}x$ and $L': y \mapsto W^{(l+1)}y$, and any positive diagonal matrix $D = \operatorname{diag}(s_1, ..., s_{C_l})$,

$$L'(A(L(x))) = W^{(l+1)}A(W^{(l)}x) = (W^{(l+1)}D^{-1})A(D W^{(l)}x)$$

for all $x$ and positively homogeneous nonlinearities $A(\cdot)$. This property enables scaling each output channel of a layer and inversely scaling the corresponding input channels of the next layer, preserving the overall network function. DWF exploits this for pre-quantization equalization: by finding scaling vectors $s$ that minimize the combined quantization error of two consecutive layers,

$$E(s) = \left\| Q(W'^{(l)}) - W'^{(l)} \right\|_F^2 + \left\| Q(W'^{(l+1)}) - W'^{(l+1)} \right\|_F^2$$

where $W'^{(l)} = \operatorname{diag}(s) W^{(l)}$ and $W'^{(l+1)} = W^{(l+1)} \operatorname{diag}(s^{-1})$, and $Q(\cdot)$ is a uniform quantizer with per-tensor dynamic range adaptation.

Greedy, data-free algorithms—such as one-step and two-step equalization—greedily rescale channels to minimize quantization-induced signal degradation, operating in $O(\sum_\ell |W^{(\ell)}|)$ time. This approach significantly reduces accuracy loss under low-precision quantization, with empirical results demonstrating degradation reductions of up to 90% in challenging networks such as MobileNet-V2 (e.g., quantization gap reduced from 42.7 pp to 0.6 pp) (Meller et al., 2019).

2. Parametric Factorizations for Regularization and Sparsity

DWF can target explicit regularization by learning decompositions of each weight matrix $W$ into a product of $k$ factors, $W = A_kA_{k-1}\cdots A_1$, where each $A_i$ is a learnable parameter matrix (Kolb et al., 4 Feb 2025). This factorization is combined with an independent $L_2$ penalty on each factor:

$$\mathcal{L}(A_1,\dots,A_k) = \mathcal{L}_{\mathrm{task}}(A_k\cdots A_1) + \lambda \sum_{i=1}^k \|A_i\|_F^2$$

Through a variational identity, for fixed $W$, the minimal sum-of-squares over all $k$-factorizations yields an implicit $L_{2/k}$ quasi-norm on $W$:

$$\min_{A_k\cdots A_1 = W} \sum_{i=1}^k \|A_i\|_F^2 = k \|W\|_{2/k}^{2/k}$$

For $k=2$, this is $\ell_1$ (promoting sparsity); for $k>2$, the quasi-norm is nonconvex and more strongly sparse-promoting. Empirical results confirm that, under this regime, high sparsity (e.g., 90% on ResNet-18/CIFAR-10 at near-baseline accuracy) is achievable in a single training run, outperforming many pruning pipelines (Kolb et al., 4 Feb 2025).

3. Application-Specific Weight Factorization Schemes

DWF has been specialized for multilingual adaptation in large speech recognition models (Pham et al., 2021). Here, each weight matrix $W$ is decomposed into a shared global component $W^{(0)}$ and a language-specific correction $\Delta W^{(\ell)}$:

$W = W^{(0)} + \Delta W^{(\ell)}$

$\Delta W^{(\ell)} = u^{(\ell)}[v^{(\ell)}]^T$

with $u^{(\ell)} \in \mathbb{R}^{d_{\mathrm{out}}}$ and $v^{(\ell)} \in \mathbb{R}^{d_{\mathrm{in}}}$, imposing a rank-1 structure on the adaptation. Alternatively, multiplicative and additive rank-1 gating masks can be used. This reduces per-language adaptation overhead from $O(d_{\mathrm{in}}\cdot d_{\mathrm{out}})$ to $O(d_{\mathrm{in}}+d_{\mathrm{out}})$, enabling scaling to dozens of languages with only 10% training overhead and negligible inference cost. Empirically, DWF yields 15–27% relative word error rate improvements over strong multilingual baselines (Pham et al., 2021).

4. Optimization and Training Dynamics

The factorization structure introduces significant optimization considerations. Replacing $W$ by a product of $k$ factors increases the effective Lipschitz constant of the parameter gradient by a factor $\propto k$, reducing the allowed learning rate (empirically, $\eta_{\max} \propto 1/k$). Initialization schemes must be adjusted so that each entry is drawn from a Gaussian with variance $\sigma_W^{2/k}$, preventing activation explosion or vanishing.

Three phases emerge throughout training: initial rapid fit with unbalanced factors; factor alignment to a balanced magnitude regime; and late-stage sparsification as the implicit quasi-norm penalty dominates, driving many $|w|$ to zero (Kolb et al., 4 Feb 2025).

For functional factorization approaches (as in quantization robustness), the optimization seeks scaling vectors through greedy, layer-wise methods—using small calibration datasets to estimate channel extremes. The two-step equalization scheme further attenuates channels that are oversized in their respective successor layers, empirically closing 90% of the gap to optimal equalization (Meller et al., 2019).

5. Extensions and Relations to Regularization Frameworks

DWF subsumes and generalizes other regularization methods based on weight or activation perturbations. The FaMe (Factored Mean) model (Rudy et al., 2014) factorizes each layer’s weight matrix as $W=VU$ and injects noise into the intermediate factor, thus regularizing by simulating an ensemble of low-rank models. The test-time weights become $W=VU$, which are robustified by training-time noise. FaMe achieves strong results on MNIST ($0.91\%$ error) and is competitive on CIFAR-10/100.

In the context of pruning, inversely-proportional factorizations (as in quantization-robust DWF) serve as a pre-processing step: by equalizing the dynamic range of channels, the effect of thresholding small weights becomes more predictable, enhancing the stability of magnitude-based pruning.

Further, balancing per-channel ranges facilitates semantic interpretability across hidden representations, as scales are normalized, making activations and weights comparable across channels (Meller et al., 2019).

6. Limitations and Open Directions

Known limitations include increased hyperparameter sensitivity (factor depth $k$, regularization weight $\lambda$, learning rate $\eta$), and training/memory overheads that, while modest ($<1.5\times$ for $k\leq4$), are nonnegligible for deeper factorizations (Kolb et al., 4 Feb 2025). For quantization-focused DWF, requirements include homogeneous activations and small calibration sets for extrema estimation. Greedy layerwise algorithms are not globally optimal; global, jointly optimal solvers remain an open problem.

Potential advances include block-wise structured sparsity, dynamic adaptation of factor-depth per layer, integration with quantization-aware training, and the development of global solvers subject to cross-layer constraints. For multilingual adaptation, further compression via mixed rank strategies and exploration beyond rank-1 factors remain prospects (Pham et al., 2021).

7. Empirical Benchmarks

Empirical results across DWF variants demonstrate strong improvements over baselines. For quantization, 8-bit quantization on ImageNet with DWF reduces accuracy degradation by over an order of magnitude in DenseNet-121 (5.78 pp ↓ 0.35 pp). For sparsity, DWF achieves 90% parameter sparsity with <0.3% accuracy drop on standard image classification benchmarks, and for multilingual speech recognition, relative WER reductions of 15–27% are achieved with minimal parameter cost increase (Kolb et al., 4 Feb 2025, Pham et al., 2021, Meller et al., 2019).

Application Domain	Core DWF Mechanism	Key Empirical Gains
Quantization robustness	Channelwise functional scaling	0.6 pp quantization gap (MobileNet-V2)
Sparse regularization	Implicit $\ell_{2/k}$ penalty	90% sparsity, <1% accuracy drop (ResNet-18)
Multilingual adaptation	Shared plus rank-1 correction	27% WER reduction (27 lang. ASR)

These results substantiate DWF as a unifying framework for leveraging artificial symmetries in neural network parameterizations for quantization, regularization, sparsity, and efficient adaptation.