Learned Equivalent Scaling (LES)

• Learned Equivalent Scaling (LES) is a design pattern that introduces minimal auxiliary parameters—such as diagonal scaling matrices, channel-wise vectors, or latent coefficient blocks—to preserve an equivalent computation while simplifying approximation tasks.
• LES modifies weight representations in tasks like LLM compression, post-training quantization, and depth scaling-up, by reshaping singular spectra or redistributing quantization error based on activation-aware losses.
• Empirical evaluations show that LES improves performance in mild-to-moderate regimes, though its effectiveness depends on calibration quality, optimal insertion locations, and trade-offs with extreme compression or specialized architectures.

Searching arXiv for the cited papers and related uses of “Learned Equivalent Scaling.” Learned Equivalent Scaling (LES) denotes, in the cited literature, a family of learned reparameterizations that preserve an equivalent computation while altering the representation on which approximation or initialization is performed. In "SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices" (Lavrinovics et al., 5 Jun 2026), LES is instantiated by diagonal row- and column-scaling matrices around a weight matrix before truncated SVD. In "DMQ: Dissecting Outliers of Diffusion Models for Post-Training Quantization" (Lee et al., 17 Jul 2025), LES is a positive, channel-wise scaling vector that rewrites a layer as $$(\mathbf{X}/\boldsymbol{\tau})(\boldsymbol{\tau}^\top\!\odot\,\mathbf{W})$$ before quantization. In the detailed exposition of "LESA: Learnable LLM Layer Scaling-Up" (Yang et al., 19 Feb 2025), the term is used more broadly for an SVD-based representation in which latent inter-layer coefficients are learned to initialize inserted layers. This suggests that LES is best understood as a methodological pattern rather than a single canonical algorithm.

1. Scope and common structure

Across the three formulations, LES introduces a learned auxiliary object that is low-dimensional relative to the original parameter tensor and is optimized so that a constrained surrogate better preserves the original model behavior. In SigmaScale, the learned objects are two diagonal matrices, $S_L=\mathrm{diag}(u)$ and $S_R=\mathrm{diag}(v)$, for a weight matrix $W\in\mathbb{R}^{m\times n}$; the method adds only $O(m+n)$ extra parameters per matrix. In DMQ, the learned object is a positive, channel-wise scaling vector $\boldsymbol{\tau}\in\mathbb{R}_+^{C_{\mathrm{in}}}$ for each layer. In LESA, the learned object is a small MLP $G_W$ that predicts intermediate SVD coefficient blocks for each matrix type.

Work	Learned transformation	Target problem
SigmaScale (Lavrinovics et al., 5 Jun 2026)	diagonal row- and column-scaling matrices $S_L,S_R$	truncated SVD-based LLM compression
DMQ (Lee et al., 17 Jul 2025)	positive, channel-wise scaling vector $\boldsymbol{\tau}$	post-training quantization of diffusion models
LESA (Yang et al., 19 Feb 2025)	predicted intermediate $V$-blocks in an SVD basis via a 3-layer feed-forward network	depth scaling-up

The shared structural idea is that the transformed representation is optimized under an application-specific loss rather than chosen by an analytic rule alone. In SigmaScale, the loss is activation-aware compression error on a calibration set. In DMQ, it is weighted mean squared reconstruction error after quantizing transformed activations and weights. In LESA, it is reconstruction loss on latent SVD blocks, augmented by a norm-matching term. A common implication is that LES shifts optimization effort from the full weight tensor to a smaller set of learned scaling or coefficient parameters, while keeping the original model weights fixed or nearly fixed during the auxiliary optimization stage.

2. LES in SVD-based low-rank LLM compression

SigmaScale formulates LES for a weight matrix $S_L=\mathrm{diag}(u)$0 by introducing two diagonal “equivalent-scaling” matrices,

$S_L=\mathrm{diag}(u)$1

and forming

$S_L=\mathrm{diag}(u)$2

A rank-$S_L=\mathrm{diag}(u)$3 truncated SVD is then computed on $S_L=\mathrm{diag}(u)$4,

$S_L=\mathrm{diag}(u)$5

with reconstruction loss

$S_L=\mathrm{diag}(u)$6

The method does not merely minimize $S_L=\mathrm{diag}(u)$7 over $S_L=\mathrm{diag}(u)$8. Instead, it uses an activation-aware loss on a calibration set $S_L=\mathrm{diag}(u)$9,

$S_R=\mathrm{diag}(v)$0

or equivalently

$S_R=\mathrm{diag}(v)$1

A small regularizer is added:

$S_R=\mathrm{diag}(v)$2

The training-time procedure learns logarithmic diagonal parameters $S_R=\mathrm{diag}(v)$3, with initialization

$S_R=\mathrm{diag}(v)$4

and exponentiated scales

$S_R=\mathrm{diag}(v)$5

At each step, the method computes $S_R=\mathrm{diag}(v)$6, performs $S_R=\mathrm{diag}(v)$7, reconstructs $S_R=\mathrm{diag}(v)$8, rescales back as $S_R=\mathrm{diag}(v)$9, evaluates

$W\in\mathbb{R}^{m\times n}$0

and updates $W\in\mathbb{R}^{m\times n}$1 with Adam. Gradients are back-propagated through the SVD using either Taylor-series-based approximations or by skipping unstable denominators. In practice, SigmaScale uses Adam with $W\in\mathbb{R}^{m\times n}$2, $W\in\mathbb{R}^{m\times n}$3, and a cosine scheduler with 10% warm-up. At deployment, it constructs low-rank factors

$W\in\mathbb{R}^{m\times n}$4

so that $W\in\mathbb{R}^{m\times n}$5 and $W\in\mathbb{R}^{m\times n}$6 (Lavrinovics et al., 5 Jun 2026).

A central diagnostic is effective-rank entropy,

$W\in\mathbb{R}^{m\times n}$7

A high $W\in\mathbb{R}^{m\times n}$8 indicates that the singular-value spectrum is flat. Learned $W\in\mathbb{R}^{m\times n}$9 reshape $O(m+n)$0 so that the top $O(m+n)$1 singular values capture more mass, lowering $O(m+n)$2 on the scaled matrix. Empirically, across Llama 3.1-8B modules, the percentage drop in $O(m+n)$3 correlates approximately $O(m+n)$4 with the percentage drop in $O(m+n)$5.

The empirical evaluation covers Llama 3.1 8B-Instruct and Qwen3-8B. On Wikitext-2 perplexity at mild compression, defined as 90% retention, the reported Llama 3.1 values are baseline $O(m+n)$6, SVD-LLM $O(m+n)$7, ASVD+ $O(m+n)$8, and SigmaScale $O(m+n)$9; with KD, SigmaScale reaches $\boldsymbol{\tau}\in\mathbb{R}_+^{C_{\mathrm{in}}}$0. On five zero-shot QA benchmarks, SigmaScale matches or exceeds ASVD+ on most tasks and nearly closes the gap to the uncompressed model. At 75% retention, perplexities and accuracies remain competitive versus SVD-LLM and ASVD+, with SigmaScale sometimes superior on reasoning tasks. At 50% retention, all methods degrade sharply, and SigmaScale degrades more, indicating that learned scaling is most beneficial when $\boldsymbol{\tau}\in\mathbb{R}_+^{C_{\mathrm{in}}}$1 still captures the dominant subspace.

3. LES in post-training quantization of diffusion models

DMQ defines LES for a single linear or convolutional layer

$\boldsymbol{\tau}\in\mathbb{R}_+^{C_{\mathrm{in}}}$2

It introduces a positive, channel-wise scaling vector $\boldsymbol{\tau}\in\mathbb{R}_+^{C_{\mathrm{in}}}$3 and rewrites the layer as

$\boldsymbol{\tau}\in\mathbb{R}_+^{C_{\mathrm{in}}}$4

Both $\boldsymbol{\tau}\in\mathbb{R}_+^{C_{\mathrm{in}}}$5 and $\boldsymbol{\tau}\in\mathbb{R}_+^{C_{\mathrm{in}}}$6 are quantized with a standard Min-Max uniform quantizer $\boldsymbol{\tau}\in\mathbb{R}_+^{C_{\mathrm{in}}}$7, and the per-sample reconstruction error is

$\boldsymbol{\tau}\in\mathbb{R}_+^{C_{\mathrm{in}}}$8

Over a calibration batch $\boldsymbol{\tau}\in\mathbb{R}_+^{C_{\mathrm{in}}}$9, the weighted loss is

$G_W$0

The error-redistribution argument is explicit. Before LES,

$G_W$1

and the original squared error decomposes as

$G_W$2

After LES, the transformed errors are

$G_W$3

and the new error becomes

$G_W$4

The weight error term is therefore multiplied by $G_W$5, while the activation error is divided by $G_W$6. By choosing $G_W$7 to minimize $G_W$8, the method shifts quantization difficulty away from channels that originally contributed most to total error.

DMQ further introduces adaptive timestep weighting because activations depend on diffusion timestep $G_W$9 and quantization errors accumulate across the denoising schedule. The running average for each timestep is

$S_L,S_R$0

and the focal-style weight is

$S_L,S_R$1

This up-weights steps whose accumulated error is currently low, often early denoising steps. The optimization learns $S_L,S_R$2 per layer with fixed $S_L,S_R$3, no extra regularizer beyond enforcing $S_L,S_R$4, and clipping $S_L,S_R$5 to $S_L,S_R$6. At convergence, $S_L,S_R$7 is fused into quantization scales, leaving no runtime overhead beyond the usual quant forward (Lee et al., 17 Jul 2025).

LES in DMQ is complemented by channel-wise Power-of-Two Scaling (PTS) for activation outliers, especially in skip-connection layers. After LES fusion,

$S_L,S_R$8

where $S_L,S_R$9. Integer-quantized activation scaling can then be implemented as a left bit-shift on the weight side:

$\boldsymbol{\tau}$0

The reported numerical effects are specific. On FFHQ $\boldsymbol{\tau}$1 at W4A8, the baseline TFMQ-DM gives FID $\boldsymbol{\tau}$2, sFID $\boldsymbol{\tau}$3, and weight-quant error $\boldsymbol{\tau}$4; LES only reduces weight-quant error to $\boldsymbol{\tau}$5 and yields FID $\boldsymbol{\tau}$6, sFID $\boldsymbol{\tau}$7; SmoothQuant, for comparison, blows up FID to $\boldsymbol{\tau}$8. In the FFHQ W4A8 ablation, baseline FID $\boldsymbol{\tau}$9 improves to $V$0 with LES, to $V$1 with adaptive weights, and to $V$2 with PTS. On LSUN-Church W4A6, the full DMQ reports FID $V$3 versus prior best approximately $V$4. On ImageNet class-cond W4A6, the method reports FID $V$5 versus $V$6 for TFMQ-DM.

4. LES as an SVD-basis method for depth scaling-up

LESA addresses a different problem: model depth scaling-up for LLMs. Let a pre-trained Transformer have $V$7 layers, and let

$V$8

denote the weight matrices of a particular subcomponent. The goal is to construct a deeper model with $V$9 layers by inserting $S_L=\mathrm{diag}(u)$00 new layers so that the deeper model remains close to the original on downstream tasks and the extra layers are well-initialized for fast continued pre-training. The layer sequence is written as

$S_L=\mathrm{diag}(u)$01

and the deeper model as

$S_L=\mathrm{diag}(u)$02

where the new $S_L=\mathrm{diag}(u)$03 is predicted.

The SVD construction begins by horizontally concatenating the matrices of a fixed type,

$S_L=\mathrm{diag}(u)$04

followed by

$S_L=\mathrm{diag}(u)$05

Partition $S_L=\mathrm{diag}(u)$06 into blocks of width $S_L=\mathrm{diag}(u)$07:

$S_L=\mathrm{diag}(u)$08

with each $S_L=\mathrm{diag}(u)$09. Then

$S_L=\mathrm{diag}(u)$10

To insert an intermediate layer between layers $S_L=\mathrm{diag}(u)$11 and $S_L=\mathrm{diag}(u)$12, LESA predicts $S_L=\mathrm{diag}(u)$13 and reconstructs

$S_L=\mathrm{diag}(u)$14

For each matrix type—q_proj, k_proj, v_proj, o_proj, up_proj, down_proj, gate_proj—LESA trains a small MLP $S_L=\mathrm{diag}(u)$15 to predict the missing $S_L=\mathrm{diag}(u)$16 from its two neighbors. The input is the concatenation $S_L=\mathrm{diag}(u)$17, the architecture is a 3-layer feed-forward network with ReLU activations and hidden dimension 256, and the output is $S_L=\mathrm{diag}(u)$18. Training uses two losses,

$S_L=\mathrm{diag}(u)$19

and

$S_L=\mathrm{diag}(u)$20

The combined objective is

$S_L=\mathrm{diag}(u)$21

Optimization uses AdamW with LR $S_L=\mathrm{diag}(u)$22 for 5 epochs on the 30 triplets extracted from a 32-layer model. When expanding Llama3-8B from 32 layers to 48 layers, one predicted layer is inserted between each original $S_L=\mathrm{diag}(u)$23 and $S_L=\mathrm{diag}(u)$24 for $S_L=\mathrm{diag}(u)$25, yielding 16 new layers. During continual pre-training, all original layers are frozen and only the newly inserted ones are updated, as in LLaMA Pro; for SFT, all parameters are unfrozen (Yang et al., 19 Feb 2025).

The experimental setup specifies Llama3-8B as the base model, expanded to 48 layers and 11.5 B parameters, with LLaMA Pro and SOLAR as baselines. Datasets are Wikipedia (Nov 2024) for continual pre-training, Alpaca-GPT4 instructions for SFT, and a subset of BigCode for code pre-training ablation. Benchmarks include OpenCompass zero/few-shot on Reasoning, Language, Knowledge, Examination, and Understanding; held-out Wikipedia PPL; ARC, TruthfulQA, GSM8K, HellaSwag, and MMLU for instruction SFT; and HumanEval and MBPP for code. Hardware is 8×A100 80 GB GPUs.

The quantitative results are direct. Continual pre-training time over the full 6 k steps is 56.4 h for LLaMA Pro, 75.6 h for SOLAR, and 45.6 h for LESA. LESA begins at a lower initial loss and stabilizes after approximately 2 k steps; LLaMA Pro reaches comparable loss only at approximately 5 k; SOLAR fails to converge within 6 k steps. Selected task-performance averages after pre-training are 62.67 for LLaMA Pro @6 k, 47.86 for SOLAR @6 k, and 64.30 for LESA @6 k. Instruction-tuned SFT averages are 24.38 for Pro-SFT, 26.47 for SOLAR-SFT, and 31.57 for LESA-SFT. On code benchmarks, pass@1 is 10.98, 2.44, and 25.00 on HumanEval, and 21.69, 13.93, and 28.60 on MBPP, for Pro, SOLAR, and LESA respectively.

The ablations clarify the role of the learned latent parameterization. Without SVD, training $S_L=\mathrm{diag}(u)$26 directly on $S_L=\mathrm{diag}(u)$27 matrices still outperforms LLaMA Pro, but the initial loss is approximately 0.03 higher and task scores drop by up to 1 point. Without the norm-loss $S_L=\mathrm{diag}(u)$28, the predicted blocks collapse, with norms dropping from approximately 80 to approximately 10 and perplexity exploding. If original layers are not frozen, training curves fluctuate and convergence slows markedly. Insertion location also matters: expanding layers 1–17 yields initial PPL approximately 57.3, whereas inserting at layers 15–31 yields 6.35. On multiple model families, LESA’s initialization PPL consistently beats SOLAR, including 6.35 versus 7.81 on Llama3-8B and 1.98 versus 4.21 on Llama3-70B. On LLaMA-MoE-3.0B, however, LESA with copied router gives PPL approximately 1923 and SOLAR gives approximately 76.5; both perform poorly, suggesting that scaling MoE routers needs separate study.

5. Diagnostics, benefits, and recurring design principles

The three formulations expose a set of recurring technical principles. First, the learned transformation is activation-aware or task-aware rather than solely weight-space optimal. SigmaScale explicitly optimizes an activation-aware compression loss instead of only the Frobenius reconstruction loss, and it attributes part of the gain to the fact that weight matrices in LLMs often have “activation outliers” that distort their singular-value spectrum. DMQ similarly learns $S_L=\mathrm{diag}(u)$29 from calibration activations and diffusion timesteps rather than from weight statistics alone. LESA learns inter-layer continuity from adjacent latent coefficient blocks rather than inserting layers by heuristic duplication.

Second, LES often acts by reshaping a spectrum or redistributing error. In SigmaScale, learned $S_L=\mathrm{diag}(u)$30 lower the effective intrinsic rank of weight matrices, reflected by reductions in effective-rank entropy, and this reduction is strongly correlated with compression loss. In DMQ, $S_L=\mathrm{diag}(u)$31 explicitly redistributes quantization difficulty between weights and activations. In LESA, the SVD basis exposes latent patterns between layers, suggesting that inter-layer parameters can be learned. These are different mechanisms, but each changes the representation on which the approximation problem is solved.

Third, the empirical gains are regime-dependent. SigmaScale reports that most gains appear in the mild-to-moderate rank-retention regime, while extreme low ranks remain a challenge. DMQ reports especially strong improvements at low bit-widths such as W4A6 and W4A8, but still requires an additional mechanism, PTS, for layers with high inter-channel variance. LESA shows substantial gains for continual pre-training and code transfer, yet performs poorly on the MoE setting described above. This suggests that LES is not a universal substitute for architecture-specific handling of outliers, routing, or extreme compression regimes.

A common misconception is that LES must refer to a single diagonal scaling trick. The cited literature does not support that restriction. SigmaScale and DMQ both learn explicit scaling factors, but their objectives, domains, and invariances differ. LESA uses the label in a broader SVD-basis setting where the learned object is not a diagonal scale but a predicted latent coefficient block. A plausible implication is that the phrase currently denotes a broader design pattern: learn a compact auxiliary parameterization that preserves an equivalent computation while making a downstream approximation problem easier.

6. Computational trade-offs and unresolved directions

LES methods exchange a relatively small number of learned auxiliary parameters for additional optimization machinery. SigmaScale adds only $S_L=\mathrm{diag}(u)$32 extra parameters per matrix, which is negligible relative to $S_L=\mathrm{diag}(u)$33, but training requires repeated SVDs, with cost stated as $S_L=\mathrm{diag}(u)$34 per step, and careful handling of SVD gradients. DMQ learns only a positive vector per layer and fuses the result into static quantization scales, leaving no runtime overhead beyond the usual quant forward; nevertheless, it depends on calibration data, timestep-aware loss weighting, and explicit clipping of $S_L=\mathrm{diag}(u)$35 to maintain stability. LESA uses a small MLP and a short auxiliary training stage, but its success depends on preserving the original layers during continual pre-training and on selecting favorable insertion locations.

The limitations reported by the three works are specific rather than purely conceptual. In SigmaScale, all methods degrade sharply at 50% retention, and SigmaScale degrades more than its comparators in that aggressive regime. In DMQ, LES alone does not eliminate extreme outliers, especially in skip-connection layers, motivating the additional channel-wise Power-of-Two Scaling and a voting algorithm for robust factor selection with a small calibration set. In LESA, removing the norm-loss leads to collapse, removing SVD weakens initialization, and applying the method to the MoE configuration considered yields very poor perplexity.

Taken together, these results position LES as a technically narrow but flexible modeling strategy. In low-rank compression, it reshapes singular spectra before truncation; in diffusion PTQ, it redistributes quantization difficulty across channels and timesteps; in depth scaling-up, it learns a latent coefficient trajectory from which new layers can be reconstructed. The available evidence supports LES as a useful mechanism when the approximation target is constrained—rank-limited, bit-limited, or architecture-limited—but also indicates that its effectiveness depends on calibration quality, the structure of outliers, and the extent to which the constrained surrogate still captures the dominant subspace or representation trajectory.