Learnable Residual Scaling in Deep Networks

• Learnable residual scaling is a framework that integrates trainable multiplicative factors into residual branches to modulate signal flow and ensure stable gradient propagation.
• It has been applied in neural compression, PINN-based PDE solvers, deep transformers, and normalization-free architectures, using scalars, vectors, or block-wise gates.
• The technique enables dynamic adaptation and implicit regularization, preventing vanishing/exploding signals while enhancing model capacity and overall network performance.

Learnable residual scaling is a general framework for modulating the contribution of residual branches in deep architectures through explicit, often trainable, multiplicative factors. This mechanism addresses structural limitations inherent to residual networks—including vanishing or exploding signal magnitude, early-stage feature domination, and unstable training at scale—by adaptively controlling the amplitude of residual updates at each layer or stage. Learnable scaling has emerged repeatedly in diverse contexts, such as neural compression, PINN-based PDE solvers, deep transformers, and deep normalization-free architectures. Its implementations range from simple per-stage scalars to vector- or block-wise gates, and its theoretical underpinning is closely tied to the capacity control and implicit regularization provided by scaling the exponential ensemble of residual paths.

1. Mathematical Formulation and Variants

The core structural form of learnable residual scaling is the insertion of one or more trainable scalars, vectors, or matrices that modulate each residual block’s output before it is added back to the main stream. In the most generic $L$-layer residual network, the scaled update reads: $x_{l+1} = x_l + \alpha_l F_l(x_l)$ where $\alpha_l$ is a scalar (learnable or fixed), and $F_l$ is the non-linear transform at layer $l$. This element extends to several domains:

• Neural Compression (RFSQ): For multi-stage residual finite scalar quantization, each stage $k$ uses a learnable scale $s_k$:

$$\tilde{r}_k = s_k \cdot r_{k-1}, \quad q_k = \mathrm{FSQ}_k(\tilde{r}_k), \quad \hat{r}_k = s_k^{-1} \cdot q_k, \quad r_k = r_{k-1} - \hat{r}_k$$

The parameter $s_k$ is a scalar per stage, enabling dynamic adjustment of residual amplitude (Zhu, 20 Aug 2025).

• Stacked Residual PINNs: Each correction block is weighed by a learnable scalar $\alpha_k$, enforced positive:

$x_{l+1} = x_l + \alpha_l F_l(x_l)$0

The loss incorporates an $x_{l+1} = x_l + \alpha_l F_l(x_l)$1 penalty on $x_{l+1} = x_l + \alpha_l F_l(x_l)$2 to prevent unbounded correction (Eshkofti et al., 18 Mar 2025).

• Transformers (DeRes): Two residual streams are adaptively merged at each layer using a vector-wise gate:

$x_{l+1} = x_l + \alpha_l F_l(x_l)$3

where $x_{l+1} = x_l + \alpha_l F_l(x_l)$4 is a dimension-wise gate from a learned function of both paths, with additional block-attention scaling on $x_{l+1} = x_l + \alpha_l F_l(x_l)$5 (Cheng et al., 6 Jun 2026).

• Norm-Agnostic Residual Networks (NAG): The update is decoupled into magnitude and direction, with $x_{l+1} = x_l + \alpha_l F_l(x_l)$6 and an input-gated $x_{l+1} = x_l + \alpha_l F_l(x_l)$7:

$x_{l+1} = x_l + \alpha_l F_l(x_l)$8

Here, $x_{l+1} = x_l + \alpha_l F_l(x_l)$9 controls the maximal rotation induced by the $\alpha_l$0-th block (Figliolia et al., 15 Jun 2026).

• Residual Expansion Theorem: For $\alpha_l$1 residual blocks each scaled by $\alpha_l$2, the functional expansion exhibits explicit scaling:

$\alpha_l$3

Scaling $\alpha_l$4 interdicts combinatorial path explosion (Dherin et al., 3 Oct 2025).

2. Motivations: Signal Control, Stability, and Expressivity

The rationale for learnable residual scaling arises from several intertwined factors:

• Residual magnitude decay: In multi-stage or deep-stack setups, naïve unscaled addition causes successive residuals to shrink dramatically (as in RFSQ), or, conversely, unbounded accumulation leads to norm explosion (as in vanilla deep ResNets) (Zhu, 20 Aug 2025, Dherin et al., 3 Oct 2025, Figliolia et al., 15 Jun 2026).
• Gradient stability: Attenuated or exploding residuals result in poorly conditioned gradients. Learnable scaling offers a direct mechanism to maintain a controllable Jacobian norm across depth, crucial for training stability at scale.
• Capacity control: By tuning the $\alpha_l$5 (either globally or per-layer), one can interpolate the network’s effective capacity between a shallow, low-complexity model and its full deep ensemble form (Dherin et al., 3 Oct 2025).
• Adaptivity and specialization: Scalar or vector gates, learned through end-to-end optimization, allow the network to route information dynamically across residual, attention, or corrective blocks, yielding per-dimension or per-sample specialization (Cheng et al., 6 Jun 2026, Figliolia et al., 15 Jun 2026).

3. Training, Regularization, and Implementation

Key aspects of implementation for learnable residual scaling include:

• Initialization: Scaling parameters are often initialized to either identity ($\alpha_l$6) or (for deep networks) to $\alpha_l$7 (SkipInit), or to $\alpha_l$8 or $\alpha_l$9 to preempt signal explosion (Zhu, 20 Aug 2025, Dherin et al., 3 Oct 2025, Figliolia et al., 15 Jun 2026).
• Optimization: Scaling parameters are updated by standard gradient methods, sometimes with $F_l$0 penalties to discourage runaway amplification (e.g., regularizing $F_l$1 in RFSQ or $F_l$2 in PINN stacks) (Eshkofti et al., 18 Mar 2025).
• Positivity constraints: Where required, the absolute value of a parameter enforces non-negativity, notably in stacked PINN corrections (Eshkofti et al., 18 Mar 2025).
• Granularity and sharing: Per-stage, per-layer, or per-dimension scaling may all be used, depending on application demands and computational constraints (Cheng et al., 6 Jun 2026, Zhu, 20 Aug 2025).
• Gated path fusion: In advanced architectures (DeRes), gating is implemented via learned affine transformations followed by sigmoid nonlinearity, yielding per-dimension fusion between parallel residual streams (Cheng et al., 6 Jun 2026).
• Regularization and loss: Regularization terms for scaling parameters are sometimes included in the total loss, but, for many cases (e.g., RFSQ with standard weight decay), explicit penalties were empirically found unnecessary (Zhu, 20 Aug 2025).

4. Theoretical Insights and Structural Implications

The presence and adaptability of residual scaling have important theoretical consequences:

• Path-ensemble expansion: The number of effective computational paths grows combinatorially with depth, leading to output and gradient explosion unless scaling factors $F_l$3 are introduced to dampen higher-order contributions (Dherin et al., 3 Oct 2025).
• Capacity and regularization: Learnable scaling modulates both the representational power and the generalization properties of deep models. Scaling can regularize geometric complexity by adjusting the network’s effective ensemble width—thereby providing implicit regularization (Dherin et al., 3 Oct 2025).
• Norm separation: In NAG architectures, decoupling magnitude and direction ensures each layer’s update remains impactful, regardless of the global norm, thereby preventing early layers from overwhelming later ones (Figliolia et al., 15 Jun 2026).
• Gradient propagation: Architectures like DeRes that combine identity skip with learned or gated alternative residuals guarantee both stable gradient backpropagation and adaptive feature selection (Cheng et al., 6 Jun 2026).
• Adaptive depth scaling: Learning-based mixture-of-depths mechanisms leverage per-block scaling and gating to decide layer execution dynamically, enabling effective reuse of compute and improved depth-spanning influence (Figliolia et al., 15 Jun 2026).

5. Empirical Benefits Across Domains

Extensive experimentation underscores the practical benefits of learnable residual scaling:

• Neural Compression (RFSQ): Four-stage RFSQ with learnable scaling achieves $F_l$4 (a $F_l$528% reduction) and perceptual loss $F_l$6 ($F_l$744.5% improvement) over single-stage baselines on ImageNet (Zhu, 20 Aug 2025).
• Stacked Residual PINNs: Adding $F_l$8-weighted correction blocks improves relative $F_l$9 error from $l$0 (no correction) to $l$1 (five blocks), outperforming earlier stacked PINN baselines without learned scaling (Eshkofti et al., 18 Mar 2025).
• Transformer-based CTR Models (DeRes): Vector-gated dual-path residuals improve test AUC to $l$2, outperforming both fixed identity and single-path learnable variants, with dual-path vector gating yielding a $l$3 pp gain over best single-path (Cheng et al., 6 Jun 2026).
• Norm-Agnostic Residual Networks (NAG): NAG shows enhanced training loss reduction, stable gradient/magnitude dynamics, and, under mixture-of-depths execution, maintains performance while reducing executed FLOPs by 20–25% for fixed training compute (Figliolia et al., 15 Jun 2026).
• Normalization-Free Deep ResNets: Residual Expansion analysis demonstrates that principled scaling by $l$4 enables thousands-layer depth without normalization, with learnability offering further adaptation to training requirements (Dherin et al., 3 Oct 2025).

6. Broader Implications and Architectural Generalizations

Learnable residual scaling has broad architectural significance:

• Normalization-free deep networks: Proper scaling is essential to overcome signal explosion in very deep networks, replacing the need for normalization layers in some setups (Dherin et al., 3 Oct 2025).
• Multi-stage quantization and hierarchical coding: Per-stage scaling ensures that each quantization block contributes at its optimal dynamic range, a principle extensible to vector quantization, pruning, and mixed-precision strategies (Zhu, 20 Aug 2025).
• Curriculum-based PDE learning: Learnable scaling enables progressive sharpening of solutions in PINNs by gradually increasing the influence of sharper-residual blocks (Eshkofti et al., 18 Mar 2025).
• Gated composition and adaptive routing: Vector and matrix-wise scaling (e.g., DeRes) integrate stability, adaptivity, and selective forgetting, facilitating specialized and robust information flow across depth (Cheng et al., 6 Jun 2026, Figliolia et al., 15 Jun 2026).
• Capacity/interpolation: Networks can dynamically interpolate between shallow and deep effective ensembles, turning on complexity only as necessary via learnable scaling (Dherin et al., 3 Oct 2025).

7. Limitations, Recommendations, and Open Directions

While learnable residual scaling is theoretically well motivated and empirically validated, certain practical questions persist:

• Initialization and tuning: The optimal initialization (e.g., $l$5, $l$6) and degree of parameterization (scalar vs vector gating) may depend on architecture, task, and dataset scale (Dherin et al., 3 Oct 2025, Cheng et al., 6 Jun 2026).
• Regularization requirement: Explicit regularization of scaling parameters is sometimes beneficial (e.g., in stacked PINNs); in other contexts, standard optimization and weight decay suffice (Eshkofti et al., 18 Mar 2025, Zhu, 20 Aug 2025).
• Scaling granularity: The trade-off between per-layer, per-channel, or more fine-grained scaling remains an architecture- and application-specific choice (Cheng et al., 6 Jun 2026, Zhu, 20 Aug 2025).
• Pathological overfitting: Excessively large or poorly constrained scaling can, in principle, lead to runaway corrections or path dominance, but empirical studies show this is well-controlled with regularization and appropriate loss design (Eshkofti et al., 18 Mar 2025, Cheng et al., 6 Jun 2026).

A plausible implication is that learnable residual scaling offers not only a technical fix for well-known signal propagation issues in deep models but also a powerful, adaptable tool for modulating model complexity, stability, and expressivity across a wide variety of architectures and domains.