---
title: Vanishing & Exploding Gradients
url: https://www.emergentmind.com/topics/vanishing-and-exploding-gradients
type: topic
---

# Vanishing & Exploding Gradients

Vanishing and exploding gradients are fundamental phenomena encountered when training deep neural networks, especially recurrent architectures, that cause the magnitude of gradients to either shrink or grow exponentially with network depth or recurrent horizon. These issues have direct ramifications for optimization dynamics, trainability, and the ability to capture long-range dependencies. The problem originates from the repeated multiplication of Jacobians during backpropagation, leading to exponential accumulation or attenuation of signal through depth or time. Over the past decade, rigorous mathematical formulations, theoretical bounds, and architectural- as well as algorithmic-level remedies have been developed to address these challenges.

## 1. Formal Definition and Core Mechanisms

Let \( \mathcal{L} \) denote a loss computed on the output of a deep neural network with depth \( L \) (feedforward) or time horizon \( T \) (recurrent). The chain rule for backpropagation yields gradients with respect to a parameter (or intermediate state) located early in the network:

\[
g = \frac{\partial \mathcal{L}}{\partial \theta_s} = \left[ \prod_{k=s+1}^{L} J_k \right]^T \frac{\partial \mathcal{L}}{\partial x_L}
\]
where \( J_k = \partial x_k / \partial x_{k-1} \) is the Jacobian of layer \( k \) or the recurrent transition at step \( k \).

The operator norm for the backpropagated gradient chain upper-bounds its magnitude as:

\[
\| g \| \leq \left( \prod_{k=s+1}^{L} \| J_k \| \right) \left\| \frac{\partial \mathcal{L}}{\partial x_L} \right\|
\]

- If \( \| J_k \| < 1 \) on average, the gradient norm decays exponentially (vanishing gradients).
- If \( \| J_k \| > 1 \) on average, the gradient norm grows exponentially (exploding gradients) [1211.5063] [1803.09327].

In recurrent neural networks (RNNs), the weight sharing across time steps can make these effects even more severe: products like \( W^{T}W^{T}\ldots W^{T} \) amplify deviations from spectral isometry dramatically with sequence length.

## 2. Spectral and Statistical Characterization

The spectral radius of the layer-to-layer or recurrent Jacobian is the key determinant:

- For a linear network (or local linearization), repeated action by weight matrices whose largest singular value (or eigenvalue) is \( <1 \) guarantees exponential decay of gradients, and \( >1 \) guarantees exponential explosion [1211.5063].
- For non-linear networks, the maximal singular value of \( J_k \) is further modulated by the maximum of the activation derivative [1803.09327]. In RNNs, the condition \( \| W_{\text{rec}} \|_2 \cdot \gamma < 1 \) (nonlinearity derivative bound \( \gamma \)) suffices for vanishing, \( > 1 \) for explosion.

The input-output Jacobian in deep ReLU networks carries statistical properties determined by architectural constants, e.g., the sum of reciprocals of the hidden widths (\( \beta = \sum 1/n_k \)). The variance of squared gradient entries increases as \( \exp(c \beta) \) [1801.03744]. Sufficiently wide (infinite-width) networks mitigate this effect.

## 3. RNN-Specific Dynamics and the Curse of Memory

Classical analysis reveals that for vanilla RNNs, the product of recurrent Jacobians governs credit assignment across time. If the hidden-to-hidden transition has largest singular value below (above) 1 (after accounting for the activation derivative), long-term gradients vanish (explode), severely restricting learning of long-range temporal dependencies.

A more nuanced challenge termed the “curse of memory” appears in RNNs engineered for long memory (e.g., with spectral radius near 1): as the memory time constant grows, the sensitivity of outputs to infinitesimal parameter changes grows rapidly—even if gradients themselves do not explode. This phenomenon manifests as large output variance under small weight changes in long-memory RNNs, and is mitigated by element-wise recurrence or diagonal recurrent architectures with careful parametrization and normalization [2405.21064].

## 4. Algorithmic Remedies and Architectural Strategies

A variety of techniques have been developed to ameliorate vanishing and exploding gradients, each targeting a different aspect of the underlying propagation chain:

- **SVD and Spectral Parameterization:** Explicitly controlling the singular values of recurrent or feedforward weights, e.g., via efficient SVD parameterization, maintains their operator norms within a narrow band around 1, thereby stabilizing gradient norms across arbitrary depth/time [1803.09327].
- **Orthonormal/Unitary Constraints:** Enforcing orthonormality or unitarity on weight matrices ensures all singular values are exactly 1, which, for repeated application, preserves the backpropagated gradient norm exactly. This can be achieved via direct parameterization or through orthogonality regularization in convolutional or recurrent settings [2006.05066][2210.01245].
- **Gradient Flossing / Lyapunov Controls:** Dynamically steering the spectrum of long-term Jacobians using Lyapunov exponent regularization (gradient flossing) ensures all dominant modes are neutral, i.e., exponents near zero, so that neither vanishing nor exploding behavior dominates. The method is algorithmically implemented via QR-regularized tangent space propagation and differentiation [2312.17306].
- **Norm-Preserving / Volume-Preserving Layers:** Special architectural blocks guarantee that products of Jacobians have determinant (and often singular values) equal to one, establishing equilibrium between expansion and contraction. These methods exploit rotations, permutations, and coupled volume-preserving activations [1911.09576][2006.12169].
- **Gradient Norm Clipping:** For exploding gradients, norm clipping caps the gradient norm to a predefined maximum, preventing oversized update steps. While simple and effective for taming explosions, it does not address vanishing gradients [1211.5063].
- **Initialization Schemes:** Critical initialization, e.g., choosing variances to preserve the expected norm of the forward and backward pass, is essential for ReLU and Maxout nets. Analytical calculation of output moments allows derivation of statistically stable initialization constants [1801.03744][2301.06956].
- **Normalization Layers:** Channel, batch, and layer normalization operate on activations to stabilize gradient propagation by enforcing unit variance along the forward path, which indirectly bounds backward signals. Channel normalization can completely eliminate vanishing gradients in deep CNNs when used with proper scaling [1907.09539].
- **Skip/Residual Connections:** The inclusion of identity skip connections implicitly biases Jacobians toward norms closer to 1, mitigating decay or explosion in gradient signal [2408.01215].
- **Sampling-Based and Regularization Schemes:** Dynamically selecting mini-batches or adding regularizers that control the step-by-step change in long-term gradient norms provides further stabilization against pathological propagation [1606.07767].
- **Gradient Postprocessing:** Activated gradients or Z-Score-normalized gradients (ZNorm) apply nonlinearities or standardization to the raw gradient vector before the parameter update, directly stretching small gradients and shrinking large ones, thereby achieving per-step control [2408.01215][2107.04228].
- **ODE-Inspired Networks:** Discretizations of Hamiltonian or equilibrium-manifold ODEs, with symplecticity or equilibrium iteration, guarantee non-vanishing and/or controlled gradient explosion by construction across arbitrary depth [2105.13205][1908.08574].

## 5. Empirical Evidence and Quantitative Analysis

Empirical evaluation across synthetic and real datasets consistently demonstrates that networks equipped with norm- or spectrum-stabilizing techniques:

- Achieve uniform layer-wise or time-step gradient norms, even in extremely deep (e.g., 30–50,000 layers) or long-horizon (e.g., 10,000 steps) settings [2210.01245][1803.09327][1911.09576][2006.12169].
- Outperform baseline architectures in long-term dependency synthetic benchmarks (addition, copy, temporal/spatial XOR), linguistics (Penn Treebank perplexity), and image classification tasks (MNIST, CIFAR, COCO, ImageNet), attaining both faster convergence and superior accuracy.
- Stabilize gradient-norm ratios such that the maximum-to-minimum norm is constant (≈1) across layers and throughout training for volume/self-normalizing designs [2006.12169][1911.09576].
- Show that batch normalization, while effective, may in fact cause gradient explosion in settings where the inductive bias does not precisely match self-normalization [2006.12169].

Key experimental benchmarks have validated that enforced orthogonality, SVD-based spectrum control, or volume preservation enables effective training of deep or recurrent networks that previously would fail due to vanishing or exploding gradients.

## 6. Theoretical Guarantees, Limitations, and Open Directions

Theoretical results solidify when and why certain approaches are effective:

- Orthogonal/unitary, SVD-parametrized, or volume-preserving layers admit provable upper and lower bounds on gradient norms across depth, guaranteeing stability [1803.09327][2210.01245][1911.09576][2006.12169].
- Gradient flossing delivers upper bounds on the condition number of long-term Jacobians by actively regularizing Lyapunov exponents [2312.17306].
- The “curse of memory” places an additional condition that even in the absence of classical gradient explosion, parameter sensitivity can diverge for networks with extremely long memory; element-wise recurrence and eigenvalue reparameterization are essential for practical optimization [2405.21064].
- For multi-layer deep RNNs, vertical (layer-to-layer) vanishing/exploding gradients persist even when temporal propagation is stabilized by gating, and specialized cells such as the STAR achieve gradient norm preservation by construction [1911.11033].

Some limitations remain: enforcing strict orthogonality can limit expressiveness if not parameterized carefully; choices of regularization strengths or mixing coefficients require empirical tuning; and certain regimes (e.g., highly stochastic or adversarial training) may still challenge gradient stability. There is ongoing exploration into further generalizations, such as block-diagonal/self-normalizing transforms for convolution or self-attention architectures, adaptive regularization, and methods controlling higher-order derivatives and smoothness.

## 7. Broader Implications and Best Practices

The vanishing and exploding gradient phenomena shape nearly every aspect of deep learning algorithm and architecture design. Modern guidelines emphasize:

- Using spectral or orthogonality constraints for deep/residual or recurrent layers to enforce approximate isometry of signal propagation.
- Preferring diagonal or element-wise designs (SSMs, LSTM/GRU forget/input gates, diagonal RNNs) to avoid the curse of memory when long dependencies are required [2405.21064].
- Employing suitable normalization schemes (channel, batch, layer) and initialization constants derived from analytical bounds.
- Applying explicit gradient postprocessing in scenarios where architectural remedies alone do not suffice.

As architectures and tasks grow in complexity, precise control of gradient signal propagation remains a primary criterion for robust, scalable, and generalizable deep learning systems.

---

**References:**
- [1211.5063] Pascanu et al., "On the difficulty of training Recurrent Neural Networks"
- [1803.09327] Vorontsov et al., "Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization"
- [2006.05066] Kwon et al., "Deeply Shared Filter Bases for Parameter-Efficient Convolutional Neural Networks"
- [2312.17306] Hsu et al., "Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobians"
- [2210.01245] Recati, "Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networks"
- [1911.09576] MacDonald et al., "Volume-preserving Neural Networks"
- [1906.08482] Ribeiro et al., "Beyond exploding and vanishing gradients: analysing RNN training using attractors and smoothness"
- [1801.03744] Hanin, "Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?"
- [2006.12169] Wang, "Bidirectionally Self-Normalizing Neural Networks"
- [2105.13205] Falorsi et al., "Hamiltonian Deep Neural Networks Guaranteeing Non-vanishing Gradients by Design"
- [1911.11033] Turkoglu et al., "Gating Revisited: Deep Multi-layer RNNs That Can Be Trained"
- [2408.01215] Wang et al., "ZNorm: Z-Score Gradient Normalization Accelerating Skip-Connected Network Training without Architectural Modification"
- [2405.21064] Behrmann et al., "Recurrent neural networks: vanishing and exploding gradients are not the end of the story"
- [2107.04228] Jin et al., "Activated Gradients for Deep Neural Networks"
- [1907.09539] Heckel, "Channel Normalization in Convolutional Neural Network avoids Vanishing Gradients"
- [1606.07767] Melis and Jaeger, "Sampling-based Gradient Regularization for Capturing Long-Term Dependencies in Recurrent Neural Networks"

These works collectively provide foundational, theoretical, and practical insight into the origination, analysis, and mitigation of vanishing and exploding gradients in modern deep and recurrent neural networks.

Source: https://www.emergentmind.com/topics/vanishing-and-exploding-gradients