---
title: Residual Expansion Theorem
url: https://www.emergentmind.com/topics/residual-expansion-theorem
type: topic
---

# Residual Expansion Theorem

The Residual Expansion Theorem is an explicit functional expansion for deep residual networks introduced in "On residual network depth" [2510.03470]. It analyzes a residual architecture as a hierarchy of terms indexed by powers of a residual scaling parameter $\lambda$, thereby converting the informal claim that ResNets behave like ensembles of many shallower models into a precise analytic statement. In the formulation of the theorem, depth determines the number and structure of computation paths, while the scaling of residual branches determines whether these paths produce stable, depth-independent contributions or a combinatorial explosion in the output signal [2510.03470].

## 1. Formal statement

The theorem is stated for a residual network
$$
f(x)=D_\eta \circ R_\theta \circ E_\xi(x),
$$
with residual tower
$$
R_\theta=(1+\lambda F_n)\circ \dots \circ (1+\lambda F_1),
$$
where each residual block has the form
$$
(1+\lambda F_i)(z)=z+\lambda F_i(z).
$$
Here, $F_i$ is the nonlinear residual branch and $\lambda$ is a scaling parameter [2510.03470].

The explicit functional expansion for the residual tower is
$$
R_\theta(z)=
z
+\lambda \sum_{i=1}^n F_i(z)
+\lambda^2 \sum_{1\le i<j\le n} F_j'(z)F_i(z)
+\mathcal{O}(\lambda^3),
$$
and for the whole residual network
$$
\begin{aligned}
f(x)=\;&\underbrace{D_\eta(E_\xi(x))}_{\text{Order 0: Base Model } M_0}
+\lambda \underbrace{\sum_{i=1}^{n} W_\eta F_i(E_\xi(x))}_{\text{Order 1: Ensemble } M_1} \\
&+\lambda^2 \underbrace{\sum_{1\le i<j\le n} W_\eta F_j'(E_\xi(x))F_i(E_\xi(x))}_{\text{Order 2: Ensemble } M_2}
+\ldots
\end{aligned}
$$
where $W_\eta$ is the output layer and $F_j'$ is the Jacobian of $F_j$ evaluated at $E_\xi(x)$. The expansion extends to all orders $\mathcal{O}(\lambda^k)$ [2510.03470].

This theorem is presented as the first explicit functional expansion of a deep residual network. Its immediate significance is that it expresses residual depth in terms of concrete analytic contributions rather than only optimization heuristics or empirical analogy [2510.03470].

## 2. Hierarchical ensemble structure

A central consequence of the expansion is a hierarchical ensemble interpretation. The order-$0$ term $M_0$ is the base model, typically shallow or even linear. The order-$1$ term $M_1$ is an ensemble of $n$ models in which a single residual branch is active. The order-$2$ term $M_2$ is an ensemble of $\binom{n}{2}\sim n^2$ models with two interacting residual branches, and these interactions involve Jacobians. Higher-order terms contain combinatorially more paths and higher-order derivatives [2510.03470].

Every functional composition of residual branches corresponds to a distinct path or model in this ensemble. In this sense, network depth directly determines the size and richness of the implicit ensemble. The theorem therefore formalizes the earlier intuition, associated in the paper with Veit et al. (2016), that ResNets resemble “an ensemble of exponentially many shallow networks” [2510.03470].

| Order | Number of paths | Interpretation |
|---|---:|---|
| $0$ | $1$ | Base model (no residual) |
| $1$ | $n$ | $1$ block active |
| $2$ | $\binom{n}{2}$ | $2$ blocks, interactions (first derivatives involved) |
| $\dots$ | $\mathcal{O}(n^k)$ | Deeper ensemble layers, higher derivatives |
| all | $2^n$ | All possible computation paths |

The decomposition should not be confused with a literal collection of independently trained shallow networks. The expansion is written in terms of one residual architecture and its residual branches, Jacobians, and higher-order compositions. This suggests that the ensemble viewpoint is functional rather than architectural [2510.03470].

## 3. Combinatorial growth and the output explosion

The theorem also quantifies how depth affects the number of computation paths. The number of order-$k$ terms is $\binom{n}{k}$, hence $\mathcal{O}(n^k)$, and summing across all orders yields $2^n$ total terms, corresponding to all possible combinations of residual block activations through the skip connections [2510.03470].

Without scaling, that is, when $\lambda=1$, increasing depth causes the total output magnitude to grow exponentially with $n$. The paper describes this as a “combinatorial explosion.” According to the theorem’s interpretation, this explosion is the fundamental cause of instability and untrainability in deep, unnormalized residual networks [2510.03470].

This result gives a structural explanation for a long-observed empirical fact: very deep residual architectures historically depended on normalization layers to remain trainable. In the theorem’s framework, the issue is not merely optimization difficulty in an abstract sense, but the unchecked accumulation of increasingly numerous higher-order functional contributions [2510.03470].

## 4. Scaling, normalization, and normalization-free training

The theorem provides a first-principles explanation for why scaling residual branches can stabilize deep residual networks. The key idea is that scaling by $\lambda$ directly counteracts the combinatorial growth in the ensemble expansion. The paper emphasizes that the theorem yields a specific choice of $\lambda$ ensuring that the magnitude of higher-order ensemble contributions remains independent of network depth [2510.03470].

A distinguished case is $\lambda=1/n$. Then
$$
M_1(x)=\frac{1}{n}\sum_{i=1}^n W_\eta F_i(E_\xi(x)),
$$
which is an average rather than a sum, and
$$
M_2(x)=\frac{1}{n^2}\sum_{1\le i<j\le n} W_\eta F_j'(E_\xi(x))F_i(E_\xi(x)).
$$
With this scaling, the output magnitude at each ensemble order becomes independent of depth $n$ [2510.03470].

This perspective is used to explain the historical role of normalization layers such as BatchNorm or LayerNorm. It is also used to reinterpret normalization-free methods such as Fixup and SkipInit. In the paper’s account, those methods are effective because scaling directly counteracts combinatorial explosion at the functional level, not merely because of optimizer analysis or heuristic analogy to Batch Normalization [2510.03470].

The paper also records an experimental qualification. Scaling by $1/n$ makes extreme depth trainable without normalization, but it may underutilize capacity. Empirically better performance is reported for $\lambda$ between $1/n$ and $1/\sqrt{n}$, for example $\lambda=1/\sqrt n$ [2510.03470].

## 5. Capacity control, regularization, and geometric complexity

The theorem assigns a second role to $\lambda$ beyond stabilization: it acts as a capacity control. When $\lambda=0$, the model reduces to the base model $M_0$, which is the lowest-capacity and shallowest case. As $\lambda$ increases, higher-order ensemble terms contribute more strongly, increasing representational capacity. If $\lambda$ becomes too large, the combinatorial explosion returns, producing unstable models or divergence. Intermediate choices such as $1/\sqrt n$ are reported to balance capacity and stability while enabling rich function classes, increased test accuracy, and maintained generalization [2510.03470].

The appendix gives an explicit first-order expression for model complexity, defined as geometric complexity via the average squared Jacobian norm:
$$
\langle f, D \rangle_{G}
=
\langle M_0, D \rangle_G
+
2\lambda \frac{1}{|D|}
\sum_{x\in D}
\operatorname{Tr}
\left(
[W_\eta E'_\xi(x)]^T
\sum_{i=1}^n
W_\eta F_i'(E_\xi(x))E'_\xi(x)
\right)
+
\mathcal{O}(\lambda^2).
$$
This formula shows explicitly how $\lambda$ controls the degree to which higher-order, potentially more complex behavior is available to the model [2510.03470].

On this basis, the paper concludes that scaling via $\lambda$ serves as a direct regularizer on the complexity of the learned function. The significance of the theorem is therefore not limited to trainability; it also supplies a functional mechanism linking depth, scaling, expressivity, and regularization within a single expansion framework [2510.03470].

## 6. Geometric interpretation and scope of the term

The paper’s geometric interpretation states that the parameter space of a deep residual network embeds all shallower networks. With proper scaling, good solutions for shallower networks remain present and accessible, which stabilizes and facilitates optimization at greater depths [2510.03470]. This interpretation is consistent with the theorem’s broader message: depth enlarges the available hierarchical ensemble, while scaling determines whether that enlargement is usable.

The term “Residual Expansion Theorem” in this sense is specific to residual architectures. It is distinct from other uses of nearby terminology in the literature. "Bäcklund transformations for Burgers Equation via localization of residual symmetries" studies a non-local residual symmetry arising from a truncated Painlevé expansion, its localization by prolongation, and determinant formulas for the $n$th Bäcklund transformation of the Burgers equation; that work concerns integrable PDE symmetries rather than residual-network depth [1401.0982]. "Expansion Theorem for Sturm-Liouville problems transmission conditions" develops eigenfunction expansions, resolvent operators, and Green’s functions for discontinuous Sturm-Liouville problems in a modified Hilbert space; despite the phrase “expansion theorem,” its subject is spectral theory with transmission conditions, not residual connections in deep learning [1303.6893]. "From resolvent expansions at zero to long time wave expansions" concerns the passage from low-energy resolvent expansions to long-time asymptotics for wave equations, again in a separate operator-theoretic setting [2408.03234].

A common terminological misconception is therefore to group these results together because they involve the words “residual” or “expansion.” In current arXiv usage represented here, the Residual Expansion Theorem denotes a theorem about deep residual architectures and their functional decomposition, whereas the other works use “residual” and “expansion” in unrelated meanings tied to integrable systems, Sturm–Liouville theory, or resolvent asymptotics [2510.03470].

Source: https://www.emergentmind.com/topics/residual-expansion-theorem