---
title: Canonical ResNet Architectures
url: https://www.emergentmind.com/topics/canonical-resnet-architectures
type: topic
---

# Canonical ResNet Architectures

A canonical ResNet (Residual Network) architecture is a deep convolutional neural network structure distinguished by the use of identity-based skip connections across stacked nonlinear transformations. Its primary innovation is the parameterization of layer-wise outputs via explicit additive merges of input activations with learned residual transformations, allowing stable optimization of deep models by mitigating vanishing gradient and expressivity barriers. First introduced by He et al., canonical ResNets are now a foundational model family for computer vision and signal processing, with extensive theoretical, architectural, and empirical refinements in subsequent literature.

## 1. Core Residual Block Formulations

Canonical ResNets organize convolutional layers into residual blocks. Each block computes an output as
$$
X_{l+1} = g\bigl(X_l + \mathcal{F}(X_l, W_l)\bigr),\quad 0\le l\le L-1
$$
where $X_l$ is the input activation, $W_l$ are the block's learnable parameters, $\mathcal{F}$ is a “residual function” comprising a sequence of convolutions, batch normalizations (BN), and nonlinearities (ReLU), and $g(\cdot)$ is typically a post-addition ReLU.

Two structurally distinct block types dominate canonical implementations:
- **Basic Block (ResNet-18/34):** Two $3\times 3$ convolutions, each with BN and ReLU, with a residual path skip connection [2103.07579].
- **Bottleneck Block (ResNet-50 and deeper):** A sequence of $1\times1$ (compression), $3\times3$, and $1\times1$ (decompression) convolutions, each enveloped by BN and ReLU. This compresses intermediate channels for computational efficiency without reducing representational capacity.

Formally:
$$
\mathcal{F}_{\text{basic}}(x) = W_2\,\sigma\bigl(\mathrm{BN}(W_1\,x)\bigr)
$$
$$
\mathcal{F}_{\text{bottleneck}}(x) = W_3\,\sigma(\mathrm{BN}(W_2\,\sigma(\mathrm{BN}(W_1\,x))))
$$
where $W_i$ denote convolutional kernels, $\mathrm{BN}$ is batch normalization, and $\sigma$ is ReLU activation [2103.07579].

## 2. Functional Interpretation and Mathematical Modeling

Recent work establishes a mathematical correspondence between canonical ResNet blocks and iterative solvers for linear systems, notably via constrained linear data–feature mapping [2112.07441]. On each resolution level $\ell$, features $u^\ell$ are obtained from data $f^\ell$ by solving
$$
A^\ell * u^\ell = f^\ell,\quad u^\ell \geq 0
$$
with $A^\ell$ a convolutional linear operator. Extracting $u^\ell$ is performed via a residual-correction iteration ("smoother"):
$$
u^{\ell,i} = u^{\ell,i-1} + B^{\ell,i} * (f^\ell - A^\ell * u^{\ell,i-1})
$$
Imposing a nonnegativity constraint via ReLU at both stages yields the canonical MgNet/ResNet block form. In pre-activation configuration, this updating is algebraically equivalent to the standard pre-activation ResNet block:
$$
x_{k+1} = x_k + A^\ell * \sigma (B^{\ell,i} * \sigma(x_k))
$$
This formalism underlies the design logic for block stacking, skip connections, and feature refinement [2112.07441].

## 3. Block Sequencing, Network Depth, and Scaling Laws

Canonical ResNets build feature pyramids by stacking multiple residual blocks per stage across increasing receptive fields and decreasing resolution. Let $L$ denote the number of weight layers, with block counts $\{B_2, B_3, B_4, B_5\}$ and per-stage output widths $\{C_2, C_3, C_4, C_5\}$. For ResNet-50:
$$
L = 1\ (\text{stem}) + 3 \sum_{i=2}^5 B_i + 1\ (\text{final FC})
$$
Example configuration $[3,4,6,3]$ yields $L=50$ [2103.07579].

Scaling strategies empirically optimize speed, accuracy, and generalization:
- **Depth scaling** is preferable under long training regimes and overfitting risks, as it adds expressivity efficiently and can be better regularized than width scaling.
- **Width scaling** may outperform depth scaling in resource-constrained or short-training regimes.
- **Slow resolution scaling** (e.g., $r = r_0 (F/F_0)^{\beta_r}$, $\beta_r \approx 0.1-0.2$) outperforms aggressive upscaling beyond $320 \times 320$ pixels [2103.07579].

## 4. Residual Pathways: Recursion Formulas and Propagation

The residual mechanism allows robust information and gradient flow across arbitrary depths. In canonical ResNets, backpropagated gradients yield a combinatorial ensemble of exponential-length paths:
$$
\frac{\partial X_L}{\partial X_{L-i}} = \prod_{k=L-i+1}^L (1 + W_k)
$$
Expanding this product gives $2^i$ paths per input, allowing deep propagation but at the cost of misuse of capacity across redundant shortcut routes [2108.08689].

To address path redundancy, alternative architectures revise the recursion formula. One improved recursion prioritizes unique-length paths:
$$
\frac{\partial X_L}{\partial X_{L-i}} = 1 + W_L + W_L W_{L-1} + \dots + W_L\cdots W_{L-i+1}
$$
Block activations then follow a second-order recurrence:
$$
X_l = (1 + W_l) X_{l-1} - W_{l-1} X_{l-2}
$$
This reduces redundancy, introduces memory, and empirically stabilizes optimization and improves accuracy [2108.08689].

## 5. Standard Modifications and Parameter-Sharing Variants

Canonical ResNets allocate unique convolutional kernel sets per block ($A^{\ell,i}, B^{\ell,i}$). Constrained linear modeling demonstrates that such allocation is redundant: sharing $A^\ell$ (and/or $B^\ell$) across blocks within the same stage preserves classification accuracy while reducing parameter count by 20–30% [2112.07441]. Precise block forms with shared parameters are:
$$
r^{\ell,i} = r^{\ell,i-1} - A^\ell * \sigma\,(B^\ell * \sigma(r^{\ell,i-1}))
$$
Empirical evidence on ImageNet and CIFAR benchmarks validates this constraint-based compression [2112.07441].

## 6. Training, Architectural Tweaks, and Empirical Performance

Canonical training with stepwise learning rates and minimal regularization produces suboptimal results compared to modern recipes. Marginal gains arise from:
- Extended or cosine learning-rate schedules
- Label smoothing, stochastic depth, dropout
- Strong data augmentations (RandAugment)
- Architectural enhancements such as ResNet-D stems and Squeeze-and-Excitation (SE) modules

Applying these modifications, the ResNet-RS family achieves state-of-the-art speed–accuracy trade-offs. For example, ResNet-RS-152, with 87M parameters and 24 GFLOPs, reaches 82.8% top-1 ImageNet accuracy after 350 epochs, while ResNet-RS-420 (192M params, 128 GFLOPs) attains 84.4% [2103.07579]. ResNet-RS models surpass EfficientNet in speed at given accuracy, and excel at transfer and semi-supervised tasks.

| Model      | Depth | Params (M) | Top-1 Acc. |
|:-----------|------:|-----------:|-----------:|
| ResNet-50  |   50  |       36   |     78.8%  |
| ResNet-152 |  152  |       87   |     82.8%  |
| ResNet-420 |  420  |      192   |     84.4%  |

## 7. Generalization and Applicability of Canonical ResNet Principles

The conceptual framework of canonical ResNets—viewing block stacking as residual correction for linear (or nonlinear) systems—underpins the design of variants such as MgNet, ResNetXt, and parameter-shared ResNets. The principles:
- Promote block and parameter efficiency without hampering expressivity
- Enable systematic, formula-driven architectural design [2108.08689]
- Facilitate transfer to supervised, semi-supervised, and multimodal tasks

The canonical ResNet architecture thus serves as both a universally recognized performance baseline and a flexible source of structural design for deep learning research and deployment.

Source: https://www.emergentmind.com/topics/canonical-resnet-architectures