Canonical ResNet Architectures

• Canonical ResNet architectures are deep convolutional networks that use residual blocks with identity-based skip connections to mitigate vanishing gradients.
• They comprise two main block types—Basic and Bottleneck—with structured convolutions, batch normalization, and ReLU activations to optimize feature learning.
• Recent innovations include parameter-sharing and scaling strategies that reduce redundancy and enhance efficiency on benchmarks like ImageNet.

A canonical ResNet (Residual Network) architecture is a deep convolutional neural network structure distinguished by the use of identity-based skip connections across stacked nonlinear transformations. Its primary innovation is the parameterization of layer-wise outputs via explicit additive merges of input activations with learned residual transformations, allowing stable optimization of deep models by mitigating vanishing gradient and expressivity barriers. First introduced by He et al., canonical ResNets are now a foundational model family for computer vision and signal processing, with extensive theoretical, architectural, and empirical refinements in subsequent literature.

1. Core Residual Block Formulations

Canonical ResNets organize convolutional layers into residual blocks. Each block computes an output as

$$X_{l+1} = g\bigl(X_l + \mathcal{F}(X_l, W_l)\bigr),\quad 0\le l\le L-1$$

where $X_l$ is the input activation, $W_l$ are the block's learnable parameters, $\mathcal{F}$ is a “residual function” comprising a sequence of convolutions, batch normalizations (BN), and nonlinearities (ReLU), and $g(\cdot)$ is typically a post-addition ReLU.

Two structurally distinct block types dominate canonical implementations:

• Basic Block (ResNet-18/34): Two $3\times 3$ convolutions, each with BN and ReLU, with a residual path skip connection (Bello et al., 2021).
• Bottleneck Block (ResNet-50 and deeper): A sequence of $1\times1$ (compression), $3\times3$, and $1\times1$ (decompression) convolutions, each enveloped by BN and ReLU. This compresses intermediate channels for computational efficiency without reducing representational capacity.

Formally:

$$\mathcal{F}_{\text{basic}}(x) = W_2\,\sigma\bigl(\mathrm{BN}(W_1\,x)\bigr)$$

$X_l$0

where $X_l$1 denote convolutional kernels, $X_l$2 is batch normalization, and $X_l$3 is ReLU activation (Bello et al., 2021).

2. Functional Interpretation and Mathematical Modeling

Recent work establishes a mathematical correspondence between canonical ResNet blocks and iterative solvers for linear systems, notably via constrained linear data–feature mapping (He et al., 2021). On each resolution level $X_l$4, features $X_l$5 are obtained from data $X_l$6 by solving

$X_l$7

with $X_l$8 a convolutional linear operator. Extracting $X_l$9 is performed via a residual-correction iteration ("smoother"):

$W_l$0

Imposing a nonnegativity constraint via ReLU at both stages yields the canonical MgNet/ResNet block form. In pre-activation configuration, this updating is algebraically equivalent to the standard pre-activation ResNet block:

$W_l$1

This formalism underlies the design logic for block stacking, skip connections, and feature refinement (He et al., 2021).

3. Block Sequencing, Network Depth, and Scaling Laws

Canonical ResNets build feature pyramids by stacking multiple residual blocks per stage across increasing receptive fields and decreasing resolution. Let $W_l$2 denote the number of weight layers, with block counts $W_l$3 and per-stage output widths $W_l$4. For ResNet-50:

$W_l$5

Example configuration $W_l$6 yields $W_l$7 (Bello et al., 2021).

Scaling strategies empirically optimize speed, accuracy, and generalization:

• Depth scaling is preferable under long training regimes and overfitting risks, as it adds expressivity efficiently and can be better regularized than width scaling.
• Width scaling may outperform depth scaling in resource-constrained or short-training regimes.
• Slow resolution scaling (e.g., $W_l$8, $W_l$9) outperforms aggressive upscaling beyond $\mathcal{F}$0 pixels (Bello et al., 2021).

4. Residual Pathways: Recursion Formulas and Propagation

The residual mechanism allows robust information and gradient flow across arbitrary depths. In canonical ResNets, backpropagated gradients yield a combinatorial ensemble of exponential-length paths:

$\mathcal{F}$1

Expanding this product gives $\mathcal{F}$2 paths per input, allowing deep propagation but at the cost of misuse of capacity across redundant shortcut routes (Liao et al., 2021).

To address path redundancy, alternative architectures revise the recursion formula. One improved recursion prioritizes unique-length paths:

$\mathcal{F}$3

Block activations then follow a second-order recurrence:

$\mathcal{F}$4

This reduces redundancy, introduces memory, and empirically stabilizes optimization and improves accuracy (Liao et al., 2021).

5. Standard Modifications and Parameter-Sharing Variants

Canonical ResNets allocate unique convolutional kernel sets per block ($\mathcal{F}$5). Constrained linear modeling demonstrates that such allocation is redundant: sharing $\mathcal{F}$6 (and/or $\mathcal{F}$7) across blocks within the same stage preserves classification accuracy while reducing parameter count by 20–30% (He et al., 2021). Precise block forms with shared parameters are:

$\mathcal{F}$8

Empirical evidence on ImageNet and CIFAR benchmarks validates this constraint-based compression (He et al., 2021).

6. Training, Architectural Tweaks, and Empirical Performance

Canonical training with stepwise learning rates and minimal regularization produces suboptimal results compared to modern recipes. Marginal gains arise from:

• Extended or cosine learning-rate schedules
• Label smoothing, stochastic depth, dropout
• Strong data augmentations (RandAugment)
• Architectural enhancements such as ResNet-D stems and Squeeze-and-Excitation (SE) modules

Applying these modifications, the ResNet-RS family achieves state-of-the-art speed–accuracy trade-offs. For example, ResNet-RS-152, with 87M parameters and 24 GFLOPs, reaches 82.8% top-1 ImageNet accuracy after 350 epochs, while ResNet-RS-420 (192M params, 128 GFLOPs) attains 84.4% (Bello et al., 2021). ResNet-RS models surpass EfficientNet in speed at given accuracy, and excel at transfer and semi-supervised tasks.

Model	Depth	Params (M)	Top-1 Acc.
ResNet-50	50	36	78.8%
ResNet-152	152	87	82.8%
ResNet-420	420	192	84.4%

7. Generalization and Applicability of Canonical ResNet Principles

The conceptual framework of canonical ResNets—viewing block stacking as residual correction for linear (or nonlinear) systems—underpins the design of variants such as MgNet, ResNetXt, and parameter-shared ResNets. The principles:

• Promote block and parameter efficiency without hampering expressivity
• Enable systematic, formula-driven architectural design (Liao et al., 2021)
• Facilitate transfer to supervised, semi-supervised, and multimodal tasks

The canonical ResNet architecture thus serves as both a universally recognized performance baseline and a flexible source of structural design for deep learning research and deployment.