Residual Networks of Residual Networks (RoR)

• The paper introduces a method that recursively embeds residual functions to mitigate deep network degradation and simplify optimization.
• It leverages hierarchical shortcut connections at block, group, and root levels to ensure robust gradient propagation and improved convergence.
• Empirical results on benchmarks like CIFAR and ImageNet demonstrate that RoR and Pyramidal RoR offer significant performance gains and regularization benefits.

Residual Networks of Residual Networks (RoR) constitute a family of deep convolutional architectures that generalize the concept of residual learning by recursively embedding residual mappings within other residual mappings. RoR architectures extend the foundational structure of ResNets by introducing multiple hierarchical shortcut connections—spanning block, group, and root levels—which significantly enhance gradient flow, improve optimization dynamics, and achieve state-of-the-art performance across multiple image classification benchmarks (Zhang et al., 2016, Zhang et al., 2017).

1. Rationale and Underlying Principles

Residual learning, as instantiated in standard ResNets, relies on the observation that it is easier to optimize the mapping $F(\mathbf{x}) + \mathbf{x}$ than to learn $F(\mathbf{x})$ directly. RoR extends this insight, positing that nesting residual functions—learning the "residual of a residual"—further simplifies optimization. This approach addresses the degradation problem observed in very deep ResNets, in which increasing depth can lead to optimization difficulty and degraded accuracy. RoR leverages multi-level identity shortcut paths, resulting in direct gradient propagation across all hierarchy levels and improved convergence characteristics (Zhang et al., 2016).

2. Architectural Formulation

A standard residual unit is defined as

$$\mathbf{y}_l = \mathcal{F}(\mathbf{x}_l, \{W_l\}) + h(\mathbf{x}_l), \qquad \mathbf{x}_{l+1} = f(\mathbf{y}_l),$$

where $h(\mathbf{x}_l)$ is the identity or projection shortcut and $f$ denotes ReLU activation. RoR introduces $m$ hierarchical levels of shortcuts:

• Final-level (within block): Standard residual connections.
• Middle-level (within group): Shortcuts over each group of residual blocks.
• Root-level (global): Shortcut spanning all residual blocks.

For RoR-3, with $L$ residual units split into three groups, the group outputs are recursively formed with group- and root-level shortcuts: $$\begin{aligned} y_{L/3} &= g^{(1)}(x_1) + h(x_{L/3}) + F(x_{L/3}, W_{L/3}),\ y_{2L/3} &= g^{(1)}(x_{L/3+1}) + h(x_{2L/3}) + F(x_{2L/3}, W_{2L/3}),\ y_L &= g^{(1)}(x_1) + g^{(2)}(x_{2L/3+1}) + h(x_L) + F(x_L, W_L), \end{aligned}$$ where $g^{(1)}$ and $g^{(2)}$ are root- and group-level shortcuts, respectively.

RoR instantiates this multi-level design analogously on conventional ResNet ("RoR-3"), Pre-activation ResNet ("Pre-RoR-3"), and Wide ResNet ("RoR-3-WRN") backbones (Zhang et al., 2016).

3. Theoretical Implications

The recursive application of residual learning in RoR results in a network that learns:

$F(\mathbf{x})$0

effectively partitioning the optimization into learning smaller corrective terms at each level. This stratification of identity paths alters the gradient propagation such that, for loss $F(\mathbf{x})$1,

$F(\mathbf{x})$2

ensuring multiple “+1” identity factors and further mitigating vanishing gradient phenomena (Zhang et al., 2016).

4. Pyramidal RoR: Channel Width and Block Structure

Pyramidal RoR (Zhang et al., 2017) addresses coherence loss arising from abrupt doubling of feature-map channels at stage boundaries in vanilla ResNets and RoR. Instead, channel width $F(\mathbf{x})$3 in block $F(\mathbf{x})$4 is increased linearly:

$F(\mathbf{x})$5

where $F(\mathbf{x})$6 is the total number of blocks and $F(\mathbf{x})$7 sets the final width. This gradual growth preserves feature continuity and leads to improved classification performance.

The choice of the residual block structure further impacts performance. Empirical assessments favor a "single-ReLU" block structure (BN–Conv–BN–ReLU–Conv–BN, then addition), attaining lower test error versus the pre-activation structure.

5. Training Protocols and Regularization

RoR and Pyramidal RoR utilize SGD optimization, with batch sizes and learning rate schedules adapted to the dataset in use. Stochastic Depth ("drop-path") regularization is systematically employed, with survival probabilities $F(\mathbf{x})$8 linearly decaying across layers (from $F(\mathbf{x})$9 to $$\mathbf{y}_l = \mathcal{F}(\mathbf{x}_l, \{W_l\}) + h(\mathbf{x}_l), \qquad \mathbf{x}_{l+1} = f(\mathbf{y}_l),$$0), both to improve gradient flow and to mitigate overfitting:

$$\mathbf{y}_l = \mathcal{F}(\mathbf{x}_l, \{W_l\}) + h(\mathbf{x}_l), \qquad \mathbf{x}_{l+1} = f(\mathbf{y}_l),$$1

This regime yields significant training speedups and robustness (Zhang et al., 2017).

6. Empirical Performance

Extensive benchmarks on CIFAR-10, CIFAR-100, SVHN, and ImageNet demonstrate the quantitative advantage of RoR and Pyramidal RoR architectures.

Method	CIFAR-10	CIFAR-100	SVHN
RoR-3-WRN58-4+SD (Zhang et al., 2016)	3.77%	19.73%	1.59%
Pyramidal RoR+SD (146, α=270), 38M (Zhang et al., 2017)	2.96%	16.40%	1.59%
Pre-RoR-3-164+SD	4.51%	21.94%	—

On ImageNet, fine-tuning RoR–3 over pre-trained ResNets consistently reduces Top-1 and Top-5 error rates by 0.2–0.3%.

7. Limitations, Extensions, and Future Directions

RoR’s architecture ensures stronger optimization and pervasive gradient pathways. However, increasing depth beyond a certain threshold (e.g., >182 layers) may degrade performance unless combined with pre-activation architectures. Future research directions proposed include adaptive weighting of shortcut levels, automated determination of hierarchical depth/group partitions, and integration with architectural advances such as attention mechanisms or neural architecture search (NAS) (Zhang et al., 2016). Pyramidal RoR further enhances channel width scheduling and residual block design, yielding improved information preservation and regularization (Zhang et al., 2017).