Residual Classification Flow

• Residual Classification Flow (RCF) is a framework that combines deep residual network dynamics with density estimation for robust classification and out-of-distribution detection.
• RCF leverages cooperative and competitive residual dynamics to generate class-dependent feature bands even when full attractor convergence is not reached.
• RCF extends normalizing flows by using residual mappings to model multimodal data distributions, yielding significant improvements in OOD detection metrics.

Residual Classification Flow (RCF) encompasses a set of formal, dynamical, and architectural concepts that describe how deep residual networks (ResNets) perform data classification through the integration of residual signals across network layers and, in recent developments, how expressive normalizing flows based on residual connections can be leveraged for density modeling and out-of-distribution (OOD) detection. This paradigm, analyzed both from the dynamical systems perspective and the standpoint of expressive density estimation, highlights the interplay between network architecture, transient dynamics, and robust representation learning in deep neural networks (Zisselman et al., 2020, Lagzi, 2021).

1. Mathematical Foundations of Residual Dynamics

In a canonical fully connected ResNet—with input dimension $N$, depth $T$, shared weight matrix $W \in \mathbb{R}^{N \times N}$, logistic activation $f(z) = (1 + e^{-z})^{-1}$—the network state after the skip connection at layer $t$ is denoted $\mathbf{x}(t)$, with the nonlinear transform $\mathbf{y}(t)$ of the layer block. The discrete-time dynamics are given by: $$\mathbf{x}(t+1) = \mathbf{x}(t) + \mathbf{y}(t+1), \quad \mathbf{y}(t+1) = f\left(W\mathbf{x}(t) + \mathbf{b}\right)$$ This recurrence can be interpreted continuously as: $$\dot{x}_i(t) = y_i(t), \quad y_i(t) = f\left(z_i(t)\right), \quad z_i(t) = \sum_{j=1}^N w_{ij} x_j(t) + b_i$$ The dynamics of the residuals $\mathbf{y}(t)$ satisfy: $$\dot{y}_i(t) = y_i(t)(1 - y_i(t)) \sum_{j=1}^N w_{ij} y_j(t)$$ which forms a nonlinear, cooperative-competitive Lotka–Volterra system controlling the evolution of internal features (Lagzi, 2021).

2. Cooperation, Competition, and Feature Encoding

The network dynamics reflect the interaction between cooperative (excitatory) and competitive (inhibitory) modes. The effective drive on neuron $i$ is: $H_i(\mathbf{y}) = \sum_{j=1}^N w_{ij} y_j$ Splitting into cooperative and competitive terms,

$$H_i(\mathbf{y}) = C_i(\mathbf{y}) - D_i(\mathbf{y})$$

feature encoding emerges through cooperative amplification (positive weights) and competitive suppression (negative weights). Fixed points (attractors) satisfy either $y_i^* \in \{0, 1\}$ or $H_i(\mathbf{y}^*) = 0$, so individual dimensions can saturate in response to recurring features across input examples. Stable attractors encode shared data structure; these define the basins to which training trajectories converge (Lagzi, 2021).

3. Transient Dynamics and Classification Mechanism

In deep ResNets, full convergence to attractors is typically not achieved within network depth $T$. The residual trajectory $\{\mathbf{y}(t)\}_{t=1}^T$ forms class-dependent bands in feature space. The accumulated post-skip state,

$$\mathbf{x}(T) = \mathbf{x}(0) + \sum_{t=1}^T \mathbf{y}(t)$$

serves as the classifier’s input, with scoring by $K_c^{\top} \mathbf{x}(T)$ for each class $c$. Small differences in transient flow trajectories $\mathbf{y}(t)$ propagate to significant separations in $\mathbf{x}(T)$, enabling robust classification even in the absence of attractor convergence (Lagzi, 2021).

4. Residual Flows for Expressive Density Estimation

Residual flows extend the residual mapping framework to normalizing flows: let $f_k(x) = x + v_k(x)$ for $K$ residual blocks, where each $v_k$ is a nonlinear network (often using affine coupling as in RealNVP). The full bijection $f = f_K \circ ... \circ f_1$ transforms data from input space to a latent space with base Gaussian distribution $p_0(z) = \mathcal{N}(z; 0, I)$. The log density is given by: $$\log p(x) = \log p_0(f(x)) + \sum_{k=1}^K \log |\det J_{f_k}(u_k)|$$ centered classwise features—typically penultimate layer activations—serve as flow inputs, enabling modeling of multimodal, skewed distributions beyond linear Gaussian approximations. The initial Gaussian flow approximates the Mahalanobis baseline; flexible residual flows then “tune” the density to match true data structure (Zisselman et al., 2020).

5. Training, OOD Detection, and Empirical Results

Residual flows are trained by negative log-likelihood over centered in-distribution activations. Initialization with an analytic linear flow (covariance fit) ensures numerical stability, with deep blocks fine-tuned via Adam optimizers: $$\mathcal{L}(\theta) = -\sum_{i} \log p_{\theta}(x_i)$$ For OOD detection, the model scores test examples layerwise and classwise using fitted density models, then aggregates via a weighted sum tuned on a hold-out set. When comparing ResNet-34 trained on CIFAR-100 versus OOD data from Tiny-ImageNet, Mahalanobis TNR@95\% is 56.7\%, while Residual Flow achieves 77.5\%. Similar improvements are observed on other dataset pairs (CIFAR-100/LSUN, DenseNet-100/LSUN), with Residual Flow often increasing TNR@95 by 10–30 points and strictly dominating Mahalanobis/AUROC/AUPR baselines (Zisselman et al., 2020).

Architecture-Task	Mahalanobis TNR@95%	Residual Flow TNR@95%
ResNet-34: CIFAR-100/ImageNet	56.7%	77.5%
ResNet-34: CIFAR-100/LSUN	38.4%	70.4%
DenseNet-100: CIFAR-100/LSUN	83.6%	96.3%

Residual flows maintain stable training by starting near the linear (Gaussian) baseline and incrementally learning more complex densities, systematically outperforming previous Gaussian-based OOD detection systems (Zisselman et al., 2020).

6. Robustness to Perturbations and Adaptive Depth

Analysis of noise propagation reveals that ResNets with skip–sum architectures distribute input perturbations over many layers, offering greater robustness than comparable MLPs for large noise amplitudes. Sensitivity propagates according to the chain Jacobian, with early-layer perturbations having larger impact due to the product structure: $$\frac{\partial C}{\partial \mathbf{y}(s)} = K^\top \prod_{t=s}^T \left[I + \operatorname{diag}(y(t) \odot (1 - y(t)))W\right]$$ Adaptive depth control is achieved by pruning layers once the mean $\ell_1$-change in residuals falls below threshold, without accuracy loss—layers beyond effective dynamic settling contribute negligibly to classification (Lagzi, 2021).

7. Extensions and Generalizations

Residual Classification Flow architectures are generic and applicable to any approximately Gaussian feature distributions—including domains such as speech or word embedding modeling. The flow can be made class-conditional or extended with more expressive coupling schemes (e.g., invertible $1\times 1$ convolutions as in Glow). The flexibility and stability conferred by starting from analytically initialized flows and compact, layerwise residual networks underpin the methodological advantages of the RCF approach (Zisselman et al., 2020).

---

Residual Classification Flow unifies the dynamical systems perspective on deep classification with modern density estimation techniques. By leveraging stable, expressive mappings and integrating transient residual flows, RCF addresses both representation learning and robust OOD detection in deep neural architectures (Zisselman et al., 2020, Lagzi, 2021).