Progressive Feedforward Collapse in ResNets

• Progressive Feedforward Collapse (PFC) is a geometric phenomenon in ResNets where within-class variability decreases and class means converge toward a simplex ETF with depth.
• PFC employs metrics like PFC1, PFC2, and PFC3 to quantify the evolution of feature collapse, demonstrating a trajectory from raw data to a collapsed terminal representation.
• The concept underpins applications in adversarial detection and continual learning by exploiting evolved feature geometry for improved classification reliability and robustness.

Progressive Feedforward Collapse (PFC) is a conjectured representation-geometric phenomenon in well-trained ResNets in which the degree of collapse increases during forward propagation: within-class variability decreases, class means move toward simplex equiangular tight frame (ETF) geometry, and nearest-class-center separability improves with depth (Wang et al., 2024). In contrast to classical Neural Collapse (NC), which concerns the terminal phase of training at the last layer, PFC is explicitly layer-wise and is intended to describe how intermediate features evolve from the data distribution toward the collapsed terminal representation. Subsequent work has also treated PFC as an exploitable structural prior in adversarial detection and recognition, rather than only as a descriptive property of trained networks (Sun et al., 28 May 2025).

1. Position within the neural-collapse literature

PFC was proposed as a refinement of classical NC for balanced classification problems and well-trained ResNets in their terminal phase (Wang et al., 2024). Classical NC is a last-layer phenomenon: within-class covariance vanishes, centered class means form a simplex ETF, classifier weights align with centered means, and the learned classifier becomes equivalent to nearest-class-mean classification. In the notation used for continual-learning work built directly on NC, these are the NC1–NC4 properties of feature collapse, ETF geometry of class means, classifier–prototype alignment, and decision simplification (Wang et al., 30 May 2025).

PFC extends this picture from the terminal layer to intermediate layers. The defining claim is not merely that several late layers resemble the last layer, but that collapse strengthens monotonically along depth, at least for layers before empirical effective depth. In this sense, PFC is a layer-wise extension of NC along the forward pass: NC is the terminal geometry, whereas PFC is the ordered progression toward it (Wang et al., 2024).

This distinction matters because it changes the object of analysis from a static end state to a trajectory through representation space. In the PFC view, a ResNet does not simply end in a collapsed geometry; it progressively transforms class-conditional feature clouds so that deeper layers are increasingly concentrated and increasingly symmetric. A related operational interpretation appears in adversarial-robustness work, which treats the last few layers of a network “with PFC” as having highly reliable class-centered structure (Sun et al., 28 May 2025).

2. Formal metrics and geometric characterization

The canonical formulation of PFC is given by three layer-wise metrics defined on the feature set at layer $l$, with $\boldsymbol{h}^l_{k,i}$ the feature of sample $i$ in class $k$, $\boldsymbol{h}^l_k$ the class mean, and $\boldsymbol{h}^l_G$ the global mean (Wang et al., 2024).

Metric	Definition	Interpretation
PFC1	$$\mathcal{PFC}_1(\boldsymbol{H}^l)=\dfrac{\mathrm{Tr}(\boldsymbol{\Sigma}_W^l)}{\mathrm{Tr}(\boldsymbol{\Sigma}_B^l)}$$	within-class to between-class variance ratio
PFC2	$$\mathcal{PFC}_2(\boldsymbol{H}^l)=\left\|\dfrac{\tilde{\boldsymbol{H}^l}{}^\top \tilde{\boldsymbol{H}^l}}{\|\tilde{\boldsymbol{H}^l}{}^\top \tilde{\boldsymbol{H}^l}\|_F}-\boldsymbol{E}\right\|_F$$	distance of centered means from simplex ETF geometry
PFC3	$$\mathcal{PFC}_3(\boldsymbol{H}^l)=\dfrac{1}{nK}\sum_{k=1}^K\sum_{i=1}^n \mathbb{I}\!\left[\arg\min_{c\in[K]}\|\boldsymbol{h}^l_{k,i}-\boldsymbol{h}^l_c\|_2=k\right]$$	nearest-class-center accuracy at layer $l$

Here

$\boldsymbol{h}^l_{k,i}$0

and

$\boldsymbol{h}^l_{k,i}$1

The target ETF Gram matrix is

$\boldsymbol{h}^l_{k,i}$2

The PFC conjecture states that, for a balanced classification problem and a well-trained ResNet in its terminal phase, $\boldsymbol{h}^l_{k,i}$3 and $\boldsymbol{h}^l_{k,i}$4 decrease with depth, while $\boldsymbol{h}^l_{k,i}$5 increases with depth (Wang et al., 2024). Geometrically, this means that forward propagation simultaneously reduces within-class spread, regularizes the angular configuration of class means toward a simplex ETF, and makes nearest-class-center decisions increasingly accurate.

In later robustness work, this formal layer-wise geometry is operationalized through class-centered features

$\boldsymbol{h}^l_{k,i}$6

cosine similarities between test features and layer-wise class centers, and aggregated path correlations across layers (Sun et al., 28 May 2025). That use preserves the core PFC intuition: clean examples remain aligned with their class-centered paths, whereas adversarial examples drift away from them.

3. Geodesic and surrogate-model perspectives

The main theoretical picture behind PFC is an optimal-transport interpretation of ResNet training. A residual block is written as

$\boldsymbol{h}^l_{k,i}$7

which is treated as a time discretization of

$\boldsymbol{h}^l_{k,i}$8

If $\boldsymbol{h}^l_{k,i}$9 denotes the feature distribution, mass conservation yields the continuity equation

$i$0

Under weight decay, the ResNet is argued to approximate the geodesic curve in Wasserstein space between the data distribution and the collapsed representation, using the Benamou–Brenier formulation of $i$1 as the minimum transport energy over admissible $i$2 pairs (Wang et al., 2024).

The resulting geodesic-curve assumption states that, at the terminal phase of training, a ResNet with weight decay has learned the geodesic curve in Wasserstein space between the data distribution and the collapsed representation. Forward propagation is then approximated by straight-line interpolation in feature space: $i$3 When NC holds at $i$4, $i$5, and the class means form a simplex ETF. Under this assumption, the paper proves monotonicity statements for PFC1 and PFC2 along the interpolation path, under mild alignment and small-transport conditions (Wang et al., 2024).

A second theoretical component is the Multilayer Unconstrained Feature Model (MUFM), introduced as a surrogate that treats all intermediate layer features as optimization variables and links them with an optimal-transport regularizer. Its multilayer objective is equivalent to

$i$6

so the standard UFM regularization on $i$7 is replaced by a data-dependent penalty on $i$8 (Wang et al., 2024). Rewriting reveals an explicit alignment term,

$i$9

which encourages last-layer features to remain aligned with the input data.

This leads to a nontrivial conclusion: MUFM global minima are not NC solutions in general, because the data-alignment term biases the optimum away from a purely symmetric ETF. At the same time, under MSE loss and sufficiently large $k$0, the optimal features are proved to be more collapsed than the input data along PFC1, even though they do not exhibit perfect NC (Wang et al., 2024). The theory therefore distinguishes exact terminal collapse from a broader class of data-aligned partial collapses.

4. Empirical behavior in ResNet training

Empirical evidence for PFC is reported on fully connected ResNets for MNIST and convolutional ResNets for Fashion-MNIST, CIFAR-10, STL-10, and CIFAR-100, trained with standard SGD and weight decay (Wang et al., 2024). At the final epoch, the reported pattern is consistent across datasets: PFC1 and PFC2 decrease with depth, while PFC3 increases toward $k$1. The same work also computes the PFC metrics along the continuous straight line between input and last-layer features under the geodesic assumption, and those continuous curves reportedly match the observed layer-wise trends closely.

The empirical interpretation is that deeper ResNet blocks do not merely increase discriminability in a generic sense; they impose a specific geometric simplification. Each block moves features toward lower within-class dispersion and more ETF-like between-class arrangement, while also making nearest-center classification progressively more accurate. This suggests that the forward pass can be read as a sequence of increasingly collapsed intermediate representations, rather than as a sequence whose geometry becomes informative only at the penultimate layer (Wang et al., 2024).

A later adversarial-robustness study uses precisely this interpretation. It describes PFC as the regime in which “the output features of the final few layers consistently demonstrate excellent clustering properties,” with layer-wise variability collapse, distance between class means and ETF, and layer-wise nearest class center accuracy as the operative signatures (Sun et al., 28 May 2025). In that work, layer-wise cosine-similarity plots for a ResNet-20 “with PFC” motivate the use of deeper layers for adversarial detection and intermediate layers for adversarial recognition.

5. Operational uses and adjacent extensions

PFC has been used as an explicit prior for adversarial detection and recognition in classical ResNets. The method constructs layer-wise class-centered paths from stored centroids $k$2, computes cosine similarities

$k$3

and aggregates them by hierarchical similarity fusion,

$k$4

The detection statistic is

$k$5

thresholded by an Otsu-style criterion, while adversarial recognition uses weighted voting across selected intermediate layers rather than the deepest layers, because deeper PFC layers can make adversarial examples cluster tightly around the wrong class centroid (Sun et al., 28 May 2025).

The reported results make the geometric use of PFC concrete. On CIFAR-10, the layer-wise feature-path method achieves $k$6 clean accuracy and $k$7 adversarial accuracy on ResNet-20 with PFC, compared with $k$8 and $k$9 for adversarial training; on a standard ResNet-18 it achieves $\boldsymbol{h}^l_k$0 clean accuracy and $\boldsymbol{h}^l_k$1 adversarial accuracy (Sun et al., 28 May 2025). The same study argues that this reveals “inherent adversarial robustness in DNNs,” in the sense that robust internal geometric structure exists even when final-logit adversarial accuracy is only about $\boldsymbol{h}^l_k$2.

A distinct but related extension appears in continual learning under the name Progressive Neural Collapse (ProNC). ProNC does not study layer-wise forward-depth collapse; instead, it progressively expands an ETF target across tasks, initializing the first ETF from the first task’s empirical features and then adding new class prototypes as vertices for later tasks while minimizing shifts of the old vertices (Wang et al., 30 May 2025). The training objective combines cross-entropy with an alignment loss

$\boldsymbol{h}^l_k$3

and a feature-distillation loss

$\boldsymbol{h}^l_k$4

This suggests a broader family of “progressive collapse” programs in which ETF geometry is turned from a descriptive endpoint into an explicit training target.

An operator-theoretic line of work on recurrent networks supplies another adjacent perspective. KPFlow decomposes hidden-state updates under gradient descent as

$\boldsymbol{h}^l_k$5

where $\boldsymbol{h}^l_k$6 is a parameter operator analogous to the NTK and $\boldsymbol{h}^l_k$7 is a linearized flow propagator (Hazelden et al., 8 Jul 2025). That work argues that the same operator view is directly applicable to feedforward architectures and offers a mechanism for PFC through eigenmode selection by NTK-like parameter operators and propagation operators. This suggests that layer-wise collapse may be interpretable as a spectral bias of the composite operators governing gradient flow, rather than only as an empirical geometric regularity.

6. Limitations, open questions, and terminological scope

The original PFC formulation remains conjectural. Its monotonicity claims are supported by empirical measurements and by analysis under the geodesic-curve assumption, but they are not proved for full end-to-end stochastic training in realistic settings (Wang et al., 2024). The geodesic approximation itself is motivated by weight-decay bounds rather than established universally; the analysis is restricted to ResNet architectures, standard weight decay, and balanced classification problems. In addition, the PFC metrics track NC1-, NC2-, and NC4-like behavior, but not NC3-style feature–weight alignment across intermediate layers (Wang et al., 2024).

Application-oriented work inherits additional constraints. The adversarial-robustness study evaluates only AutoPGD with fixed $\boldsymbol{h}^l_k$8 bound $\boldsymbol{h}^l_k$9, on CIFAR-10, using ResNet-20 and ResNet-18; it does not explore CW, DeepFool, black-box attacks, ImageNet-scale datasets, or modern architectures such as ViTs and ConvNeXt (Sun et al., 28 May 2025). The continual-learning ProNC framework assumes clear task boundaries, known class sets per task, replay, and fixed-capacity networks with linear classifier heads (Wang et al., 30 May 2025). These limitations leave open whether analogous progressive-collapse mechanisms are universal across architectures, modalities, and training regimes.

A recurrent source of confusion is terminological. In deep-learning representation geometry, PFC denotes Progressive Feedforward Collapse; in adjacent work, “Progressive Neural Collapse” refers to a continual-learning ETF-expansion framework rather than the layer-wise ResNet conjecture (Wang et al., 30 May 2025). Outside this area, the acronym PFC is also used for unrelated notions, including progressive floor collapse in structural mechanics (Beck, 2008), Priority-based Flow Control in RoCEv2 networking (Liu et al., 2015), Parallel Feedforward Compensation in distributed control (Li et al., 2021), and perturbation-feedback control viewpoints such as T-PFC in robotics (Parunandi et al., 2019). Those usages are acronym collisions, not conceptual variants of the ResNet phenomenon.

The open questions follow directly from these boundaries. The literature explicitly asks whether geodesic and PFC behavior extend beyond ResNets, whether stronger guarantees can be derived without straight-line interpolation assumptions, how PFC correlates with generalization and robustness, and whether OT-style regularizers or other PFC-motivated objectives can be used to steer representation geometry deliberately (Wang et al., 2024). A plausible implication is that PFC will remain important less as a fixed doctrine than as a framework for linking depth, geometry, and training dynamics across multiple settings in modern deep learning.