---
title: Progressive Feedforward Collapse in ResNets
url: https://www.emergentmind.com/topics/progressive-feedforward-collapse-pfc
type: topic
---

# Progressive Feedforward Collapse in ResNets

Progressive Feedforward Collapse (PFC) is a conjectured representation-geometric phenomenon in well-trained ResNets in which the degree of collapse increases during forward propagation: within-class variability decreases, class means move toward simplex equiangular tight frame (ETF) geometry, and nearest-class-center separability improves with depth [2405.00985]. In contrast to classical Neural Collapse (NC), which concerns the terminal phase of training at the last layer, PFC is explicitly layer-wise and is intended to describe how intermediate features evolve from the data distribution toward the collapsed terminal representation. Subsequent work has also treated PFC as an exploitable structural prior in adversarial detection and recognition, rather than only as a descriptive property of trained networks [2506.02016].

## 1. Position within the neural-collapse literature

PFC was proposed as a refinement of classical NC for balanced classification problems and well-trained ResNets in their terminal phase [2405.00985]. Classical NC is a last-layer phenomenon: within-class covariance vanishes, centered class means form a simplex ETF, classifier weights align with centered means, and the learned classifier becomes equivalent to nearest-class-mean classification. In the notation used for continual-learning work built directly on NC, these are the NC1–NC4 properties of feature collapse, ETF geometry of class means, classifier–prototype alignment, and decision simplification [2505.24254].

PFC extends this picture from the terminal layer to intermediate layers. The defining claim is not merely that several late layers resemble the last layer, but that collapse strengthens monotonically along depth, at least for layers before empirical effective depth. In this sense, PFC is a layer-wise extension of NC along the forward pass: NC is the terminal geometry, whereas PFC is the ordered progression toward it [2405.00985].

This distinction matters because it changes the object of analysis from a static end state to a trajectory through representation space. In the PFC view, a ResNet does not simply end in a collapsed geometry; it progressively transforms class-conditional feature clouds so that deeper layers are increasingly concentrated and increasingly symmetric. A related operational interpretation appears in adversarial-robustness work, which treats the last few layers of a network “with PFC” as having highly reliable class-centered structure [2506.02016].

## 2. Formal metrics and geometric characterization

The canonical formulation of PFC is given by three layer-wise metrics defined on the feature set at layer \(l\), with \(\boldsymbol{h}^l_{k,i}\) the feature of sample \(i\) in class \(k\), \(\boldsymbol{h}^l_k\) the class mean, and \(\boldsymbol{h}^l_G\) the global mean [2405.00985].

| Metric | Definition | Interpretation |
|---|---|---|
| **PFC1** | \(\mathcal{PFC}_1(\boldsymbol{H}^l)=\dfrac{\mathrm{Tr}(\boldsymbol{\Sigma}_W^l)}{\mathrm{Tr}(\boldsymbol{\Sigma}_B^l)}\) | within-class to between-class variance ratio |
| **PFC2** | \(\mathcal{PFC}_2(\boldsymbol{H}^l)=\left\|\dfrac{\tilde{\boldsymbol{H}^l}{}^\top \tilde{\boldsymbol{H}^l}}{\|\tilde{\boldsymbol{H}^l}{}^\top \tilde{\boldsymbol{H}^l}\|_F}-\boldsymbol{E}\right\|_F\) | distance of centered means from simplex ETF geometry |
| **PFC3** | \(\mathcal{PFC}_3(\boldsymbol{H}^l)=\dfrac{1}{nK}\sum_{k=1}^K\sum_{i=1}^n \mathbb{I}\!\left[\arg\min_{c\in[K]}\|\boldsymbol{h}^l_{k,i}-\boldsymbol{h}^l_c\|_2=k\right]\) | nearest-class-center accuracy at layer \(l\) |

Here
\[
\boldsymbol{\Sigma}_W^l = \frac{1}{nK}\sum_{k,i}(\boldsymbol{h}^l_{k,i}-\boldsymbol{h}^l_k)(\boldsymbol{h}^l_{k,i}-\boldsymbol{h}^l_k)^\top,
\quad
\boldsymbol{\Sigma}_B^l = \frac{1}{K}\sum_k(\boldsymbol{h}^l_k-\boldsymbol{h}^l_G)(\boldsymbol{h}^l_k-\boldsymbol{h}^l_G)^\top,
\]
and
\[
\tilde{\boldsymbol{H}^l}
=
\big[
\boldsymbol{h}^l_1-\boldsymbol{h}^l_G,\dots,\boldsymbol{h}^l_K-\boldsymbol{h}^l_G
\big].
\]
The target ETF Gram matrix is
\[
\boldsymbol{E}
=
\frac{1}{\sqrt{K-1}}
\Big(
\boldsymbol{I}_K-\tfrac{1}{K}\boldsymbol{1}_K\boldsymbol{1}_K^\top
\Big).
\]

The PFC conjecture states that, for a balanced classification problem and a well-trained ResNet in its terminal phase, \(\mathcal{PFC}_1(\boldsymbol{H}^l)\) and \(\mathcal{PFC}_2(\boldsymbol{H}^l)\) decrease with depth, while \(\mathcal{PFC}_3(\boldsymbol{H}^l)\) increases with depth [2405.00985]. Geometrically, this means that forward propagation simultaneously reduces within-class spread, regularizes the angular configuration of class means toward a simplex ETF, and makes nearest-class-center decisions increasingly accurate.

In later robustness work, this formal layer-wise geometry is operationalized through class-centered features
\[
\mu_c^l=\mathrm{Norm}\!\left(\frac{1}{n_c}\sum_{i=1}^{n_c} h_{c,i}^l\right),
\]
cosine similarities between test features and layer-wise class centers, and aggregated path correlations across layers [2506.02016]. That use preserves the core PFC intuition: clean examples remain aligned with their class-centered paths, whereas adversarial examples drift away from them.

## 3. Geodesic and surrogate-model perspectives

The main theoretical picture behind PFC is an optimal-transport interpretation of ResNet training. A residual block is written as
\[
\boldsymbol{x}^{(l+1)}=\boldsymbol{x}^{(l)}+v_l(\boldsymbol{x}^{(l)}),
\]
which is treated as a time discretization of
\[
\frac{d\boldsymbol{x}}{dt}=v(\boldsymbol{x},t),\quad t\in[0,1].
\]
If \(\mu_t\) denotes the feature distribution, mass conservation yields the continuity equation
\[
\frac{d\mu_t}{dt}+\nabla\cdot(v_t\,\mu_t)=0.
\]
Under weight decay, the ResNet is argued to approximate the geodesic curve in Wasserstein space between the data distribution and the collapsed representation, using the Benamou–Brenier formulation of \(W_2^2(\mu_0,\mu_1)\) as the minimum transport energy over admissible \((\mu_t,v_t)\) pairs [2405.00985].

The resulting geodesic-curve assumption states that, at the terminal phase of training, a ResNet with weight decay has learned the geodesic curve in Wasserstein space between the data distribution and the collapsed representation. Forward propagation is then approximated by straight-line interpolation in feature space:
\[
\boldsymbol{h}^{(t)}_{k,i}
=
(1-t)\boldsymbol{h}^{(0)}_{k,i}
+
t\boldsymbol{h}^{(1)}_{k,i},
\quad t\in[0,1].
\]
When NC holds at \(t=1\), \(\boldsymbol{h}^{(1)}_{k,i}=\boldsymbol{h}^{(1)}_k\), and the class means form a simplex ETF. Under this assumption, the paper proves monotonicity statements for PFC1 and PFC2 along the interpolation path, under mild alignment and small-transport conditions [2405.00985].

A second theoretical component is the Multilayer Unconstrained Feature Model (MUFM), introduced as a surrogate that treats all intermediate layer features as optimization variables and links them with an optimal-transport regularizer. Its multilayer objective is equivalent to
\[
\min_{\boldsymbol{W},\boldsymbol{H}}
\;
\mathcal{L}_{\mathrm{CE/MSE}}(\boldsymbol{W},\boldsymbol{H},\boldsymbol{Y})
+
\frac{\lambda_{\boldsymbol{W}}}{2K}\|\boldsymbol{W}\|_F^2
+
\frac{\lambda}{2Kn}\|\boldsymbol{H}-\boldsymbol{X}\|_F^2,
\]
so the standard UFM regularization on \(\|\boldsymbol{H}\|_F^2\) is replaced by a data-dependent penalty on \(\|\boldsymbol{H}-\boldsymbol{X}\|_F^2\) [2405.00985]. Rewriting reveals an explicit alignment term,
\[
-\frac{\lambda}{Kn}\mathrm{Tr}(\boldsymbol{X}\boldsymbol{H}^\top),
\]
which encourages last-layer features to remain aligned with the input data.

This leads to a nontrivial conclusion: MUFM global minima are not NC solutions in general, because the data-alignment term biases the optimum away from a purely symmetric ETF. At the same time, under MSE loss and sufficiently large \(\lambda\), the optimal features are proved to be more collapsed than the input data along PFC1, even though they do not exhibit perfect NC [2405.00985]. The theory therefore distinguishes exact terminal collapse from a broader class of data-aligned partial collapses.

## 4. Empirical behavior in ResNet training

Empirical evidence for PFC is reported on fully connected ResNets for MNIST and convolutional ResNets for Fashion-MNIST, CIFAR-10, STL-10, and CIFAR-100, trained with standard SGD and weight decay [2405.00985]. At the final epoch, the reported pattern is consistent across datasets: PFC1 and PFC2 decrease with depth, while PFC3 increases toward \(1\). The same work also computes the PFC metrics along the continuous straight line between input and last-layer features under the geodesic assumption, and those continuous curves reportedly match the observed layer-wise trends closely.

The empirical interpretation is that deeper ResNet blocks do not merely increase discriminability in a generic sense; they impose a specific geometric simplification. Each block moves features toward lower within-class dispersion and more ETF-like between-class arrangement, while also making nearest-center classification progressively more accurate. This suggests that the forward pass can be read as a sequence of increasingly collapsed intermediate representations, rather than as a sequence whose geometry becomes informative only at the penultimate layer [2405.00985].

A later adversarial-robustness study uses precisely this interpretation. It describes PFC as the regime in which “the output features of the final few layers consistently demonstrate excellent clustering properties,” with layer-wise variability collapse, distance between class means and ETF, and layer-wise nearest class center accuracy as the operative signatures [2506.02016]. In that work, layer-wise cosine-similarity plots for a ResNet-20 “with PFC” motivate the use of deeper layers for adversarial detection and intermediate layers for adversarial recognition.

## 5. Operational uses and adjacent extensions

PFC has been used as an explicit prior for adversarial detection and recognition in classical ResNets. The method constructs layer-wise class-centered paths from stored centroids \(\mu_c^l\), computes cosine similarities
\[
s_c^l=\frac{h_{\text{test}}^l\cdot \mu_c^l}{\|h_{\text{test}}^l\|_2\|\mu_c^l\|_2},
\]
and aggregates them by hierarchical similarity fusion,
\[
S_c=\sum_{l=1}^{L}\omega_l s_c^l.
\]
The detection statistic is
\[
S_{\max}=\max_c S_c,
\]
thresholded by an Otsu-style criterion, while adversarial recognition uses weighted voting across selected intermediate layers rather than the deepest layers, because deeper PFC layers can make adversarial examples cluster tightly around the wrong class centroid [2506.02016].

The reported results make the geometric use of PFC concrete. On CIFAR-10, the layer-wise feature-path method achieves \(82.77\%\) clean accuracy and \(44.17\%\) adversarial accuracy on ResNet-20 with PFC, compared with \(77.64\%\) and \(52.94\%\) for adversarial training; on a standard ResNet-18 it achieves \(80.01\%\) clean accuracy and \(46.1\%\) adversarial accuracy [2506.02016]. The same study argues that this reveals “inherent adversarial robustness in DNNs,” in the sense that robust internal geometric structure exists even when final-logit adversarial accuracy is only about \(5\%\).

A distinct but related extension appears in continual learning under the name Progressive Neural Collapse (ProNC). ProNC does not study layer-wise forward-depth collapse; instead, it progressively expands an ETF target across tasks, initializing the first ETF from the first task’s empirical features and then adding new class prototypes as vertices for later tasks while minimizing shifts of the old vertices [2505.24254]. The training objective combines cross-entropy with an alignment loss
\[
\mathcal{L}_{\mathrm{align}}(\bm{\mu}^{t}_{k,i},\mathbf{e}_{k,t})
=
\frac{1}{2}\left(\mathbf{e}_{k,t}^\top \bm{\mu}^{t}_{k,i}-1\right)^2
\]
and a feature-distillation loss
\[
\mathcal{L}_{\mathrm{distill}}(\bm{\mu}^{(t-1)}_{k,i},\bm{\mu}^{(t)}_{k,i})
=
\frac{1}{2}\left((\bm{\mu}^{(t-1)}_{k,i})^\top \bm{\mu}^{(t)}_{k,i}-1\right)^2.
\]
This suggests a broader family of “progressive collapse” programs in which ETF geometry is turned from a descriptive endpoint into an explicit training target.

An operator-theoretic line of work on recurrent networks supplies another adjacent perspective. KPFlow decomposes hidden-state updates under gradient descent as
\[
\delta z=-PKP^*\,\mathrm{Err},
\]
where \(K\) is a parameter operator analogous to the NTK and \(P\) is a linearized flow propagator [2507.06381]. That work argues that the same operator view is directly applicable to feedforward architectures and offers a mechanism for PFC through eigenmode selection by NTK-like parameter operators and propagation operators. This suggests that layer-wise collapse may be interpretable as a spectral bias of the composite operators governing gradient flow, rather than only as an empirical geometric regularity.

## 6. Limitations, open questions, and terminological scope

The original PFC formulation remains conjectural. Its monotonicity claims are supported by empirical measurements and by analysis under the geodesic-curve assumption, but they are not proved for full end-to-end stochastic training in realistic settings [2405.00985]. The geodesic approximation itself is motivated by weight-decay bounds rather than established universally; the analysis is restricted to ResNet architectures, standard weight decay, and balanced classification problems. In addition, the PFC metrics track NC1-, NC2-, and NC4-like behavior, but not NC3-style feature–weight alignment across intermediate layers [2405.00985].

Application-oriented work inherits additional constraints. The adversarial-robustness study evaluates only AutoPGD with fixed \(\ell_\infty\) bound \(8/255\), on CIFAR-10, using ResNet-20 and ResNet-18; it does not explore CW, DeepFool, black-box attacks, ImageNet-scale datasets, or modern architectures such as ViTs and ConvNeXt [2506.02016]. The continual-learning ProNC framework assumes clear task boundaries, known class sets per task, replay, and fixed-capacity networks with linear classifier heads [2505.24254]. These limitations leave open whether analogous progressive-collapse mechanisms are universal across architectures, modalities, and training regimes.

A recurrent source of confusion is terminological. In deep-learning representation geometry, PFC denotes Progressive Feedforward Collapse; in adjacent work, “Progressive Neural Collapse” refers to a continual-learning ETF-expansion framework rather than the layer-wise ResNet conjecture [2505.24254]. Outside this area, the acronym PFC is also used for unrelated notions, including progressive floor collapse in structural mechanics [0808.2846], Priority-based Flow Control in RoCEv2 networking [1509.03559], Parallel Feedforward Compensation in distributed control [2110.12787], and perturbation-feedback control viewpoints such as T-PFC in robotics [1902.01389]. Those usages are acronym collisions, not conceptual variants of the ResNet phenomenon.

The open questions follow directly from these boundaries. The literature explicitly asks whether geodesic and PFC behavior extend beyond ResNets, whether stronger guarantees can be derived without straight-line interpolation assumptions, how PFC correlates with generalization and robustness, and whether OT-style regularizers or other PFC-motivated objectives can be used to steer representation geometry deliberately [2405.00985]. A plausible implication is that PFC will remain important less as a fixed doctrine than as a framework for linking depth, geometry, and training dynamics across multiple settings in modern deep learning.

Source: https://www.emergentmind.com/topics/progressive-feedforward-collapse-pfc