Cumulative-Goodness Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant

Abstract: Forward-Forward (FF) training allows each layer to learn from a local goodness criterion. In cumulative-goodness variants, however, later layers can inherit a task that earlier layers have already partially separated. We formalize this phenomenon as layer free-riding: under the softplus FF criterion, the class-discrimination gradient reaching block $d$ decays exponentially with the positive margin accumulated by preceding blocks. We then study three local remedies -- per-block, hardness-gated, and depth-scaled -- that recover current-layer separation measures without relying on backpropagated gradients. On CIFAR-10 and CIFAR-100, these remedies dramatically improve layer-separation statistics, with $4\times$--$45\times$ gains in deeper layers, while changing accuracy by less than one percentage point for non-degenerate training procedures. Tiny ImageNet provides a tougher cross-dataset check for our selected block-wise configuration and reveals the same qualitative gap between layer-health diagnostics and final accuracy. Calibration experiments further show that architecture and augmentation choices have a larger effect on final accuracy than the training-rule modifications studied here. Cumulative free-riding is therefore a real and repairable optimization pathology. Nonetheless, for the FF training rules, architectures, and datasets we study, it is not the dominant factor limiting achievable accuracy.

• The paper demonstrates that cumulative-goodness free-riding causes exponential gradient attenuation in deep forward-forward networks.
• It introduces three block-local repair methods that restore layer diagnostics, improving deep block separation metrics up to 45×.
• Experimental results on CIFAR-10, CIFAR-100, and Tiny ImageNet show that enhanced layer health does not significantly boost overall accuracy.

Cumulative-Goodness Free-Riding in Forward-Forward Networks

Theoretical Foundations and Diagnosis of Free-Riding

The paper rigorously formalizes and analyzes cumulative-goodness free-riding in Forward-Forward (FF) networks, where each layer is trained via a local goodness criterion without global backpropagation. In cumulative-goodness schemes, later blocks profit from the successful class separation achieved by earlier layers, resulting in exponentially attenuated parameter gradients for deep layers. The centerpiece is Theorem~1, which analytically describes this attenuation: under the softplus FF criterion, the gradient at block $d$ is rescaled by a ratio $R_\gamma$ that decays exponentially with the accumulated upstream margin. Empirical verification is conducted on CIFAR-10 with multiple $\gamma$ settings, demonstrating monotonic growth of current-block margin when $\gamma=0$ and severe collapse for $\gamma>0$—as predicted by theory.

Figure 1: Empirical verification of Theorem~1: cumulative margin-induced attenuation of current-block gradients across $\gamma$ regimes, validating the exponential decay mechanism.

Three quantitative signatures define the pathology: (1) flat separation—per-layer wrong-label discrimination changes little with depth; (2) loss collapse—deep blocks' losses vanish, indicating no meaningful training signal; and (3) depth saturation—only two blocks capture $\geq$98% of four-block accuracy across all tested variants.

Remedies: Local Discrimination and Adaptive Gating

To mitigate this pathology, the paper introduces three block-local repairs:

1. Strictly local ($\gamma=0$): Eliminates cumulative history, forcing each block to discriminate based solely on its own margin.
2. Hardness-gated collaboration: Allows cumulative history only for unresolved examples, adaptively gating the contribution depending on prior block separation.
3. Depth-scaled discrimination loss: Adds a per-block discrimination term with a weight increasing linearly with depth, thereby enforcing stronger pressure on deeper blocks.

Each remedy is designed to restore per-layer independence diagnostics without cross-block backpropagation, and their theoretical guarantee includes a gradient floor unaffected by upstream margin (Theorem~2).

Empirical Characterization: Accuracy vs. Layer Health

Across CIFAR-10, CIFAR-100, and Tiny ImageNet, all three block-local repairs uniformly enhance layer-separation statistics (layer-health metrics rise by up to $45\times$ in deepest blocks). However, the impact on final classification accuracy is consistently less than one percentage point for non-degenerate settings. Controlled ablation experiments show that architectural and augmentation choices affect accuracy by tens of percentage points, dwarfing the effect of free-riding repairs. For example, $\gamma=0$ (local-only) achieves $R_\gamma$0 S1 TTA on CIFAR-10, closely matching or exceeding prior strict-FF baselines such as ASGE (Gong et al., 15 Sep 2025)—yet repairing free-riding does not close the gap to BP-trained models.

Figure 2: Per-block separation diagnostics across depth for three $R_\gamma$1 regimes; strictly local variants maintain high separation deeper, cumulative variants collapse.

Figure 3: Goodness decomposition at $R_\gamma$2 and $R_\gamma$3: own vs. inherited block-wise contributions showing increased independence after repair.

Figure 4: kappa spectrum analysis at $R_\gamma$4: depth amplification of anti-free-riding effect.

Figure 5: Training dynamics at $R_\gamma$5: per-layer goodness evolution showcasing diverging block health across collaboration modes.

Practical Implications and Controlled Ablation

The FF algorithm's efficacy is strongly rooted in architectural co-design and augmentation, not merely in objective function amendments. Naive FF applied to a standard CNN yields only $R_\gamma$6 accuracy, whereas an FF-optimized backbone reaches $R_\gamma$7. Augmentation is a confound; strong BP-augmentation with the same backbone delivers $R_\gamma$8, surpassing FF. The paper's detailed ablation hierarchy ensures the negative result is not contaminated by architectural or augmentation variance.

Block-wise repairs enable healthy layer metrics but do not translate into improved accuracy, a dissociation formalized by Propositions 1 and 2: redistributing block-wise evidence leaves the cumulative margin, and thus predictions, unchanged unless it alters the overall score ordering. Controlled paired-bootstrap analysis on CIFAR-100 shows prediction disagreements between variants remain nearly net-zero in label correctness, with accuracy shifts $R_\gamma$9\,pp despite large diagnostic swings.

Figure 6: Depth effect comparison, L3/L0 vs. L7/L0 goodness ratio, demonstrating depth amplification of anti-free-riding.

Generalization to Larger Datasets and Deeper Networks

Extension to Tiny ImageNet and CIFAR-100 validates the qualitative separation between layer-health repair and accuracy. Depth amplifies anti-free-riding: at $\gamma$0, adaptive gating produces monotonically increasing per-block goodness, breaking the shallow-network gating threshold and enabling deep block specialization. Yet, accuracy remains nearly invariant under these changes, further supporting the central dissociation claim.

Figure 7: Accuracy vs. free-riding ratio across multiple L4+L8 configurations; only moderate correlation observed, confirming dissociation.

Figure 8: Depth-truncation accuracy: incremental block contributions persist, but repairing free-riding does not materially affect peak accuracy.

Implications for the FF Paradigm and Future Directions

The mechanistic dissociation between layer-health and accuracy narrows the trajectory for further research on FF systems. While cumulative free-riding is a real optimization pathology—exponentially attenuating gradients and starving deep blocks—it is not the dominant bottleneck for achievable accuracy in strict-FF training regimes. This points research toward designing inference rules and representations capable of exploiting redistributed block-wise evidence, rather than focusing exclusively on per-block separation.

The theoretical framework is highly generalizable: the parameter-gradient attenuation applies to any decreasing barrier function, and the diagnosis is relevant for all local-learning schemes where later modules inherit solved subproblems.

Figure 9: Early-exit profiles: adaptive gating produces strong blocks for efficient early exit.

Figure 10: Depth-truncation accuracy: cumulative accuracy remains insensitive to block-level interventions.

Conclusion

Cumulative-goodness free-riding is a rigorously demonstrated and repairable phenomenon in FF networks, arising from exponential gradient attenuation due to upstream margin accumulation. Block-local repairs restore healthy layer metrics with strong theoretical guarantees, but do not impact final accuracy beyond a minimal threshold. The practical and architectural context dominates achievable performance, and repairing free-riding, though mechanistically essential, is not accuracy-dominant. This mechanistic dissociation guides future FF research toward inference and representation designs that materially affect class-score ordering. The claim is scoped to softplus-goodness objectives, Transformer/MoE architectures, and image-domain datasets; further exploration in different local objectives and large-scale settings remains open.