Forward-Forward (FF) Algorithm

• Forward-Forward (FF) is a biologically plausible training algorithm that replaces backpropagation by using only forward passes and layer-local goodness functions.
• It employs positive and negative data pairs to compute local losses, enabling efficient, competitive learning in CNNs, SNNs, and quantized networks.
• FF promotes scalable, energy-efficient neural computation through innovations like adaptive sparsity, channel-wise losses, and distributed training methods.

The Forward-Forward (FF) algorithm is a biologically plausible alternative to backpropagation that enables deep neural network training using only forward passes and layer-local objectives, thereby circumventing the need for global error propagation and weight transport. The FF paradigm relies on constructing two types of data per sample—positives (correct label/data pairings) and negatives (corrupted label or input)—and training each layer to discriminate between them via a scalar goodness function. Since its inception, FF has rapidly evolved from early fully connected perceptrons to competitive architectures on convolutional, recurrent, and spiking domains, with extensive advances in goodness function design, energy efficiency, parallelizability, and large-scale applicability.

1. Core Principles of the Forward-Forward Algorithm

At its foundation, the FF algorithm eschews the global backward pass of backpropagation in favor of purely forward computation. Each layer maintains a local scalar "goodness" function—commonly the sum of squared activations, but increasingly generalizable—to differentiate positive from negative samples:

$$G(y^{(\ell)}) = \sum_{i=1}^{N_{\ell}} (y_i^{(\ell)})^2$$

For each training step, both positive (correct) and negative (incorrect or corrupted) inputs are constructed, and two forward passes are performed per layer. A local loss per layer is computed to maximize goodness for positives and minimize it for negatives. The most frequently used per-layer loss is a sum of two logistic terms centered at a threshold $\theta$:

$$\mathcal{L}^{(\ell)} = \log\big[1+\exp(-G(y^{(\ell)}_{+})+\theta)\big] + \log\big[1+\exp(G(y^{(\ell)}_{-})-\theta)\big]$$

Layer weights are updated using only local gradients derived from their own activations and goodness, allowing each layer to function as an independent learning unit (Hinton, 2022, Adamson, 15 Apr 2025).

2. Goodness Functions: Diversity and Impact

The choice of goodness function is pivotal for FF's efficacy and is a subject of extensive benchmarking. While the sum-of-squares remains canonical, over 20 alternatives have been systematically benchmarked—including game-theoretic, sparse $\ell_1$, predictive coding, triplet margin, energy-margin, and adaptive sparsity mechanisms (Shah et al., 23 Nov 2025, Yuksel et al., 28 Mar 2026). Empirically, sparse and information-theoretic goodness functions (e.g., top-$k$, entmax-weighted, predictive coding) yield significant accuracy gains, with adaptive sparsity (entmax-1.5) achieving up to +30.7 percentage points on Fashion-MNIST compared to sum-of-squares (Yuksel et al., 28 Mar 2026). Energy consumption and carbon footprint measurements reveal that certain goodness objectives deliver superior predictive performance and environmental efficiency.

Recent results highlight:

• Sparsity Principle: Restricting goodness evaluation to the most active neurons (top-$k$ or entmax-weighted) substantially enhances discriminative signal (Yuksel et al., 28 Mar 2026).
• Information-theoretic and margin-based functions: Objectives such as predictive_coding_local and softmax_energy_margin_local enable improved performance, particularly in label-scarce or fine-grained settings (Shah et al., 23 Nov 2025).

These findings establish the goodness function as a hyperparameter comparable in significance to architecture or optimizer choices.

3. Architectural Extensions and Distributed Paradigms

FF generalizes beyond fully-connected networks to convolutional neural networks (CNNs) and SNNs. In convolutional settings, goodness can be computed per class using channel-wise partitioning and spatial pooling, and sophisticated architectural blocks (e.g., Channel-wise Feature Separator and Extractor, CFSE) facilitate modular, layer-wise representational specialization (Papachristodoulou et al., 2023, Scodellaro et al., 2023).

Notable innovations include:

• Channel-wise Competitive (CwC) Loss: By partitioning channels and applying cross-entropy losses over class-specific block activations, the need for explicit negative data is eliminated (Papachristodoulou et al., 2023).
• Hyperspherical FF (HFF): Local objectives are reframed as multi-class classification on the unit hypersphere via learnable class-specific prototypes, yielding $\mathcal{O}(1)$ inference cost per input and closing the performance gap to backprop on large-scale benchmarks (Sarode et al., 30 Apr 2026).
• Hierarchical and Contrastive Learning (HCL-FF): Coarse-to-fine hierarchical label curriculum and layer-local supervised contrastive losses further enhance FF's scalability and feature discriminability (Yao et al., 24 May 2026).

Distributed training capabilities leverage the inherent decoupling of layers: pipeline-parallel and federated variants facilitate multi-node training with near-linear speedups and no accuracy degradation, supporting scalable and privacy-respecting applications (Aktemur et al., 2024).

4. Applications to Spiking, Quantized, and Low-Power Learning

The FF paradigm is well-suited for architectures in which backward error signals are unattainable or inefficient, most notably spiking neural networks (SNNs) and quantized/deployment-ready DNNs:

• Spiking Neural Networks: FF frameworks combined with surrogate gradient methods achieve competitive or superior performance to BP-based SNNs on static and temporal benchmarks, with efficient local goodness objectives over spike count statistics (Ghader et al., 19 Feb 2025). Furthermore, FF enables gradient approximation-free SNN training with adaptive complexity mechanisms for edge deployment (Xu et al., 31 Jul 2025).
• Low-Precision Edge Training: FF naturally accommodates symmetric quantization (e.g., INT8) due to its local update rule. On real edge devices, FF-INT8 achieves over 27% reduction in memory use, 8.3% energy savings, and minor (≤0.4pp) accuracy penalties relative to full-precision BP. The "look-ahead" mechanism, which injects future layer losses into local objectives, mitigates greedy layerwise limitations and recovers accuracy (Ma et al., 28 Jun 2025).

5. Layer-Wise Dynamics, Training Behavior, and Interpretability

FF networks exhibit distinct training dynamics compared to backprop-based systems:

• Shallower layers converge faster and their accuracy is more strongly correlated with overall model performance, suggesting a "front-to-back" learning dynamic (Adamson, 15 Apr 2025).
• Hierarchical curricula and contrastive objectives, such as in HCL-FF, promote coarse-to-fine knowledge formation across layers and robust class clusterings in representational space (Yao et al., 24 May 2026).
• FF-induced representations are characterized by extreme sparsity, category-specific ensemble activations, and emergent functional specialization, paralleling observations in biological cortex. Weight polarity imbalances and positive skew in activity further support FF's neurobiological plausibility (Tosato et al., 2023).

The choice of normalization (e.g., group norm, L2 norm), label injection strategy (e.g., feature forwarding at every layer), and architectural depth all interact synergistically with goodness design to determine performance and interpretability (Yuksel et al., 28 Mar 2026).

6. Limitations, Challenges, and Future Directions

Notwithstanding its advances, FF faces several open challenges:

• Scalability to extremely deep and wide architectures or those with attention-like modules requires further algorithmic and theoretical analysis.
• Greedy, decoupled layerwise updates can underutilize inter-layer synergies; hierarchical and margin-based losses, layer-collaboration strategies, and “look-ahead” penalties are active areas for remedy (Wu et al., 2024, Ma et al., 28 Jun 2025, Yao et al., 24 May 2026).
• Current models commonly require explicit or engineered negative samples, which can be circumvented by self-contrastive or diversity-maximizing objectives (Chen et al., 2024, Zhu et al., 22 May 2025).
• Extending to self-supervised, unsupervised, and reinforcement learning frameworks, as well as large-scale corpus and multimodal settings (e.g., ImageNet, Tiny-ImageNet), are prominent directions (Sarode et al., 30 Apr 2026, Yao et al., 24 May 2026).
• Hardware deployment—neuromorphic cores, microcontrollers, and in-memory learning—benefits directly from FF’s local, forward-only structure, but further co-design is needed for reliability at low bit-depth or in presence of device noise (Wu et al., 2024, Ma et al., 28 Jun 2025).

Continued benchmarking across goodness functions, goodness sparsity levels, environmental efficiency, and robustness to adversarial/hardware noise is essential to identifying optimal design patterns (Shah et al., 23 Nov 2025, Yuksel et al., 28 Mar 2026).

7. Summary Table: Key FF Advances and Associated Papers

Innovation/Theme	Core Publication(s)	Quantitative/Qualitative Insights
Vanilla FF, local loss, MLP/CNN	(Hinton, 2022); (Scodellaro et al., 2023)	99.16% MNIST (CNN); competitive with BP up to moderate scale
Goodness function benchmarking	(Shah et al., 23 Nov 2025); (Yuksel et al., 28 Mar 2026)	Sparse/contrastive functions outperform sum-of-squares
Channel-wise/competitive learning	(Papachristodoulou et al., 2023)	CwC loss bridges FF–BP performance gap in CNNs
Distributed/parallel FF pipeline	(Aktemur et al., 2024)	3.75× speed-up with 4 nodes, no loss in accuracy
Spiking/Neuromorphic FF	(Ghader et al., 19 Feb 2025); (Xu et al., 31 Jul 2025)	SNN: 98.34% MNIST, 89.82% Fashion, 51.05% CIFAR-10, edge efficiency
Quantized edge FF-INT8	(Ma et al., 28 Jun 2025)	–27% memory, –8% energy, ≤0.4 pp accuracy loss
Hierarchical & contrastive learning	(Yao et al., 24 May 2026)	HCL-FF: +5.46% (CIFAR-10), +17.00% (CIFAR-100) over DeeperForward
Hyperspherical, prototype FF	(Sarode et al., 30 Apr 2026)	25.7% ImageNet-1k (HFF), $\mathcal{O}(1)$ inference

The Forward-Forward algorithm constitutes a versatile, scalable, and biologically plausible alternative to backpropagation. Ongoing research consistently improves its accuracy, efficiency, and interpretability across learning paradigms, solidifying it as a foundational methodology in next-generation neural computation.