Error Highways: Scaling Predictive Coding to Very Deep Networks

Abstract: Predictive coding networks (PCNs) offer a biologically-plausible, local-learning alternative to back-propagation of errors (backprop). Nevertheless, they have remained largely confined to shallow architectures and evaluated on simple machine intelligence benchmarks. A central obstacle to scaling PCNs is that the learning signal decays rapidly as it propagates away from the clamped boundaries, leaving interior layers effectively unchanged. To directly counter this problem, we propose highway error propagation (HEP), a scheme that augments the free energy function underlying predictive coding (PC) by altering its neural structure with feedback matrices $V_{L\to i}$ that couple selected hidden states directly to the clamped output error. Since this coupling is linear in the hidden state, the highway pathway delivers a correction at every inference step whose magnitude is independent of depth, in contrast to vanilla PC where the output error reaches the $i$-th hidden layer with attenuation that decays exponentially in depth. This bypasses the Jacobian chain while preserving the local PC synaptic update rule. On MNIST and Fashion-MNIST, we show that HEP effectively trains MLPs of up to 128 layers with accuracy that is robust with respect to depth.

• The paper introduces highway error propagation (HEP) to overcome the vanishing learning signals in deep predictive coding networks.
• HEP injects output error directly into selected hidden layers using fixed feedback matrices, bypassing the exponential decay in standard PCN inference.
• Empirical tests on MLPs up to 128 layers show that HEP maintains robust accuracy on MNIST and Fashion-MNIST, matching or surpassing BP performance.

Highway Error Propagation: Scaling Predictive Coding in Deep Networks

Introduction

The paper "Error Highways: Scaling Predictive Coding to Very Deep Networks" (2606.22744) addresses the primary bottleneck hindering the application of predictive coding networks (PCNs) to deep architectures. PCNs are more biologically plausible than backpropagation (BP), featuring local learning updates suitable for neuroAI implementations. However, their ability to scale to deep networks has been restricted by rapid decay of learning signals as depth increases, resulting in ineffective training of interior layers far from the clamped output. The authors introduce highway error propagation (HEP), which directly injects output error into hidden layers through dedicated feedback matrices, bypassing the exponential attenuation suffered by vanilla PC inference.

Theoretical Basis and Depth Scaling Problem

Predictive coding networks operate via message-passing mechanisms where each layer predicts the activity of the layer below and minimizes prediction errors using local updates. The standard formulation clamps input and output, initializing free interior states via feedforward projection, and then employs gradient descent on a free energy functional comprising prediction and supervisory errors. At equilibrium, PCNs approximate BP error signals through a product of Jacobians but suffer geometric signal decay when the spectral norm of these Jacobians is $<1$. Transient analysis reveals that with feedforward initialization, the supervisory error at the output must traverse each intermediate layer in sequential inference steps, attenuating multiplicatively by the activity step size $\lambda$ per hop, leading to an exponentially vanishing learning signal for deep layers.

Figure 1: Per-layer RMS prediction error versus layer and inference iteration in a depth-64 MLP; vanilla PC and $\mu$PC stall at the input-side layers, while HEP injects error directly into hidden layers.

This phenomenon manifests empirically as negligible error in early layers after typical inference budgets, leaving these layers effectively unchanged during training—a challenge analogous to the vanishing gradient problem in BP-based networks.

Existing Remedies and Motivation for HEP

Previous attempts to ameliorate depth scaling in PCNs include optimization modifications (ePC, Meta-PCN) and architectural changes (muPC, DFA, LRA, DKP-PC). Optimization-based methods often violate locality or biological plausibility, whereas muPC stabilizes forward propagation via depth-aware weight parameterization and skip connections, but does not solve the systemic decay of the learning signal. Feedback alignment approaches such as DFA provide output-to-hidden pathways in non-PCN settings, and DKP-PC employs a one-off feedback injection pre-inference; however, signal delivery remains transient or only shallow models are demonstrated.

The question persists: can we construct a scheme activating all hidden layers with robust learning signals, without forward skip connections, and maintaining local update dynamics? HEP is devised precisely to answer this in a manner congruent with neurobiological architectures—emulating the heterarchical, recurrent connectivity observed in mammalian brains.

Highway Error Propagation (HEP): Architecture and Free Energy Formulation

HEP augments the PCN with a set $S$ of endpoints (selected hidden layers), each receiving a synaptic feedback matrix $V_{L\to i}$ from the output error. These matrices are fixed, randomly initialized, and the highway strength coefficient $\alpha$ modulates the signal injected every inference step. The correction delivered to each endpoint is independent of network depth and does not propagate through a chain of Jacobians and step-size attenuation.

Figure 2: Error highways transmit output error straight to selected hidden layers in a single hop, preserving local updates while preventing exponential signal decay.

The augmented free energy functional is:

$$\tilde{F}(z) = E(z) + \alpha \sum_{i \in S} z_{i}^\top V_{L\to i} \, \mathrm{sg}(e_L),$$

where $E(z)$ is the standard PC energy. Updates to $z_i$ in inference add the highway nudge; weight updates remain standard local Hebbian updates, preserving biological locality.

Normalization and Forward Pass Stabilization

Deep PCNs require activity normalization to prevent instability in the forward pass; the paper adopts RMS norm prior to projection at each layer. This step ensures consistent signal scales across depth, bounding prediction errors and weight updates. RMS normalization does not ameliorate the decaying error signal but is essential for stable training when combined with HEP.

Experimental Evaluation

The authors evaluate HEP in skip-free MLP architectures of up to 128 layers on MNIST and Fashion-MNIST. Accuracy traces a sharp contrast: vanilla PC achieves chance performance beyond ~16 layers, while HEP maintains robust accuracy comparable to BP, even as depth increases out to 128 layers. Notably, BP also declines with depth due to vanishing gradients in this skip-free regime. HEP's efficacy is uniquely attributable to the highway term, as ablation studies varying $\alpha$ demonstrate accuracy transitions from collapse (low $\lambda$0) to stability (moderate $\lambda$1) and deterioration (high $\lambda$2).

Figure 3: Test accuracy on MNIST at depth 64 as a function of highway strength $\lambda$3, showing that moderate values are required for effective deep training.

Empirical evidence further shows that error highways activate interior layers promptly in inference—even the deepest hidden layers receive meaningful error signals in the first few steps, sharply contrasting with the gradual crawl observed in vanilla PC and muPC settings.

Highway density ablations indicate that sparse connectivity suffices; injecting the output error into every $\lambda$4-th layer ($\lambda$5 up to 10) still yields high accuracy, though too wide gaps eventually restore attenuation effects and instability. Adam optimizer is shown to be critical in deep settings, compensating for ill-conditioned inference landscapes.

Figure 4: RMS prediction error across layers and inference iterations for depth-64, comparing optimizer behavior.

Figure 5: Energy convergence of selected layers under plain gradient descent, showing only output-adjacent layers settle within the inference budget.

Figure 6: Energy convergence of the same layers under Adam; all layers activate early and stabilize within the budget.

Implications and Future Directions

HEP enables predictive coding networks to scale to depths previously inaccessible, preserving biologically plausible, local learning rules without relying on architectural skip connections. This bridging of scalability and locality fills a crucial gap in neuroAI and biologically-motivated credit assignment research. The demonstrated stability and performance suggest that HEP may serve as a viable foundation for deep learning paradigms eschewing global error propagation, potentially useful for continual and online learning in neuromorphic or edge compute environments.

Future avenues include integration of HEP with convolutional structures, adaptive endpoint selection strategies, synergy with existing normalization innovations (e.g., batch norm variants), and exploration in larger-scale and more complex datasets. Extension to recurrent, generative, and heterarchical predictive coding models may further elucidate the constraints and advantages of this approach vis-à-vis latent representation learning and task transferability.

Conclusion

The introduction of highway error propagation establishes a critical mechanism for overcoming vanishing error signals in deep predictive coding networks. HEP delivers supervisory signals directly into hidden layers, maintaining local update dynamics and stabilizing accuracy across increased depths. Empirical results robustly support the effectiveness of this approach relative to previous PCN architectures and BP, and its theoretical and practical implications set a promising direction for neurobiologically-inspired deep learning models.