Highway Error Propagation (HEP) in Deep PCNs

• Highway Error Propagation is a bio-plausible modification of predictive coding that adds long-range error highways to mitigate signal attenuation in deep networks.
• HEP alters the free energy formulation by integrating a direct error injection term while preserving local synaptic plasticity.
• Empirical results demonstrate that HEP enables deep multi-layer networks to achieve performance comparable to backpropagation, even where vanilla predictive coding fails.

Searching arXiv for the specified paper and closely related predictive coding literature. First, retrieve the target paper by arXiv id. Now, search for related predictive coding papers for minimal contextual grounding. Highway Error Propagation (HEP) is a simple, bio-plausible augmentation of vanilla predictive coding (PC) for very deep predictive coding networks (PCNs). It equips PCNs with long-range feedback “highways” that directly inject the output error into selected hidden layers so that the learning signal does not vanish with depth. In the formulation introduced in "Error Highways: Scaling Predictive Coding to Very Deep Networks" (Mohammadi et al., 22 Jun 2026), HEP modifies the free energy used during PC inference while leaving the local synaptic update rule unchanged. The stated motivation is a central obstacle in deep PCNs: in vanilla PC, the learning signal decays rapidly as it propagates away from clamped boundaries, leaving interior layers effectively unchanged; HEP counteracts this by coupling selected hidden states directly to the clamped output error.

1. Predictive coding setting and the depth-scaling problem

In the supervised PCN considered for HEP, the activity or state of layer $i$ is denoted by $z_i$, with $z_0$ clamped to the input $\mathbf{o}$ and $z_L$ clamped to the one-hot target $\mathbf{y}$. Each layer predicts the next layer through

$\hat z_i = f_{i-1}(z_{i-1}),\quad i=1,\dots,L.$

Vanilla PC is organized around a free-energy objective over the free hidden states $z_1,\dots,z_{L-1}$: $$E(z)\;=\;\sum_{i=1}^{L-1}\tfrac12\bigl\lVert z_i - f_{i-1}(z_{i-1})\bigr\rVert^2\;+\;\ell\!\bigl(z_L,\mathbf{y}\bigr), \tag{1}$$ where $\ell$ is, e.g., cross-entropy. Layerwise prediction errors are defined as

$z_i$0

The scaling problem addressed by HEP is specific: the output-derived learning signal attenuates as it traverses many Jacobians. The paper identifies this attenuation as the reason PCNs have remained largely confined to shallow architectures and simple machine intelligence benchmarks. In that sense, HEP is not a replacement for predictive coding, but an intervention on the error-transport path inside the predictive-coding inference dynamics (Mohammadi et al., 22 Jun 2026).

2. Vanilla predictive coding dynamics and local synaptic plasticity

Vanilla PC performs iterative inference, described as the E-step, by gradient descent on $z_i$1. Writing $z_i$2, each free state updates according to

$z_i$3

This expression shows that the correction to $z_i$4 depends on a local prediction error term $z_i$5 and a top-down term $z_i$6. In deep networks, the latter is the route through which output supervision reaches earlier hidden layers.

Once inference converges, the local Hebbian M-step updates each weight matrix $z_i$7 by

$z_i$8

A key property emphasized in the HEP formulation is locality: the update depends only on $z_i$9 and $z_0$0. This locality underlies PC’s bio-plausibility. HEP is designed so that this locality is preserved exactly, even though the inference dynamics are modified. A common misunderstanding is that deep-error injection would necessarily alter the synaptic learning rule; in the HEP construction, it does not.

3. HEP as an augmented free-energy objective

HEP adds error highways from the output error into a selected subset of hidden layers $z_0$1. The mechanism is implemented by sampling once and fixing random matrices

$z_0$2

with entries $z_0$3, together with a single scalar strength $z_0$4.

The augmented free energy is

$z_0$5

where $z_0$6 is treated as a constant, or stop-gradient, so that the highway term only influences $z_0$7 and never feeds back into $z_0$8 (Mohammadi et al., 22 Jun 2026).

Two structural features are central. First, the highway pathway is linear in the hidden state. Second, the output error is coupled directly to selected hidden states, rather than being transmitted only through the intervening Jacobian chain. This suggests that HEP should be understood as a change to the neural structure of inference rather than a change to the form of local synaptic plasticity.

4. Inference under HEP and the Jacobian bypass

Gradient descent on $z_0$9 yields the hidden-state update

$\mathbf{o}$0

Accordingly, every endpoint $\mathbf{o}$1 receives at each inference step a direct nudge of magnitude $\mathbf{o}$2, bypassing all intermediate Jacobians. Hidden layers not in $\mathbf{o}$3 evolve as in vanilla PC.

Despite this modification to inference, the M-step remains unchanged: $\mathbf{o}$4

The paper characterizes this as preserving full locality of synaptic updates while bypassing the long Jacobian chain for the output error. One implication, stated directly in the formulation, is that HEP alters the path by which supervision reaches hidden states but does not alter the fact that feedforward synapses are updated using only layer-local quantities. This distinguishes HEP from modifications that would embed nonlocal terms directly into $\mathbf{o}$5.

5. Error attenuation analysis

The theoretical argument for HEP focuses on why vanilla PC’s supervisory signal vanishes and why HEP’s does not. At steady state, PC reproduces backprop errors: $\mathbf{o}$6

If each Jacobian has spectral norm $\mathbf{o}$7, then $\mathbf{o}$8 decays geometrically in $\mathbf{o}$9. The depth dependence is therefore explicit in the product of Jacobians. Under this condition, deeper hidden layers receive exponentially attenuated error signals.

The paper also analyzes transient or early-step PC under a feedforward warm start. In that regime, $z_L$0 crawls back one layer per inference step and picks up a factor of the activity step size $z_L$1 at each hop. The result is

$z_L$2

HEP contrasts with both cases by delivering an $z_L$3 correction to every chosen layer in one hop, so the highway signal’s magnitude is independent of depth (Mohammadi et al., 22 Jun 2026). This is the core sense in which HEP eliminates depth-dependent attenuation: it does not rely on repeated multiplication by intermediate Jacobians in order to expose a selected hidden layer to the output error.

A frequent misconception is that HEP merely accelerates the same recursive error transport already present in vanilla PC. The formulation is stronger than that. It explicitly bypasses the Jacobian chain for the highway-delivered term, while the ordinary PC term remains intact.

6. Algorithmic description and empirical behavior in deep MLP PCNs

The high-level training procedure is summarized as Train-HEP-PCN. For each batch, the network clamps $z_L$4 and $z_L$5, initializes hidden states by one feedforward pass, performs $z_L$6 inference steps using the HEP-augmented hidden-state update, and then applies synaptic plasticity through the unchanged local M-step. In pseudocode form, the inference stage computes $z_L$7 for $z_L$8, computes $z_L$9, forms

$\mathbf{y}$0

adds $\mathbf{y}$1 when $\mathbf{y}$2, and updates $\mathbf{y}$3. The M-step then updates $\mathbf{y}$4 and $\mathbf{y}$5 using the same local quantities as vanilla PC.

Empirically, the paper trains thin MLP PCNs with width $\mathbf{y}$6, ReLU, and RMS-norm on MNIST and Fashion-MNIST at depths $\mathbf{y}$7, comparing vanilla PC ($\mathbf{y}$8), HEP, and skip-free backprop (BP). The representative MNIST test accuracies are as follows (Mohammadi et al., 22 Jun 2026):

Depth	Method	Test accuracy
4	vanilla PC / HEP / BP	$\mathbf{y}$9 / $\hat z_i = f_{i-1}(z_{i-1}),\quad i=1,\dots,L.$0 / $\hat z_i = f_{i-1}(z_{i-1}),\quad i=1,\dots,L.$1
8	vanilla PC / HEP / BP	$\hat z_i = f_{i-1}(z_{i-1}),\quad i=1,\dots,L.$2 / $\hat z_i = f_{i-1}(z_{i-1}),\quad i=1,\dots,L.$3 / $\hat z_i = f_{i-1}(z_{i-1}),\quad i=1,\dots,L.$4
16	vanilla PC / HEP / BP	$\hat z_i = f_{i-1}(z_{i-1}),\quad i=1,\dots,L.$5 / $\hat z_i = f_{i-1}(z_{i-1}),\quad i=1,\dots,L.$6 / $\hat z_i = f_{i-1}(z_{i-1}),\quad i=1,\dots,L.$7
32	vanilla PC / HEP / BP	$\hat z_i = f_{i-1}(z_{i-1}),\quad i=1,\dots,L.$8 / $\hat z_i = f_{i-1}(z_{i-1}),\quad i=1,\dots,L.$9 / $z_1,\dots,z_{L-1}$0
64	vanilla PC / HEP / BP	$z_1,\dots,z_{L-1}$1 / $z_1,\dots,z_{L-1}$2 / $z_1,\dots,z_{L-1}$3
128	vanilla PC / HEP / BP	$z_1,\dots,z_{L-1}$4 / $z_1,\dots,z_{L-1}$5 / $z_1,\dots,z_{L-1}$6

Fashion-MNIST is reported to show the same pattern: vanilla PC collapses beyond depth $z_1,\dots,z_{L-1}$7, but HEP remains above $z_1,\dots,z_{L-1}$8 to $z_1,\dots,z_{L-1}$9 even at $$E(z)\;=\;\sum_{i=1}^{L-1}\tfrac12\bigl\lVert z_i - f_{i-1}(z_{i-1})\bigr\rVert^2\;+\;\ell\!\bigl(z_L,\mathbf{y}\bigr), \tag{1}$$0 layers, closely matching BP on this skip-free architecture. A sweep of $$E(z)\;=\;\sum_{i=1}^{L-1}\tfrac12\bigl\lVert z_i - f_{i-1}(z_{i-1})\bigr\rVert^2\;+\;\ell\!\bigl(z_L,\mathbf{y}\bigr), \tag{1}$$1 at $$E(z)\;=\;\sum_{i=1}^{L-1}\tfrac12\bigl\lVert z_i - f_{i-1}(z_{i-1})\bigr\rVert^2\;+\;\ell\!\bigl(z_L,\mathbf{y}\bigr), \tag{1}$$2 shows a sharp transition from chance to full performance as $$E(z)\;=\;\sum_{i=1}^{L-1}\tfrac12\bigl\lVert z_i - f_{i-1}(z_{i-1})\bigr\rVert^2\;+\;\ell\!\bigl(z_L,\mathbf{y}\bigr), \tag{1}$$3 grows from $$E(z)\;=\;\sum_{i=1}^{L-1}\tfrac12\bigl\lVert z_i - f_{i-1}(z_{i-1})\bigr\rVert^2\;+\;\ell\!\bigl(z_L,\mathbf{y}\bigr), \tag{1}$$4 to $$E(z)\;=\;\sum_{i=1}^{L-1}\tfrac12\bigl\lVert z_i - f_{i-1}(z_{i-1})\bigr\rVert^2\;+\;\ell\!\bigl(z_L,\mathbf{y}\bigr), \tag{1}$$5, which the paper interprets as confirmation that the highway term alone enables deep learning. The experiments also report that thinning highways, by connecting only every $$E(z)\;=\;\sum_{i=1}^{L-1}\tfrac12\bigl\lVert z_i - f_{i-1}(z_{i-1})\bigr\rVert^2\;+\;\ell\!\bigl(z_L,\mathbf{y}\bigr), \tag{1}$$6th layer, still trains well up to moderate $$E(z)\;=\;\sum_{i=1}^{L-1}\tfrac12\bigl\lVert z_i - f_{i-1}(z_{i-1})\bigr\rVert^2\;+\;\ell\!\bigl(z_L,\mathbf{y}\bigr), \tag{1}$$7, indicating that even sparse highway coverage suffices.

These results address two possible objections. First, HEP is not presented as depending on forward skip connections: the paper states that vanilla PCNs with no forward skip connections can be trained with HEP. Second, the method does not appear to require a highway into every hidden layer in order to be effective; sparse coverage can suffice. A plausible implication is that the decisive ingredient is not dense auxiliary connectivity per se, but direct access of hidden layers to a depth-independent supervisory term.