---
title: Highway Error Propagation (HEP) in Deep PCNs
url: https://www.emergentmind.com/topics/highway-error-propagation-hep
type: topic
---

# Highway Error Propagation (HEP) in Deep PCNs

Searching arXiv for the specified paper and closely related predictive coding literature.
First, retrieve the target paper by arXiv id.
Now, search for related predictive coding papers for minimal contextual grounding.
Highway Error Propagation (HEP) is a simple, bio-plausible augmentation of vanilla predictive coding (PC) for very deep predictive coding networks (PCNs). It equips PCNs with long-range feedback “highways” that directly inject the output error into selected hidden layers so that the learning signal does not vanish with depth. In the formulation introduced in "Error Highways: Scaling Predictive Coding to Very Deep Networks" [2606.22744], HEP modifies the free energy used during PC inference while leaving the local synaptic update rule unchanged. The stated motivation is a central obstacle in deep PCNs: in vanilla PC, the learning signal decays rapidly as it propagates away from clamped boundaries, leaving interior layers effectively unchanged; HEP counteracts this by coupling selected hidden states directly to the clamped output error.

## 1. Predictive coding setting and the depth-scaling problem

In the supervised PCN considered for HEP, the activity or state of layer \(i\) is denoted by \(z_i\), with \(z_0\) clamped to the input \(\mathbf{o}\) and \(z_L\) clamped to the one-hot target \(\mathbf{y}\). Each layer predicts the next layer through
\[
\hat z_i = f_{i-1}(z_{i-1}),\quad i=1,\dots,L.
\]

Vanilla PC is organized around a free-energy objective over the free hidden states \(z_1,\dots,z_{L-1}\):
\[
E(z)\;=\;\sum_{i=1}^{L-1}\tfrac12\bigl\lVert z_i - f_{i-1}(z_{i-1})\bigr\rVert^2\;+\;\ell\!\bigl(z_L,\mathbf{y}\bigr),
\tag{1}
\]
where \(\ell\) is, e.g., cross-entropy. Layerwise prediction errors are defined as
\[
\varepsilon_i \;=\; z_i \;-\; f_{i-1}(z_{i-1}),\quad i=1,\dots,L-1,
\qquad
\varepsilon_L \;=\;-\,\nabla_{z_L}\,\ell(z_L,\mathbf{y}).
\]

The scaling problem addressed by HEP is specific: the output-derived learning signal attenuates as it traverses many Jacobians. The paper identifies this attenuation as the reason PCNs have remained largely confined to shallow architectures and simple machine intelligence benchmarks. In that sense, HEP is not a replacement for predictive coding, but an intervention on the error-transport path inside the predictive-coding inference dynamics [2606.22744].

## 2. Vanilla predictive coding dynamics and local synaptic plasticity

Vanilla PC performs iterative inference, described as the E-step, by gradient descent on \(E\). Writing \(J_i=\partial f_i/\partial z_i\), each free state updates according to
\[
z_i^{(t+1)}
= z_i^{(t)}
\;-\;\frac{\partial E}{\partial z_i}
\;=\;
z_i^{(t)}
\;-\;
\bigl(\varepsilon_i^{(t)} - J_i^{(t)\top}\,\varepsilon_{i+1}^{(t)}\bigr).
\tag{2}
\]

This expression shows that the correction to \(z_i\) depends on a local prediction error term \(\varepsilon_i\) and a top-down term \(J_i^\top \varepsilon_{i+1}\). In deep networks, the latter is the route through which output supervision reaches earlier hidden layers.

Once inference converges, the local Hebbian M-step updates each weight matrix \(W_i\) by
\[
\Delta W_i
\;\propto\;
\bigl(\varepsilon_{i+1}\odot f'_{i}(W_i\,z_i+b_i)\bigr)\,z_i^{\top}.
\tag{3}
\]

A key property emphasized in the HEP formulation is locality: the update depends only on \(\varepsilon_{i+1}\) and \(z_i\). This locality underlies PC’s bio-plausibility. HEP is designed so that this locality is preserved exactly, even though the inference dynamics are modified. A common misunderstanding is that deep-error injection would necessarily alter the synaptic learning rule; in the HEP construction, it does not.

## 3. HEP as an augmented free-energy objective

HEP adds error highways from the output error into a selected subset of hidden layers \(S\subset\{1,\dots,L-1\}\). The mechanism is implemented by sampling once and fixing random matrices
\[
\{V_{L\to i}\in\mathbb{R}^{n_i\times n_L}\mid i\in S\}
\]
with entries \(\sim\mathcal{N}(0,\sigma_v^2)\), together with a single scalar strength \(\alpha>0\).

The augmented free energy is
\[
\widetilde F(z)
\;=\;
E(z)
\;+\;
\alpha\;\sum_{i\in S}\;z_i^{\!\top}\,V_{L\to i}\;\mathrm{sg}\bigl(\varepsilon_L\bigr),
\tag{4}
\]
where \(\mathrm{sg}(\varepsilon_L)\) is treated as a constant, or stop-gradient, so that the highway term only influences \(z_i\) and never feeds back into \(\varepsilon_L\) [2606.22744].

Two structural features are central. First, the highway pathway is linear in the hidden state. Second, the output error is coupled directly to selected hidden states, rather than being transmitted only through the intervening Jacobian chain. This suggests that HEP should be understood as a change to the neural structure of inference rather than a change to the form of local synaptic plasticity.

## 4. Inference under HEP and the Jacobian bypass

Gradient descent on \(\widetilde F\) yields the hidden-state update
\[
z_i^{(t+1)}
= z_i^{(t)}
\;-\;
\frac{\partial \widetilde F}{\partial z_i}
\;=\;
z_i^{(t)}
\;-\;
\bigl(\varepsilon_i^{(t)} - J_i^{(t)\top}\,\varepsilon_{i+1}^{(t)}\bigr)
\;-\;
\alpha\,V_{L\to i}\,\mathrm{sg}\bigl(\varepsilon_L^{(t)}\bigr)\;\mathbb{I}[i\in S].
\tag{5}
\]

Accordingly, every endpoint \(i\in S\) receives at each inference step a direct nudge of magnitude \(\alpha\,V_{L\to i}\,\varepsilon_L\), bypassing all intermediate Jacobians. Hidden layers not in \(S\) evolve as in vanilla PC.

Despite this modification to inference, the M-step remains unchanged:
\[
\Delta W_i
\;=\;
-\eta_W\,\frac{\partial \widetilde F}{\partial W_i}
\;=\;
-\eta_W\,\frac{\partial E}{\partial W_i}
\;\propto\;
\bigl(\varepsilon_{i+1}\odot f'_i(W_i z_i+b_i)\bigr)\,z_i^{\top}.
\tag{6}
\]

The paper characterizes this as preserving full locality of synaptic updates while bypassing the long Jacobian chain for the output error. One implication, stated directly in the formulation, is that HEP alters the path by which supervision reaches hidden states but does not alter the fact that feedforward synapses are updated using only layer-local quantities. This distinguishes HEP from modifications that would embed nonlocal terms directly into \(\Delta W_i\).

## 5. Error attenuation analysis

The theoretical argument for HEP focuses on why vanilla PC’s supervisory signal vanishes and why HEP’s does not. At steady state, PC reproduces backprop errors:
\[
\varepsilon_i
= J_i^{\top}\,\varepsilon_{i+1}
= \Bigl(\prod_{k=i}^{L-1}J_k\Bigr)^{\!\top}\,\varepsilon_L.
\tag{7}
\]

If each Jacobian has spectral norm \(\|J_k\|<1\), then \(\|\varepsilon_i\|\) decays geometrically in \(L-i\). The depth dependence is therefore explicit in the product of Jacobians. Under this condition, deeper hidden layers receive exponentially attenuated error signals.

The paper also analyzes transient or early-step PC under a feedforward warm start. In that regime, \(\varepsilon_L\) crawls back one layer per inference step and picks up a factor of the activity step size \(\lambda\ll1\) at each hop. The result is
\[
\Delta z_i
= \mathcal{O}\bigl(\lambda^{\,L-i}\bigr)\,\varepsilon_L
\quad\Longrightarrow\quad
\text{no learning signal reaches deep layers within finite \(T\).}
\tag{8}
\]

HEP contrasts with both cases by delivering an \(\mathcal{O}(\alpha)\) correction to every chosen layer in one hop, so the highway signal’s magnitude is independent of depth [2606.22744]. This is the core sense in which HEP eliminates depth-dependent attenuation: it does not rely on repeated multiplication by intermediate Jacobians in order to expose a selected hidden layer to the output error.

A frequent misconception is that HEP merely accelerates the same recursive error transport already present in vanilla PC. The formulation is stronger than that. It explicitly bypasses the Jacobian chain for the highway-delivered term, while the ordinary PC term remains intact.

## 6. Algorithmic description and empirical behavior in deep MLP PCNs

The high-level training procedure is summarized as Train-HEP-PCN. For each batch, the network clamps \(z_0\leftarrow o\) and \(z_L\leftarrow y\), initializes hidden states by one feedforward pass, performs \(T\) inference steps using the HEP-augmented hidden-state update, and then applies synaptic plasticity through the unchanged local M-step. In pseudocode form, the inference stage computes \(\varepsilon_i = z_i - f_{i-1}(z_{i-1})\) for \(i=1\dots L-1\), computes \(\varepsilon_L = -\nabla_{z_L}\ell(z_L,y)\), forms
\[
\zeta_i \leftarrow \varepsilon_i - J_i^\top \varepsilon_{i+1},
\]
adds \(\alpha\cdot V_{L\to i}\cdot \mathrm{sg}(\varepsilon_L)\) when \(i\in S\), and updates \(z_i \leftarrow z_i - \eta_z\cdot \zeta_i\). The M-step then updates \(W_i\) and \(b_i\) using the same local quantities as vanilla PC.

Empirically, the paper trains thin MLP PCNs with width \(128\), ReLU, and RMS-norm on MNIST and Fashion-MNIST at depths \(L\in\{4,8,16,32,64,128\}\), comparing vanilla PC (\(\alpha=0\)), HEP, and skip-free backprop (BP). The representative MNIST test accuracies are as follows [2606.22744]:

| Depth | Method | Test accuracy |
|---|---|---|
| 4 | vanilla PC / HEP / BP | \(97.8\pm0.1\%\) / \(97.8\pm0.3\%\) / \(98.0\pm0.1\%\) |
| 8 | vanilla PC / HEP / BP | \(92.1\pm1.0\%\) / \(96.4\pm0.7\%\) / \(97.8\pm0.1\%\) |
| 16 | vanilla PC / HEP / BP | \(35.5\pm11.2\%\) / \(96.4\pm0.2\%\) / \(97.7\pm0.0\%\) |
| 32 | vanilla PC / HEP / BP | \(13.4\pm2.2\%\) / \(95.6\pm0.3\%\) / \(97.1\pm0.1\%\) |
| 64 | vanilla PC / HEP / BP | \(12.5\pm2.3\%\) / \(95.9\pm0.5\%\) / \(94.9\pm0.2\%\) |
| 128 | vanilla PC / HEP / BP | \(12.8\pm2.5\%\) / \(95.8\pm0.6\%\) / \(95.0\pm0.1\%\) |

Fashion-MNIST is reported to show the same pattern: vanilla PC collapses beyond depth \(16\), but HEP remains above \(82\%\) to \(86\%\) even at \(128\) layers, closely matching BP on this skip-free architecture. A sweep of \(\alpha\) at \(L=64\) shows a sharp transition from chance to full performance as \(\alpha\) grows from \(10^{-3}\) to \(0.1\), which the paper interprets as confirmation that the highway term alone enables deep learning. The experiments also report that thinning highways, by connecting only every \(k\)th layer, still trains well up to moderate \(k\), indicating that even sparse highway coverage suffices.

These results address two possible objections. First, HEP is not presented as depending on forward skip connections: the paper states that vanilla PCNs with no forward skip connections can be trained with HEP. Second, the method does not appear to require a highway into every hidden layer in order to be effective; sparse coverage can suffice. A plausible implication is that the decisive ingredient is not dense auxiliary connectivity per se, but direct access of hidden layers to a depth-independent supervisory term.

Source: https://www.emergentmind.com/topics/highway-error-propagation-hep