---
title: Predictive Coding Networks (PCNs)
url: https://www.emergentmind.com/topics/predictive-coding-networks-pcns
type: topic
---

# Predictive Coding Networks (PCNs)

Searching arXiv for recent and foundational papers on predictive coding networks.
Predictive Coding Networks (PCNs) are neural architectures motivated by predictive coding in neuroscience, in which higher layers generate top-down predictions, lower layers compute prediction errors, and internal activities are iteratively adjusted so as to minimize a free-energy or squared-error objective. In machine learning, this label covers hierarchical generative models with explicit value and error units, convolutional networks with local recurrent feedback for object recognition, and more general graph-based formulations; in that broader sense, the same formalism can be used for classification, generation, and associative memory, and can be defined on arbitrary graph topologies [2202.09467].

## 1. Formal foundations

A standard mathematical presentation treats a PCN as a hierarchical generative model with activities \(\bar{x}_{l,t}\), predictions \(\bar{\mu}_{l,t}\), and prediction errors \(\bar{\varepsilon}_{l,t}\). For layers \(l=0,\dots,L-1\),
\[
\bar{\mu}_{l,t} = \bar{\theta}_{l+1} f(\bar{x}_{l+1,t}),
\qquad
\bar{\varepsilon}_{l,t} = \bar{x}_{l,t} - \bar{\theta}_{l+1} f(\bar{x}_{l+1,t}),
\]
and the global energy is
\[
\mathcal{F}_t = \frac{1}{2}\sum_l \|\bar{\varepsilon}_{l,t}\|^2.
\]
Inference performs gradient descent on \(\mathcal{F}_t\) with respect to latent activities, while learning updates the weights using the same objective [2202.09467].

A complementary formulation treats predictive coding as constrained optimization over both activations and weights. In a linear network, one introduces residuals on each edge, such as \(r_1 = h_1 - W_1 x\), \(r_2 = h_2 - W_2 h_1\), and \(r_3 = y - W_3 h_2\), and defines the PC energy as the sum of squared residuals. In that view, inference minimizes the energy over activations, learning updates weights by local gradients, and the overall procedure is naturally bi-level [2511.11092].

In supervised PCNs, input and target layers are clamped while hidden activities remain free. The standard training cycle therefore has two phases: an inference phase, which finds hidden activities that reduce total prediction error for a fixed set of weights, and a learning phase, which updates the weights at or near that inferred equilibrium. In the terminology of prospective configuration, inference “prospectively” adjusts hidden activities toward a configuration that satisfies the task constraint, after which local synaptic updates consolidate that configuration into the parameters [2207.12316].

## 2. Architectural realizations and recurrent predictive coding blocks

One influential architectural realization is the deep convolutional PCN for object recognition. In this design, the network is a stack of recurrent predictive-coding blocks placed between adjacent convolutional layers. Each block contains a feedforward convolution \(\mathbf{W}_{l-1,l}\), a feedback transposed convolution \(\mathbf{W}_{l,l-1}\), and a \(1\times 1\) bypass convolution \(\mathbf{W}_{l-1,l}^{bp}\). For a block between layers \(l-1\) and \(l\), top-down prediction and bottom-up error are written as
\[
\mathbf{p}_{l-1}(t) = (\mathbf{W}_{l,l-1})^T \mathbf{r}_l(t),
\qquad
\mathbf{e}_{l-1}(t) = \mathbf{r}_{l-1} - \mathbf{p}_{l-1}(t),
\]
with the implemented error typically passed through a ReLU:
\[
\mathbf{e}_{l-1}(t) = \text{ReLU}\left(\mathbf{r}_{l-1} - \mathbf{p}_{l-1}(t)\right).
\]
The higher-layer state is initialized by a standard feedforward convolution and then recurrently refined over \(T\) cycles [1805.07526].

The recurrent update in that convolutional PCN takes the form
\[
\mathbf{r}_l(t) = \mathbf{r}_l(t-1) + \alpha_l\,\text{FFConv}(\mathbf{e}_{l-1}(t)),
\]
where \(\alpha_l\) is a learnable, non-negative scalar defined per filter. After \(T\) cycles, the block output is
\[
\mathbf{r}_l = \mathbf{r}_l(T) + \text{BPConv}(\mathbf{r}_{l-1}^{BN}).
\]
This preserves the standard CNN components—convolutions, ReLUs, batch normalization, pooling, global average pooling, and a softmax classifier—but changes the computational dynamics by inserting local feedback loops between adjacent layers [1805.07526].

An earlier convolutional formulation used a VGG-style scaffold with 6–8 convolutional layers, 3\(\times\)3 kernels, ReLU nonlinearities, occasional 2\(\times\)2 max-pooling, and a global average pooling plus fully connected classifier. Feedback connections were implemented by transposed convolutions, with tied or untied weights; the tied case enforces \(\mathbf{W}_{l,l-1}=\mathbf{W}_{l-1,l}^T\), whereas the untied case learns feedback and feedforward weights independently. That paper emphasized a dynamical-systems view: when unfolded in time, a PCN becomes “deeper and deeper over time,” because the same architecture is reused across multiple cycles of recurrent computation [1802.04762].

## 3. Inference, local learning, and relations to backpropagation and target propagation

The defining operational feature of PCNs is the separation between fast activity dynamics and slower synaptic dynamics. Given clamped variables, hidden activities follow a local gradient flow of the form
\[
\frac{d \bar{x}_l}{dt} \propto -\frac{\partial \mathcal{F}_t}{\partial \bar{x}_{l,t}},
\]
and, at equilibrium, weights are updated by local rules such as
\[
\Delta \bar{\theta}_{l+1} \propto -\frac{\partial}{\partial \bar{\theta}_{l+1}} \frac{1}{2}\|\bar{\varepsilon}_l\|^2
= - \bar{\varepsilon}_l\, f(\bar{x}_{l+1})^\top.
\]
This is the canonical Hebbian-like three-factor form used in many predictive-coding accounts of learning [2202.09467].

The relation between PCNs and backpropagation is nuanced. One line of theory shows that, under various limits, predictive coding approximates or reproduces BP. A particularly strong result is Zero-Divergence Inference Learning (Z-IL), which, with \(\gamma=1\), predictive-state initialization, and layer-wise update scheduling, yields exact backpropagation not only in fully connected networks but also in convolutional neural networks and many-to-one recurrent neural networks. In that setting, the PCN weight updates \(\Delta \theta\) are exactly equal to the corresponding BP or BPTT updates \(\Delta w\) [2103.03725].

A different theoretical line studies PCNs in the standard predictive-coding regime, where they are not forced to mimic BP. There, prospective configuration is analyzed as a variant of generalized expectation-maximization, and the resulting theory proves convergence to critical points of the BP loss function. The same analysis establishes a close relationship to target propagation: in an input-unconstrained invertible setting, the equilibrium activities of the PCN coincide with target-propagation targets, whereas in the usual clamped-input setting the equilibrium represents a balance between feedforward consistency and feedback-imposed target structure [2207.12316].

A recurring misconception is that “predictive coding” always implies a single learning algorithm. The literature surveyed here instead contains several regimes: approximate BP, exact BP via Z-IL, prospective configuration that is not tuned to approximate BP, and deep convolutional object-recognition PCNs trained end-to-end with backpropagation through time. This suggests that PCNs are best understood as a family of energy-based or error-driven constructions sharing the same core prediction-error principle, rather than a single canonical training protocol.

## 4. Depth, parameterization, conditioning, and scaling

Deep PCNs have historically been difficult to train. A central diagnosis is that the inference landscape becomes poorly conditioned as width and depth increase. For linear PCNs, the activity Hessian \(H_{\mathbf{z}}\) is positive definite, so inference is a convex optimization problem, but its condition number \(\kappa(H_{\mathbf{z}})\) grows rapidly with depth and width, especially in residual architectures. Standard parameterizations also suffer from vanishing or exploding forward passes, which is particularly harmful because PC inference is typically initialized from a feedforward pass [2505.13124].

The \(\mu\)PC parameterization addresses the forward-scaling problem by importing Depth-\(\mu\)P ideas into predictive coding. In the residual-network setting studied there, \(\mu\)PC uses \(a_1=d_{\mathrm{in}}^{-1/2}\), \(a_\ell=(NL)^{-1/2}\) for hidden layers, \(a_L=N^{-1}\), and weights initialized as \(\mathcal{N}(0,1)\). With this parameterization, 100+ layer PCNs can be trained reliably; the paper reports stable training of residual networks up to 128 layers on simple classification tasks, competitive performance, and zero-shot transfer of both weight and activity learning rates across widths and depths [2505.13124].

A complementary asymptotic analysis shows that the set of width- and depth-stable feature-learning parameterizations for linear PCNs is exactly the same as for BP, and that in a regime where width \(N\) is much larger than depth \(L\), the equilibrated PC energy converges to the BP loss. Under those conditions, PC gradients converge to BP gradients. Experiments in nonlinear MLPs and residual networks show that the same phenomenon appears in practice as long as an activity equilibrium seems to be reached [2602.07697].

Several other scaling remedies target specific deep-PC pathologies. One study shows that classification accuracy in generative PCNs can peak and then deteriorate because layer-wise convergence speeds become imbalanced; the proposed remedies are regularizing the weight matrices at each layer by restricting the relative size of matrix singular values, or using a simpler biologically plausible scheme of weight capping [2208.07114]. Another introduces highway error propagation, which augments the free energy with feedback matrices \(V_{L\to i}\) that couple selected hidden states directly to the clamped output error; because this correction is linear in the hidden state, its magnitude is independent of depth, and the method trains MLPs of up to 128 layers with accuracy that is robust with respect to depth [2606.22744]. A different line of work replaces iterative relaxation with closed-form hierarchical Gaussian filtering, restores precision-weighted prediction errors, learns activations, weights, and precisions under a single free-energy objective, and reports FashionMNIST performance that approaches backpropagation in epoch-level wall-clock cost while converging in fewer epochs [2605.20293].

## 5. Empirical behavior and application regimes

The first wave of deep convolutional PCNs established that recurrent predictive coding could improve object recognition with relatively shallow architectures. On CIFAR-10 and CIFAR-100, the 9-layer PCN-A-6 with untied weights reached \(93.83\%\) and \(72.58\%\), compared to \(90.61\%\) and \(62.11\%\) for its plain feedforward counterpart, and the paper emphasized that the CIFAR-100 gain was \(+10.47\) percentage points over the plain CNN. On SVHN, PCN-D-6 (untied) achieved \(2.28\%\) best error, and on MNIST, PCN-E-4 (untied) achieved \(0.36\%\) error [1802.04762].

The later convolutional PCN with local recurrent processing extended this line to ImageNet and compared directly against classical and state-of-the-art CNNs. On CIFAR-100, PCN-D-5 obtained \(21.77\%\) error, improving over its plain 9-layer counterpart at \(25.31\%\). On SVHN, PCN-B-5 achieved \(1.96\%\) error. On ImageNet, PCN-E-5, with 13 layers and 17.26M parameters, achieved single-crop \(25.31\%\) top-1 and \(7.79\%\) top-5 error, outperforming ResNet-34 and coming close to ResNet-50 while being substantially shallower [1805.07526].

A consistent empirical signature across these object-recognition studies is that increasing the number of recurrent cycles \(T\) improves accuracy. In the 2018 convolutional PCN, performance improved from PCN-B-1 to PCN-B-5 on SVHN, from PCN-D-1 to PCN-D-5 on CIFAR-100, and steadily on ImageNet as \(T\) increased from 1 to 5. The earlier VGG-style PCN likewise reported monotonic accuracy gains with the number of cycles up to 6 [1805.07526].

Internal-state analyses supply an important interpretive dimension. In the local recurrent PCN, the \(L_2\) norm of prediction error \(\|\mathbf{e}_{l-1}(t)\|_2\) decreases over recurrent cycles for all intermediate layers, indicating convergence toward stable internal representations. The same work also shows that the recurrent updates, although defined only to reduce prediction error, tend to align with the negative gradient of the classification loss, and that final-layer prediction errors rescaled back to input space yield saliency maps that highlight regions containing the object or discriminative visual features [1805.07526].

## 6. Optimization variants, software, and implementation practice

Beyond the core predictive-coding loop, a substantial literature studies how inference and learning should be optimized in practice. PredProp is a stochastic optimization method for PCNs that preconditions both state and weight updates using covariances of propagated errors and activities. In that formulation, precision weighting is applied not to raw errors but to propagated gradients, yielding an approximate natural-gradient method. The paper reports that PredProp performs favorably over Adam in the tested configurations, and that optimization methods for weight parameters also benefit from using PredProp’s error precision during inference [2111.08792].

The software ecosystem has also become more explicit about the dynamical-systems character of PC inference. JPC is a JAX library for discriminative, generative, and hybrid predictive-coding models that formulates inference as an ordinary differential equation and integrates it with Diffrax solvers. In that implementation, the canonical energy is
\[
\mathcal{F} = \sum_{\ell=1}^L \left\| \mathbf{z}_\ell - f_\ell(W_\ell \mathbf{z}_{\ell-1}) \right\|^2,
\]
and inference solves the gradient-flow ODE \(d\mathbf{z}/dt = -\nabla_{\mathbf{z}} \mathcal{F}\). The library reports that a second-order Heun solver achieves significantly faster runtimes than standard Euler integration, with comparable performance on a range of tasks and network depths [2412.03676].

These optimization and tooling developments underscore a practical point: the performance of a PCN is determined not only by the free-energy objective, but also by the numerical method used for activity equilibration, the conditioning of the activity landscape, and the way precision or curvature information is incorporated. This suggests that implementation details are not peripheral in PCNs; they are part of the effective learning algorithm.

## 7. Topological generalizations, mathematical diagnostics, and unresolved issues

A major conceptual extension is the move from layered PCNs to predictive coding graphs (PCGs). In a PCG, one defines nodes \(a_\alpha\), predictions \(\mu_\alpha = f(\sum_\beta w_{\alpha\beta} a_\beta)\), errors \(\epsilon_\alpha = a_\alpha - \mu_\alpha\), and energy \(E_G = \sum_\alpha \epsilon_\alpha^2\) on an arbitrary graph. With a specific block-sparse choice of the weight matrix, the PCG reduces exactly to a layered PCN; during testing, that layered PCN is equivalent to a standard feedforward neural network. In this precise sense, predictive coding graphs are a mathematical superset of feedforward artificial neural networks [2603.06142].

A different mathematical extension recasts linear PCNs as cellular sheaves on their computational graphs. In that framework, activations are \(0\)-cochains, edge-wise prediction errors are \(1\)-cochains, the coboundary operator maps activations to errors, and PC inference becomes diffusion under the sheaf Laplacian \(L = (\delta^0)^\top \delta^0\). Sheaf cohomology then characterizes irreducible error patterns that inference cannot remove, especially in recurrent topologies where feedback loops induce contradictions. The paper’s Hodge decomposition identifies when such internal contradictions concentrate harmonic error in regions that diffusion barely reaches, causing learning to stall [2511.11092].

Several controversies remain active. One concerns biological plausibility. Many PCN formulations emphasize local learning, no global error signal, and explicit error units, but deep object-recognition PCNs were trained with backpropagation through time through the unrolled recurrent computation [1805.07526]. Exact BP implementations such as Z-IL are biologically plausible by the standards adopted in that paper, yet they also require specific scheduling and initialization conditions [2103.03725]. Closed-form hierarchical Gaussian filtering removes iterative relaxation and automatic differentiation, but it changes the effective algorithmic regime by restoring precision-weighted message passing and analytic posterior updates [2605.20293]. These differences do not negate the predictive-coding principle; they show that the term “biologically plausible PCN” is itself model-dependent.

Another persistent issue is computational cost. Iterative inference can be substantially more expensive than a single forward pass, especially in large or ill-conditioned models, and several scaling papers explicitly frame hardware co-design as a necessary future direction. Open technical questions include how to control conditioning in deep recurrent PCNs, how to guarantee benign cohomology or monodromy in recurrent feedback loops, how best to learn or constrain precisions, and how to extend the current scaling results from fully connected and residual architectures to convolutional networks and transformers.

Source: https://www.emergentmind.com/topics/predictive-coding-networks-pcns