---
title: 'Predictive Coding Graphs: Theory & Applications'
url: https://www.emergentmind.com/topics/predictive-coding-graphs-pcgs
type: topic
---

# Predictive Coding Graphs: Theory & Applications

Searching arXiv for recent papers on predictive coding graphs and closely related formulations.
Predictive Coding Graphs (PCGs) are predictive-coding models defined on graph-structured computation or dependency architectures rather than on strictly layered hierarchies. In this usage, a PCG augments each node with a value or state variable, a prediction generated from parent nodes, and a local prediction error, and performs inference and learning by minimizing a global energy or variational free-energy objective through local message passing [2201.13180; 2006.04182; 2603.06142]. Across recent formulations, PCGs are used to extend predictive coding to arbitrary directed graph topologies, including skip connections, lateral connections, multi-parent nodes, and recurrent or cyclic structures, while preserving the characteristic predictive-coding separation between an inference phase on activities and a learning phase on parameters [2201.13180; 2212.04656; 2603.06142]. The term “PCG” is also used in graph theory for pairwise compatibility graphs; that is a distinct literature with unrelated definitions and witness-tree criteria [1707.07436; 2202.13844].

## 1. Definition and formalism

In the arbitrary-topology predictive-coding formulation, a PCG is a directed graph $G=(V,E)$ whose vertices carry state variables and prediction errors, and whose edges encode parametric dependencies used to generate local predictions [2201.13180; 2006.04182]. A representative definition assigns to each vertex $i$ a value node $x_{i,t}$, a local prediction $\mu_{i,t}$, and a local prediction error $\epsilon_{i,t}=x_{i,t}-\mu_{i,t}$, with predictions computed from incoming neighbors through weighted nonlinear transformations [2201.13180]. One explicit form is
$$
\mu_{i,t}=\sum_j \theta_{j,i} f(x_{j,t}),
$$
with energy
$$
\mathcal{E}_t=\frac12\sum_i (\epsilon_{i,t})^2,
$$
and local weight updates
$$
\Delta\theta_{i,j}=\alpha\,\epsilon_{i,T} f(x_{j,T})
$$
after inference has converged [2201.13180].

A closely related computational-graph formulation defines each node $\alpha$ by an activity $a_\alpha\in\mathbb{R}$, a prediction
$$
\mu_\alpha=f\!\left(\sum_{\beta=1}^N w_{\alpha\beta} a_\beta\right),
$$
and a prediction error $\epsilon_\alpha=a_\alpha-\mu_\alpha$, yielding the energy
$$
E_G=\sum_{\alpha=1}^N \epsilon_\alpha^2
$$
under unit precisions [2603.06142]. This formulation explicitly positions PCGs as a generalization of predictive coding networks beyond layered hierarchies and proves that, with an appropriate hierarchical block mask, the PCG energy reduces to the standard hierarchical predictive-coding energy up to a constant [2603.06142].

A third, variational formulation starts from a differentiable computation graph and augments every intermediate variable $v_i$ with a prediction $\hat v_i$ and an error node $\epsilon_i=v_i-\hat v_i$, with a quadratic free energy
$$
\mathcal{F}\approx \sum_{i=0}^N \epsilon_i^\top \epsilon_i
$$
under identity precisions [2006.04182]. In that setting, predictive coding acts on the augmented graph $\tilde{\mathcal{G}}=\{E,V,\mathcal{E}\}$ and recovers reverse-mode differentiation through local error dynamics at convergence [2006.04182].

These definitions share a common structure. First, predictions are generated along the directed edges of the graph. Second, discrepancies between current node states and their local predictions define nodewise errors. Third, the global objective is a sum of squared local errors or a precision-weighted variant of that sum. This suggests that “Predictive Coding Graph” is best understood not as a single architecture but as a family of graph-based predictive-coding instantiations linked by the same local error-minimization principle.

## 2. Inference dynamics and local learning

The central computational mechanism of a PCG is iterative inference on node activities while parameters are held fixed, followed by local parameter updates derived from the same energy [2212.04656; 2201.13180]. In the arbitrary-graph formulation, state inference follows gradient descent on $\mathcal{E}_t$:
$$
\Delta x_{i,t}
=
-\gamma \frac{\partial \mathcal{E}_t}{\partial x_{i,t}}
=
\gamma\left(-\epsilon_{i,t}+f'(x_{i,t})\sum_{k=1}^n \epsilon_{k,t}\theta_{k,i}\right),
$$
where the first term reduces the local error at node $i$, and the second propagates downstream error information through outgoing synapses [2201.13180]. The corresponding local parameter update is
$$
\Delta\theta_{i,j}=\alpha\,\epsilon_{i,T} f(x_{j,T}),
$$
a postsynaptic-error times presynaptic-activity rule [2201.13180].

In graph-representation-learning formulations, the same inference-before-learning schedule is emphasized more explicitly. Each node carries a value unit $h_{u,t}$ and a prediction-error unit $\epsilon_{u,t}$, with node predictions
$$
\mu_{u,t}=\sum_{v\in p(u)} w_{v,u} f(h_{v,t}),
\qquad
\epsilon_{u,t}=h_{u,t}-\mu_{u,t},
$$
and global energy
$$
F_t=\frac12\sum_u (\epsilon_{u,t})^2.
$$
Activity updates then take the form
$$
h_{u,t+1}
=
h_{u,t}
-
\eta\left[
\epsilon_{u,t}
-
f'(h_{u,t})\sum_{v\in c(u)} \epsilon_{v,t} w_{v,u}
\right],
$$
for $t=0,\dots,T-1$, followed by the local weight update
$$
\Delta w_{i,j}\propto \alpha\cdot \epsilon_i\cdot f(h_j)
$$
once inference has stabilized [2212.04656].

In differentiable generalized predictive coding, the objective is extended to precision-weighted hierarchical and dynamical errors. For latent states $s^l(t)$ across layers,
$$
\epsilon^l(t)=s^l(t)-f^l(s^{l+1}(t);\theta^l),
\qquad
E=\sum_l \epsilon^{l\top}(t)\Pi^l \epsilon^l(t),
$$
and, in generalized-coordinate notation,
$$
\epsilon_v^l=\mu_v^{l-1}-g^l(\mu_x^l,\mu_v^l),\qquad
\epsilon_x^l=D\mu_x^l-f^l(\mu_x^l,\mu_v^l),
$$
with precision-weighted errors $\xi_v^l=\Sigma_v^{-1,l}\epsilon_v^l$ and $\xi_x^l=\Sigma_x^{-1,l}\epsilon_x^l$ [2112.03378]. State and parameter updates are obtained by automatic differentiation of the total precision-weighted objective [2112.03378].

A recurrent theme across these variants is that PCGs replace a one-shot forward computation plus a separate nonlocal backward pass by iterative local relaxation. The factual commonality is the inference-before-learning schedule; a plausible implication is that many empirical and theoretical differences among PCG variants arise more from graph topology, precision modeling, and inference scheduling than from the basic local learning rule itself.

## 3. Relation to feedforward networks, backpropagation, and automatic differentiation

A major development in the PCG literature is the formal connection to reverse-mode automatic differentiation and standard feedforward neural networks [2006.04182; 2103.04689; 2603.06142]. In the computation-graph perspective, predictive coding is defined directly on arbitrary differentiable DAGs by augmenting each variable with a prediction and an error node. With the free-energy objective
$$
\mathcal{F}\approx \sum_{i=0}^N \epsilon_i^\top \epsilon_i,
$$
the state dynamics are
$$
\frac{dv_i}{dt}
=
-\frac{\partial \mathcal{F}}{\partial v_i}
=
\epsilon_i-\sum_{j\in \mathcal{C}(v_i)} \epsilon_j \frac{\partial \hat v_j}{\partial v_i},
$$
and the parameter dynamics are
$$
\frac{d\theta}{dt}
=
-\frac{\partial \mathcal{F}}{\partial \theta}
=
\epsilon_i \frac{\partial \hat v_i}{\partial \theta_i}.
$$
At fixed points, the error recursion coincides with the backpropagation recursion on the original computation graph [2006.04182].

The stronger exactness result is given by the graph-theoretic generalization of Zero-Divergence Inference Learning. There, a differentiable scalar mapping is represented as a DAG, transformed into a levelled DAG by inserting identity nodes so that all directed paths between nodes have equal length, and then subjected to a level-synchronous predictive-coding schedule with $\gamma=1$ [2103.04689]. Under zero-divergence initialization, output clamping, and level-synchronous parameter updates, the paper shows that the PCG update equals the reverse-mode gradient on arbitrary architectures representable as DAGs, including residual and transformer-style graphs [2103.04689]. The relevant local activity update is
$$
\dot x_i
=
-\frac{\partial E}{\partial x_i}
=
\varepsilon_i
-
\sum_{j\in \mathrm{ch}(i)}
\left(J^{(j)}_{x_i}\right)^\top \varepsilon_j,
$$
with local parameter update
$$
\Delta\theta_i
=
\eta\,\varepsilon_i^\top \frac{\partial \hat x_i}{\partial \theta_i},
$$
and the theorem states that, under the stated construction, $\Delta\theta_\ell=-\eta\,\frac{\partial L}{\partial \theta_\ell}$ [2103.04689].

The 2026 superset result shifts the emphasis from gradient equivalence to representational containment. A hierarchical predictive-coding network is equivalent to a feedforward neural network during testing because minimizing the hierarchical PCN energy enforces $\epsilon_i^\ell=0$ layer by layer, yielding the standard feedforward equations $a_i^\ell=f(\sum_j w_{ij}^{\ell-1}a_j^{\ell-1})$ [2603.06142]. The same work proves that if the general PCG weight matrix is masked so that $W^{\ell k}=0$ unless $k=\ell-1$, then the PCG energy equals the hierarchical PCN energy up to a constant, and the activity and learning argmins coincide [2603.06142]. This establishes PCGs as a mathematical superset of predictive coding networks and, during testing, of feedforward neural networks [2603.06142].

This line of work clarifies a common misconception. PCGs are not merely heuristic or approximate graph analogues of neural networks. Under specific assumptions, they either asymptotically recover exact backpropagation on arbitrary computation graphs [2006.04182], exactly implement reverse differentiation on levelled DAGs [2103.04689], or strictly contain the standard feedforward architecture class through an energy-based generalization [2603.06142].

## 4. Generalized, dynamical, and modular PCGs

Beyond static graph inference, PCGs have been extended to generalized predictive coding with temporal dynamics, generalized coordinates, and flexible graph structure [2112.03378]. In this setting, the latent state at each layer is expanded into derivative orders,
$$
x^{(0)}(t)=x(t),\quad x^{(1)}(t)=\frac{dx}{dt},\quad x^{(2)}(t)=\frac{d^2x}{dt^2},\dots
$$
with shift operator $D x^{(n)}(t)=x^{(n+1)}(t)$ [2112.03378]. The total objective becomes a sum across layers and derivative orders,
$$
E=\sum_l \sum_{n=0}^N \epsilon^{l,(n)\top}(t)\Pi^{l,(n)}\epsilon^{l,(n)}(t),
$$
and the discrete-time dynamical pathway is written as
$$
x_{l,t}=d(x_{l,t-dt},dt)+s,
$$
with hierarchical, dynamical, and derivative errors
$$
e_{h,l}=x_{l,t}-h(x_{l+1,t}),\qquad
e_{d,l}=x_{l,t}-d(x_{l,t-dt},dt),\qquad
e_{d',l}=d'(x_{l,t})-\widehat{d'}(x_{l,t}),
$$
combined as
$$
e_l=e_{h,l}+e_{d,l}+e_{d',l}.
$$
These formulations allow hierarchical content predictions and temporal consistency constraints to act jointly on the same latent variables [2112.03378].

The graph structure in generalized predictive coding is not restricted to strict hierarchies. The model explicitly allows directed acyclic graphs and directed cyclic graphs, with node types including observation nodes, latent nodes, parameter nodes, and precision nodes, and edge types including generative edges, temporal edges, and error/precision edges [2112.03378]. Stability constraints are correspondingly technical: precision matrices should be positive-definite, and damping, bounded $dt$, and gradient clipping are used for recurrent or cyclic schedules [2112.03378].

The same paper places PCGs in a modular language using canonical microcircuit metaphors and Markov blankets. In that account, generative top-down predictions correspond to $h/f$ edges, feedforward error signals correspond to $\epsilon$ nodes propagating upward, and blanket states define module interfaces such that internal states are conditionally independent of external states given the blanket [2112.03378]. This supports the interpretation of PCGs as compositional systems built from subgraphs with well-defined message-passing boundaries.

This suggests a broader reading of PCGs than the simpler “predictive coding on arbitrary graphs” slogan. A plausible implication is that the graph formalism is not only about arbitrary connectivity; it also provides a language for integrating hierarchy, dynamics, modularity, and time-scale adaptation within one predictive-coding objective.

## 5. Applications in machine learning and graph representation learning

A prominent application area is graph representation learning, where predictive coding is used as the message-passing rule inside graph neural architectures [2212.04656]. In this literature, Graph Predictive Coding Networks (GPCNs) instantiate PCGs with hierarchical layers mirroring popular GNNs such as GCN and GAT, while replacing the usual feedforward pass with iterative predictive-coding inference [2212.04656]. At each node, the model computes a prediction from neighbors, forms an error, relaxes activities by gradient descent on the energy, and only then updates weights locally [2212.04656].

The reported empirical results show that these models are comparable to standard GNNs in raw predictive performance while improving calibration and robustness [2212.04656]. On transductive node classification, the paper reports: Cora, GCN $80.72\pm1.05\%$ and GPCN $80.70\pm1.09\%$; Citeseer, GCN $67.12\pm1.53\%$ and GPCN $67.26\pm1.28\%$; PubMed, GCN $77.10\pm1.45\%$ and GPCN $76.20\pm2.44\%$ [2212.04656]. On inductive F1, the paper reports: Cora, GCN $80.00\pm0.41\%$ and GPCN $79.66\pm0.75\%$; Citeseer, GCN $67.64\pm1.14\%$ and GPCN $69.68\pm0.37\%$; PubMed, GCN $77.00\pm0.46\%$ and GPCN $77.12\pm0.47\%$; PPI supervised, GCN $76.45\pm0.39\%$ and GPCN $78.31\pm0.47\%$; PPI unsupervised, GCN $52.44\pm0.37\%$ and GPCN $54.41\pm0.31\%$ [2212.04656].

The same study reports lower calibration error and stronger resistance to adversarial attacks. For Nettack targeted evasion, the robustness metric $\sum_{q=1}^5 q\cdot p_q$ is reported as $3.26\pm0.18$ for GPCN on Cora, compared to $2.05\pm0.07$ for GCN and $2.79\pm0.10$ for RGCN; $2.73\pm0.08$ for GPCN on Citeseer, compared to $1.98\pm0.12$ for GCN and $2.02\pm0.23$ for RGCN; and $4.21\pm0.32$ for GPCN on PubMed, compared to $1.14\pm0.02$ for GCN and $1.48\pm0.02$ for RGCN [2212.04656]. On global poisoning with Metattack at $25\%$, the reported results include Cora: GCN $44.16\pm0.88$, GPCN-GCN $54.27\pm7.25$, GAT $54.78\pm0.74$, GPCN-GAT $49.68\pm1.08$, RGCN $50.51\pm0.78$; Citeseer: GCN $57.49\pm2.13$, GPCN-GCN $58.43\pm1.70$, GAT $61.85\pm1.12$, GPCN-GAT $62.02\pm0.73$, RGCN $55.35\pm0.66$; PubMed: GCN $73.34\pm0.19$, GPCN-GCN $75.69\pm1.43$, GAT $65.41\pm0.77$, GPCN-GAT $68.94\pm0.60$, RGCN $67.95\pm0.15$ [2212.04656].

Outside GNNs, arbitrary-topology PCGs have been applied to generation, reconstruction, denoising, classification, and associative memory within the same network by clamping or initializing different subsets of nodes [2201.13180]. On fully connected PCGs for classification, the paper reports MNIST $91.76\pm0.02\%$, FashionMNIST $83.72\pm0.33\%$, and SVHN $84.51\pm0.11\%$ [2201.13180]. For denoising on MNIST, Fréchet Inception Distance values are reported as $25.61/44.53/51.38$ for PCG at noise variance $0.2/0.5/0.7$, compared to $43.93/53.79/57.56$ for an autoencoder [2201.13180].

In generalized predictive coding, applications include perception and planning on sequential data, learning dynamical predictions, and learning sampling distances in parallel [2112.03378]. The paper states that a stride of $10$ in the modulated sine-wave experiment yields a nearly stationary input at the lowest layer with an almost constant second derivative, whereas other strides such as $5$ or $9$ incur higher prediction error [2112.03378]. That claim is narrowly tied to the reported experiment; a plausible implication is that adaptive time-step selection can function as a structural degree of freedom in temporal PCGs.

## 6. Causal inference, interventions, and structure learning

PCGs have also been cast as structural causal models over arbitrary directed graphs [2306.15479]. In that formulation, each node $x_i$ has parents $\mathrm{par}(i)$, predictive mean
$$
\mu_i=\sum_{k\in \mathrm{par}(i)} W^{(k,i)} f(x_k),
$$
and Gaussian conditional
$$
p(x_i\mid \mathrm{par}(i))=\mathcal{N}(x_i;\mu_i,I),
$$
with predictive-coding energy
$$
F=\sum_i \|x_i-\mu_i\|^2+\ln(2\pi)
$$
and local weight update
$$
\Delta W_{i,j}=\alpha\, e_{i,T} f(x_{j,T})
$$
after inference [2306.15479].

The key causal result is an intervention rule that does not require explicit graph mutilation. To compute the effect of $\mathrm{do}(x_j=s)$, the procedure clamps the intervened node at all times, $x_{j,t}=s$, and sets its local error to zero for all times, $e_{j,t}=0$ [2306.15479]. The paper states the resulting interventional expectation as
$$
E(x_i \mid \mathrm{do}(x_j=s))
=
E(x_{i,T}\mid \forall t: x_{j,t}=s,\ e_{j,t}=0),
$$
arguing that zeroing the error prevents upstream correction through the intervened node and thus matches Pearl’s truncated-factorization semantics [2306.15479].

When the graph is unknown, the adjacency matrix can itself be learned by treating it as a continuous parameter in the predictive mean,
$$
\mu_i=\sum_{k=0}^N a_{k,i} f_{k,i}(x_k),
$$
and updating adjacency entries by gradient descent [2306.15479]. To enforce sparsity and acyclicity, the paper adds priors including
$$
l(A)=\exp\!\left(-\sum_{i,j}|a_{i,j}|\right)
$$
and the differentiable acyclicity term
$$
h(A)=\mathrm{tr}(\exp(A\times A))-d,
$$
which is zero iff the graph is acyclic [2306.15479].

The empirical evaluation covers both causal inference and classification. On causal graphs such as collider, confounder, mediator, chain, fork, M-bias, and butterfly-bias motifs, the paper reports that PCGs achieve low errors across associational, interventional, and counterfactual queries and outperform MultiCVAE, CAREFL, and VACA on most metrics while using a very small number of parameters [2306.15479]. One concrete example given is the butterfly graph, where PCGs report interventional MMD $0.27\pm0.11$, MeanE $0.43\pm0.24$, StdE $0.02\pm0.00$, counterfactual MSE $2.84\pm0.38$, and SSE $1.88\pm0.21$, with only $16$ parameters [2306.15479]. On MNIST and FashionMNIST, the paper reports consistent improvements of approximately $2\%$ test accuracy when test-time inference uses interventional queries that clamp inputs and set their errors to zero [2306.15479].

These results address another misconception. PCGs are sometimes described only as biologically motivated substitutes for backpropagation. The causal-modeling literature shows that the same local inference machinery can be used to answer associational, interventional, and counterfactual queries and to perform causal discovery on random graphs [2306.15479].

## 7. Scope, ambiguities, and limitations

The predictive-coding PCG literature is technically coherent but terminologically ambiguous. In graph theory, “PCG” usually denotes “Pairwise Compatibility Graph,” a tree-metric graph class defined by leaf-to-leaf distances lying inside an interval $[d_{\min},d_{\max}]$ [1707.07436]. In that literature, a graph $G=(V,E)$ is a PCG if there exists an edge-weighted tree $T$ and thresholds such that
$$
(u,v)\in E
\Longleftrightarrow
d_{\min}\le d_T(u,v)\le d_{\max},
$$
and recent work studies non-PCG graph classes, star-$k$-PCGs, and $2$-interval-PCGs [1707.07436; 2209.11860; 2202.13844]. This meaning is entirely different from predictive coding graphs. Because both literatures use the same acronym, precise contextual qualification is necessary.

Within the predictive-coding sense, the main limitations are computational and dynamical rather than definitional. Iterative inference creates overhead relative to one-pass feedforward evaluation. In graph representation learning, PCG inference adds $T$ inference steps per batch, with cost roughly $O(T\cdot Ed)$ plus error-computation overhead, and the paper notes trade-offs between lower energy, calibration gains, and runtime [2212.04656]. In arbitrary-topology PCGs, per-sample training cost is $O(T|E|)$ for inference plus $O(|E|)$ for one weight update, with inference iterations ranging from a handful in associative-memory settings to $2000$ in some generation and reconstruction experiments [2201.13180]. The 2026 superset paper states that dense PCG testing is $O(N^2T)$, and for a hierarchical PCG with comparable parameter count, testing can be a factor approximately $N\cdot T$ slower than one-pass FNN inference [2603.06142].

Convergence and stability are recurrent concerns. The arbitrary-graph PCG paper reports robust empirical convergence provided the inference step size is not too large, weight decay is applied, and sufficient inference iterations are used [2201.13180]. The generalized predictive-coding paper emphasizes positive-definite precisions, damping, bounded $dt$, gradient clipping, and truncated time horizons for BPTT through temporal edges [2112.03378]. The graph-representation-learning paper notes sensitivity to $T$, $\eta$, and learning rate, and advises damping for stability [2212.04656]. The causal PCG paper notes sensitivity to the weighting of sparsity and DAG priors, adjacency thresholds, and step sizes for state, weight, and adjacency updates [2306.15479].

Theoretical exactness also depends on specific assumptions. Asymptotic equivalence to backpropagation along arbitrary computation graphs in the 2020 work requires convergence to a fixed point and a fixed-prediction assumption during inference [2006.04182]. Exact reverse differentiation in the 2021 work requires a DAG representation, leveling by identity nodes, zero-divergence initialization, $\gamma=1$, and level-synchronous parameter updates [2103.04689]. The superset result in 2026 establishes containment and test-time equivalence under a hierarchical block mask, but does not claim that arbitrary non-hierarchical PCGs inherit all feedforward-network properties under arbitrary training protocols [2603.06142].

Taken together, these limitations do not weaken the core definition of PCGs. Rather, they mark the current frontier of the field: making arbitrary-topology predictive coding computationally efficient, stable under richer cyclic dynamics, scalable to larger graphs, and theoretically characterized beyond the special cases where exact equivalence to existing learning algorithms is already known [2112.03378; 2212.04656; 2603.06142].

Source: https://www.emergentmind.com/topics/predictive-coding-graphs-pcgs