Active Predictive Coding Networks (APCNs)

• APCNs are a class of neural architectures that extend predictive coding by incorporating precision-weighted updates over latent states and parameters for optimized inference.
• They utilize local error propagation, error precision matrices, and empirical Fisher information to approximate natural gradient descent and improve convergence.
• Empirical findings show that variants like PredProp achieve 25–40% lower mean squared error on benchmarks such as MNIST and OMNIGLOT compared to traditional optimizers.

Active Predictive Coding Networks (APCNs) constitute a rigorously optimized class of neural architectures that extend the predictive coding framework to facilitate principled, precision-weighted inference and parameter updates in deep and hierarchical networks. Leveraging local error propagation, error precision matrices, and empirical Fisher information, APCNs aim to approximate natural gradient descent and thereby accelerate convergence, improve sample efficiency, and increase robustness to scaling and architectural depth. Recent variants such as PredProp provide a concrete algorithmic instantiation of APCNs, generalizing classical predictive coding networks (PCNs) by incorporating precision-weighted updates for both neural activities and parameters, and factorizing error precisions over deep decoder sublayers. These developments have established APCNs as a leading paradigm for biologically plausible and computationally efficient stochastic optimization in multilayer neural models (Ofner et al., 2021).

1. Theoretical Principles of Predictive Coding and Precision Weighting

The core principle underpinning APCNs is minimization of a variational free-energy objective ℱ(θ, μ), where θ denotes network parameters and μ the latent states. Classical PCNs operate by propagating prediction errors up the network, minimizing local layerwise discrepancies between top-down predictions and bottom-up sensory data. In the probabilistic formulation, prediction errors at each layer i are modeled as Gaussian, characterized by error covariance matrices Σₑ^{(i)} and their inverses—the precision matrices Σₑ^{(i)−1}.

In APCNs, the activity (state) and parameter updates are further weighted by empirical precision estimates. Given an error gradient g_t=∇μℱ, the adaptive update rule for activities is: $$\frac{d \mu^{(i)}}{d t} \propto \left(\Sigma_{\mu} + \lambda_{\Sigma_\mu} I\right)^{-1} \left(\Sigma_{g_t} + \lambda_{g} I\right)^{-1} g_t,$$ where Σμ is the activity covariance, Σ{g_t} the covariance of the prediction error gradient, and λ{Σ_μ}, λ_g are regularization scalars. This update is the product of two precision-weighting factors (inverse covariances), modulating the magnitude and direction of the update vector to reflect local error and activity uncertainty (Ofner et al., 2021).

2. Natural Gradient Perspective and Fisher Information

The adaptation of APCNs is grounded in the information-geometric connection between batchwise error covariances and the Fisher information matrix (FIM). For any parameter p, the empirical FIM is: $$I(p) = \mathbb{E}\left[ \nabla_p \log p(u|p)\, \nabla_p^\top \log p(u|p) \right] = \operatorname{Cov}\left[\nabla_p \log p(u|p)\right].$$ For linear-Gaussian PCNs, the error precision matrix coincides with the Fisher information, and thus APCNs approximate natural gradient descent by preconditioning parameter and state updates using precision (inverse covariance) estimates. This analytic structure factorizes in deep or hierarchical networks, allowing each layer to compute its own update using local error and activity covariances. PredProp, as a canonical APCN, further extends this principle to multilayer deep decoders by nesting local FIMs and applying precision weighting individually at each sublayer (Ofner et al., 2021).

3. Algorithmic Structure: The PredProp Scheme

Implementation of APCNs such as PredProp employs a coordinated E-step/M-step scheme, with distinct but analogous precision-weighted updates for activities and weights. The computational outline is:

• E-step (Inference): Iteratively infer latent states μ via precision-weighted gradient descent on ℱ, using local estimates of Σμ and Σ{g_t}.
• M-step (Learning): Update parameters θ (and in multilayer decoders, sublayer θj) using precision-weighted gradients, with respective hyperparameters and covariances Σ{ĥg}, Σ_{μ_j}.

A high-level schematic pseudocode follows:

\begin{algorithmic} \State initialize\; %%%%0%%%% \While{not converged(%%%%1%%%%)} \quad/* E‐step: infer states */ \State %%%%2%%%% \For{%%%%3%%%%} \State %%%%4%%%% \State %%%%5%%%% \State %%%%6%%%% \State $g_t\leftarrow(\Sigma_{\mu}+\lambda_{\Sigma_\mu}I)^{-1} (\Sigma_{g_t}+\lambda_g I)^{-1}\,g_t%%%%7%%%%\mu\leftarrow\mu + \alpha_E\,g_t$ \EndFor \quad/* M‐step: learn weights */ \State %%%%8%%%% \State %%%%9%%%% \State %%%%10%%%% \State $g'_t\leftarrow (\Sigma_{\hat g}+\lambda_{\hat g}I)^{-1} \,g'_t\, (\Sigma_{\mu}+\lambda_{\Sigma_\mu}I)^{-1}%%%%11%%%%\theta\leftarrow\theta+\alpha_M\,g'_t$ \EndWhile \end{algorithmic}

This design ensures that uncertainty in both the predicted states and the model’s gradients directly modulates the step size and direction.

4. Layer-wise and Deep Decoder Extensions

APCNs generalize naturally to hierarchies by factorizing precisions and update rules over layers. Each layer computes and applies its local activity and error covariances, enabling scalable and parallelizable optimization. For decoders with deep, multi-layer architectures, precision estimation and weighting are performed per sublayer. The weight update for each sublayer θ_j of PCN layer i is: $$\frac{d \theta_j^{(i)}}{d t} \propto \left( \Sigma_{\hat g_j} + \lambda_{\hat g} I \right)^{-1} g'_{j,t} \left( \Sigma_{\mu_j} + \lambda_{\Sigma_\mu} I \right)^{-1}.$$ This multi-level factorization enhances convergence and stability, particularly in networks with deep decoding branches (Ofner et al., 2021).

5. Hyperparameterization and Practical Optimization

Regularization and stability are managed via hyperparameters for the covariance dampening factors (e.g., λ{Σμ}, λg), with defaults reported as λμ=0.001, λ{ĥg}=0.1 (M-step) and λ{Σ_μ}=0.9, λ_g=0.9 (E-step). Learning rates for states and weights are set to α_E≈0.9, α_M≈0.5, respectively. Covariances are computed per-batch using unbiased estimators, and a constant top-down error weighting factor (1/10) is used to balance upward and downward flows. The authors observe that both E-step and M-step precision-weighting contribute to convergence speed and solution quality; omitting either reduces performance, but does not erase the overall benefit (Ofner et al., 2021).

6. Empirical Findings and Benchmarking

Empirical evaluations demonstrate that precision-weighted APCNs (PredProp) outperform Adam and other adaptive optimizers in convergence rate and final mean squared error (MSE), especially on tasks involving multi-layer dense decoders. On single-layer PCNs (three predictive coding layers of sizes 64→64→128 with tanh decoders), PredProp achieves running-average MSE ≈0.01–0.02 compared to Adam’s ≈0.05–0.06 plateau. With multi-layer decoders (3 layers × 3-layer MLPs of width 256), PredProp yields 25–40% lower final MSE on MNIST and OMNIGLOT benchmarks. Ablations confirm that both activity and weight precision-weighting are necessary for optimal results; full APCNs are consistently superior over partial or unweighted schemes (Ofner et al., 2021).

7. Connections, Implications, and Future Trajectories

The precision-weighted update framework underlying APCNs aligns them conceptually with broader advances in exact natural gradient methods for deep learning. The observed acceleration and improved generalization are consistent with the geometric interpretation of the Fisher information as a local curvature metric. The APCN structure—separately estimating activity and error precision per layer or subnetwork, and coupling inference with learning—offers a scalable and biologically plausible pathway for stochastic optimization. A plausible implication is that further exploration of precision estimation techniques (e.g., using online or low-rank covariance tracking) and integration with other Fisher-based second-order methods may yield additional gains. Extensions to combinatorial decoders and unsupervised hierarchical representations also represent promising directions for future research (Ofner et al., 2021).