--- title: Divide-and-Conquer Predictive Coding url: https://www.emergentmind.com/topics/divide-and-conquer-predictive-coding-dcpc type: topic --- # Divide-and-Conquer Predictive Coding Divide-and-Conquer Predictive Coding (DCPC) is a structured Bayesian inference algorithm designed for generative models with explicit graphical structure, enabling biologically plausible, fully local learning and inference. DCPC addresses the longstanding gap between predictive coding's theoretical appeal and its empirical performance in high-dimensional structured inference tasks by respecting posterior correlation structure and performing provable maximum-likelihood parameter updates, while relying exclusively on local computations between neighboring nodes [2408.05834]. ## 1. Structured Generative Models and Model Architecture DCPC operates on directed acyclic graphs (DAGs) comprising observed variables $x = \{x_i\}$ and latent variables $z = \{z_j\}$, parameterized by $\theta$. The joint density factors as $$ p(x, z; \theta) = \prod_{v \in x \cup z} p(v \mid \mathrm{pa}(v); \theta_v), $$ where $\mathrm{pa}(v)$ denotes the parents of node $v$ in the graph and each conditional $p(v \mid \mathrm{pa}(v); \theta_v)$ is locally parameterized. For hierarchical models, a typical example is the chain $z_1 \to z_2 \to x$, with each node conditionally dependent on its DAG parents. This explicit factorization enables DCPC to exploit model structure during inference and learning, distinguishing it from prior predictive coding variants that often assume factorized posteriors or disregard higher-order dependencies. ## 2. Variational Objective and Theoretical Foundations The algorithm optimizes the evidence lower bound (ELBO), also known as variational free energy: $$ \mathcal{L}(q, \theta) = \mathbb{E}_{q(z)}[\log p(x, z; \theta)] - \mathbb{E}_{q(z)}[\log q(z)], $$ subject to $\mathcal{L}(q, \theta) \leq \log p(x; \theta)$. DCPC employs a particle-based empirical variational distribution $q(z)$ with $K$ samples (particles), facilitating both accurate posterior approximation and practical gradient-based optimization. ## 3. Divide-and-Conquer Algorithmic Updates DCPC introduces an interleaved two-loop procedure comprising a particle-based latent inference (inner loop) and local parameter learning (outer loop): **Inner Loop: Latent Inference** - For each latent $z_j$, DCPC updates particles via coordinate-wise Langevin sampling. The complete conditional for $z_j$ is $$ p(z_j \mid x, z_{\setminus j}; \theta) \propto p(z_j \mid \mathrm{pa}(z_j); \theta_{z_j}) \prod_{c \in \mathrm{ch}(z_j)} p(c \mid \mathrm{pa}(c); \theta_c), $$ leveraging both parent and child nodes to compute the local 'prediction error' (score) $$ \epsilon_{z_j} = \nabla_{z_j} \log p(z_j \mid \mathrm{pa}(z_j)) + \sum_{c \in \mathrm{ch}(z_j)} \nabla_{z_j} \log p(c \mid \mathrm{pa}(c)). $$ Each of $K$ particles is updated via a Langevin proposal: $$ z_j^{\mathrm{new},(k)} \sim \mathcal{N}(z_j^{\mathrm{old},(k)} + \eta \epsilon_{z_j}^{(k)}, 2\eta I), $$ followed by importance resampling with weights proportional to the ratio of model likelihoods and proposal densities. Repeated application yields approximate Gibbs samples from the conditional posterior. **Outer Loop: Parameter Learning** - Parameter updates are performed by stochastic gradient descent on the negative ELBO, $$ F(\theta) \approx -\frac{1}{K} \sum_k \log p(x, z^{(k)}; \theta) + \text{const}, $$ with local gradients: $$ \nabla_{\theta_v} F = -\frac{1}{K} \sum_{k=1}^K \nabla_{\theta_v} \log p(v^{(k)} \mid \mathrm{pa}(v)^{(k)}; \theta_v). $$ Each parameter vector $\theta_v$ is updated by local prediction errors arising from activity at $v$ and its neighbors only. ## 4. Local Computation, Prediction Errors, and Closed-Form Solutions All inference and learning computations in DCPC are local to nodes and their neighbors: - **Prediction Errors:** For each latent $z$, the local score is $$ \epsilon_{z} = \nabla_{z} \log p(z|\mathrm{pa}(z); \theta_z) + \sum_{c \in \mathrm{ch}(z)} \nabla_z \log p(c|\mathrm{pa}(c); \theta_c). $$ - **Precision (Gaussian Case):** If $p(v|\mathrm{pa}(v))$ is Gaussian $\mathcal{N}(v; \mu_v(\mathrm{pa}(v); \theta_v), \tau_v^{-1}I)$, the score reduces to $\tau (v - \mu)$, where $\tau$ is the precision. - **Closed-form Maximum Likelihood Updates:** For linear-Gaussian nodes: $$ p(z|\mathrm{pa}(z)) = \mathcal{N}(z; W \cdot \mathrm{pa}(z) + b, \Lambda^{-1}), $$ maximum likelihood solutions for $W$ and $b$ are derived via empirical moments over particles: $$ W = \left[ \sum_k \mathbb{E}[z^{(k)} \mathrm{pa}(z)^{(k)\top}] - b \sum_k \mathbb{E}[\mathrm{pa}(z)^{(k)\top}] \right] \left[ \sum_k \mathbb{E}[\mathrm{pa}(z)^{(k)} \mathrm{pa}(z)^{(k)\top}]\right]^{-1}, $$ $$ b = \frac{1}{K} \sum_k \left(\mathbb{E}[z^{(k)}] - W \mathbb{E}[\mathrm{pa}(z)^{(k)}] \right). $$ ## 5. Biological Plausibility: Locality and Minimal Global Coordination DCPC provides a constructive proof of biological plausibility. All inference and parameter updates rely on local (pre- and post-synaptic) activity, prediction-error signals, and Hebbian-like weight adjustments. Complete-conditional sampling for each latent $z$ utilizes only the local score and Langevin proposals, with importance resampling, yielding approximate Gibbs samples. Parameter gradients similarly require no information beyond the immediately adjacent nodes. No global backpropagation or nonlocal gradient flow is required. This local structure is congruent with plausibility criteria from theoretical neuroscience, contrasting with the global error propagation in standard deep learning approaches. ## 6. Algorithmic Implementation and Pseudocode A high-level summary of DCPC's algorithm follows, abstracted in Pyro-style pseudocode: ```python def DCPC_step(z_particles, θ, x, η, K): # 1. Inference sweep over latents for v in latent_nodes: # compute local prediction errors ε_v for each particle ε = [∇_v log p(v|pa(v);θ_v) + Σ_ch ∇_v log p(ch|pa(ch);θ_ch) for each of K particles] # propose new particles via Langevin proposals = [z_old + η*ε[k] + sqrt(2η)*NormalNoise() for k,z_old in enumerate(z_particles[v])] # compute importance weights weights = [ p_complete_conditional(proposals[k]) / N(proposals[k]; z_old+η*ε[k],2ηI) for k,z_old in enumerate(z_particles[v])] # resample K new particles z_particles[v] = resample(proposals,weights) # 2. Parameter update grads = {θ_v:0 for v in x∪z} for k in range(K): for v in x∪z: grads[θ_v] += ∇_{θ_v} log p(v^{(k)}|pa(v)^{(k)};θ_v) for v in x∪z: θ_v += η * (grads[θ_v]/K) return z_particles, θ ``` In practical implementations using probabilistic programming frameworks such as Pyro, each latent variable is registered with a local guide that performs Langevin proposals and resampling as above. ## 7. Empirical Performance and Benchmarking DCPC demonstrates competitive or superior performance to previous predictive coding and particle-based inference methods across a range of tasks: | Task | Baseline | Metric | DCPC | Baseline Value | |-----------------------------|----------------------------------|---------------|--------------|---------------------| | DLGM on MNIST (K=4) | MCPC (K=4) [Oliviers et al. 2024]| NLL | 102.5±0.01 | 144.6±0.7 | | | | MSE | 0.01±7.2e-6 | 8.29e-2±0.05e-2 | | CelebA Face Gen (64×64) | LPC [Zahid et al. 2024] | FID | 96.0±0.3 | ≈120 | | PGD Head-to-Head (32×32) | PGD [Kuntz et al. 2023] | FID | 89.6±0.6 | 100±2.7 | DCPC outperforms or matches MCPC, LPC, and PGD in negative log-likelihood, mean squared error, and Fréchet Inception Distance (FID), while strictly operating via local computation and respecting posterior dependencies [2408.05834]. A plausible implication is that DCPC closes the gap between biological plausibility and state-of-the-art variational inference in structured models, potentially enabling new classes of scalable, locally-learned Bayesian neural architectures. Source: https://www.emergentmind.com/topics/divide-and-conquer-predictive-coding-dcpc