Decentralized Bayesian Inference Overview

Updated 17 March 2026

Decentralized Bayesian inference is a distributed approach that computes global posteriors using local Bayesian updates and peer-to-peer consensus without a central server.
It leverages iterative updates and consensus algorithms to integrate local data from networked agents, addressing challenges like communication constraints and model heterogeneity.
Applications include federated learning, sensor networks, and multi-agent robotics, where efficient uncertainty quantification and robust inference are critical.

Decentralized Bayesian Inference encompasses a class of methodologies for performing fully Bayesian statistical inference, uncertainty quantification, and probabilistic decision-making over a network of agents, without a central coordinator or server. These approaches leverage both local data-processing and peer-to-peer communication to reach global inferential objectives such as posterior computation, estimation, or knowledge fusion. Motivated by distributed sensor systems, federated learning, multi-agent robotics, ad-hoc wireless networks, and collaborative AI, decentralized Bayesian inference frameworks confront challenges of communication constraints, network topology, data privacy, model heterogeneity, and computational scalability.

1. Foundational Principles and Problem Setting

Decentralized Bayesian inference considers a collection of agents (nodes) connected by an undirected or directed graph. Each agent possesses a private dataset and typically maintains a local belief (posterior or approximate posterior) over latent variables or model parameters. The global inferential goal is to approximate the same result that would be obtained by centrally pooling all agents' data and computing the Bayes posterior with respect to a shared prior.

Mathematically, for a global latent parameter $\theta$ and local datasets $D_i$ , the global posterior is $p(\theta|D_1,\ldots,D_N)\propto p_0(\theta)\prod_{i=1}^N p(D_i|\theta)$ . In the decentralized paradigm, agents iteratively update local beliefs by alternating a Bayesian update step (processing local likelihoods) with a consensus or fusion step (assimilating neighbors' beliefs). Variants may also support state- and action-dependent inference (as in multi-agent reinforcement learning), distributed state estimation, or decentralized multi-modal posterior fusion, depending on the application domain (Lalitha et al., 2019, Wu et al., 2023, Wang et al., 2020).

Network topology, frequently encoded in a weight matrix $W$ (e.g., row-stochastic, symmetric, matching the edge set), influences the rate and structure of information flow, consensus attainment, and statistical efficiency (Wu et al., 2023, Dedecius et al., 2012).

2. Algorithmic Frameworks and Protocols

A spectrum of algorithmic paradigms has emerged, tailored to different inference settings, statistical models, and computational requirements:

Consensus-Driven Distributed Bayes: Agents alternate local Bayes updates $q_i(\theta)\propto q_i^{\text{old}}(\theta)p(D_i|\theta)$ with consensus steps, typically geometric means, such as $q_i^{\text{new}}(\theta) \propto \prod_j q_j^{\text{old}}(\theta)^{W_{ij}}$ (Wu et al., 2023, Lalitha et al., 2019). Under suitable assumptions, this protocol ensures consistency and achieves parametric rates matching centralized Bayesian updates, modulo a communication-variance penalty determined by the spectral gap of $W$ .
Distributed Markov Chain Monte Carlo (MCMC): Variants such as decentralized Stochastic Gradient Langevin Dynamics (DSGLD), decentralized Hamiltonian Monte Carlo (HMC), and their Metropolis-adjusted versions allow agents to simulate posterior samples in parallel via local updates, consensus mixing steps, and stochastic noise injection (Gürbüzbalaban et al., 2020, Kungurtsev et al., 2021, Barbieri et al., 2022, Parayil et al., 2020). Communication protocols include synchronous and asynchronous mixing, often leveraging the doubly-stochastic mixing matrices.
Variational Bayesian Protocols: Agents fit tractable local approximations to the posterior (e.g., mean-field exponential family) via local variational inference, then aggregate local posteriors with network consensus mechanisms (e.g., log-linear parameter averaging, ADMM, or gossip-driven token passing). Decentralized coordination may be realized by stochastic variational algorithms with global consensus constraints (Anwar et al., 2018, Gong et al., 2021, Campbell et al., 2014).
Functional and Marginal Consensus: In high-dimensional models or sensor networks where agents are sensitive only to subsets of global variables, functional mirror-descent approaches update full or marginal densities over relevant variables, with geometric-mean consensus and local Bayesian data incorporation steps (Paritosh et al., 2023).
Market-Based and Game-Theoretic Protocols: Probabilistic inference is mapped to general-equilibrium economics, where agents are economic participants whose buying and selling behavior over "contingent goods" implements the probabilistic constraints of a Bayesian network, and market prices equilibrate to marginal and conditional probabilities (Pennock et al., 2013). Similarly, multi-agent “language-game” protocols use Metropolis-Hastings exchanges to achieve decentralized knowledge fusion (Matsui et al., 13 Apr 2025).

The following table summarizes representative frameworks and their core protocol features:

Framework (arXiv)	Inference Core	Consensus/Communication
(Lalitha et al., 2019, Wu et al., 2023)	Bayes rule, exponential fam.	Log-linear consensus, peer-to-peer
(Gürbüzbalaban et al., 2020, Kungurtsev et al., 2021)	Langevin/HMC sampling	Mixing/averaging, neighbor exchange
(Anwar et al., 2018, Gong et al., 2021)	Variational inference	Parameter averaging, ADMM, gossip
(Paritosh et al., 2023)	Mirror-descent on PDFs	Geometric-mean consensus
(Pennock et al., 2013)	Market equilibrium	Price adjustment, excess demand

3. Theoretical Analysis: Consistency, Efficiency, and Trade-Offs

Decentralized Bayesian inference methodologies have been analyzed for both statistical consistency and communication/statistical trade-offs.

Posterior Consistency and Parametric Rates: Under connectivity and regularity assumptions, distributed Bayesian recursions provably yield posterior consistency and contraction at the centralized parametric rate, i.e., contraction rate $\epsilon_{N,t} \asymp (Nt)^{-1/2}$ when $N$ nodes each receive $t$ samples (Wu et al., 2023, Lalitha et al., 2019). The Bayesian Bernstein–von Mises theorem extends, ensuring credible intervals have asymptotically correct frequentist coverage.
Communication/Statistical Efficiency Trade-Off: The error between the joint distributed posterior at each agent and the “ideal” global Bayes posterior is controlled by the mixing speed of the network consensus operator, quantified via the second-largest eigenvalue modulus $\lambda_2(W)$ . The per-agent contraction rate is slowed by a $N \log N/(1-\lambda_2(W))$ term, formalizing the trade-off between communication bandwidth, topology, network size, and inferential accuracy (Wu et al., 2023). Time-varying, asynchronous, and random-graph extensions have been analyzed, revealing phase transitions in convergence speed (Wu et al., 2023, Lalitha et al., 2019).
Convergence of MCMC and SVI Methods: Decentralized MCMC algorithms maintain ergodicity and sample correctly from the global posterior up to discretization and consensus error, with error controlled by the step size, consensus mixing, and network topology (Gürbüzbalaban et al., 2020, Kungurtsev et al., 2021, Barbieri et al., 2022). For variational methods, convergence to stationary ELBO points is ensured under convexity or, empirically, observed for nonconvex models (Anwar et al., 2018, Gong et al., 2021).
Special Structures: In applications where agents only care about marginals over small variable subsets (sensor networks, multirobot SLAM, cooperative mapping), performance gains are achieved by running marginal consensus protocols, passing only low-dimensional messages, and maintaining global consistency in a storage-aware fashion (Paritosh et al., 2023, Dagan et al., 2022, Dagan et al., 2021).

4. Domain-Specific Applications and Model Instantiations

Decentralized Bayesian inference underpins a wide range of networked machine learning, estimation, and control tasks:

Federated and Peer-to-Peer Learning: Agents collaboratively train predictive models (e.g., neural networks, Bayesian linear/logistic regression) from local datasets, quantifying epistemic uncertainty and producing well-calibrated predictions (Lalitha et al., 2019, Wu et al., 2023, Barbieri et al., 2022, Gürbüzbalaban et al., 2020).
Sensor Networks and State Estimation: Dynamic Bayesian diffusion estimation enables reliable recursive state estimation and parameter identification over sensor networks, with each node using both its own and neighbors' data in incremental and spatial consensus steps, guaranteeing asymptotic agreement (Dedecius et al., 2012).
Multi-Agent Coordination and Theory-of-Mind: In multi-agent MDPs, decentralized Bayesian inverse planning (Bayesian Delegation) enables agents to reason about joint task allocation and hidden intentions, coordinating both cooperative and divide-and-conquer strategies with human-level intent inference (Wang et al., 2020).
Bayesian Multi-Armed Bandits: Agents solve decentralized stochastic bandit problems, using Bayesian posterior updates (Thompson Sampling, Bayes-UCB) and geometric-mean consensus to achieve optimal regret up to a network penalty (Lalitha et al., 2020).
Distributed Compressed Sensing, Kernel Learning, and Gaussian Processes: Decentralized variational Bayesian approaches recover globally sparse signals exploiting both inter- and intra-signal coupling, or compute Gaussian process predictions via online Bayesian model averaging, with consensus-based parameter updates replacing centralized aggregation (Torkamani et al., 2020, Llorente et al., 7 Feb 2025).
Multiagent MARL and Structural Learning: Decentralized variational Bayes enables agents to jointly learn not only policies but also latent communication topologies or variable subgraph structures via variational inference over discrete masks (Bayesian ego-graph models, BayesG) (Duan et al., 20 Sep 2025).

5. Heterogeneity, Privacy, and Robustness Mechanisms

Decentralized Bayesian inference is equipped to handle structural and statistical heterogeneity, privacy, and practical communication limitations:

Model Heterogeneity and Partial Overlap: Factor-graph-based decentralized data fusion exploits conditional independence, representing the global joint posterior as the product of local factor graphs, with message passing and explicit or conservative (covariance intersection) correction for uncertain correlations in overlapping variables. Formal channel-filter constructs exist for linear-Gaussian trees, whereas nonlinear and cyclic networks use conservative fusion and message deflation to maintain consistency (Dagan et al., 2022, Dagan et al., 2021).
Approximate and Efficient Fusion: For large, structurally symmetric, or multimodal models, decentralized merging of local variational posteriors requires permutation correction (e.g., AMPS alignment), symmetrization, and post-fact alignment to avoid degeneracy in approximate inference (Campbell et al., 2014).
Asynchronous and Time-Varying Networks: Protocols based on gossip or ADMM remain correct and robust under asynchronous communication, peer dropouts, or time-varying network links, provided the union graph remains connected over time (Gong et al., 2021, Anwar et al., 2018).
Privacy and Data Protection: Communication typically consists only of natural parameters, sufficient statistics, or predictive distributions, never the raw data, supporting privacy requirements in federated or regulated ecosystems (Lalitha et al., 2019, Gong et al., 2021).
Unlearning and Statistical Rights: Federated Bayesian protocols can remove the contribution of any agent’s data post-training, by subtracting local sufficient statistics via a transactional “token” walk, thereby achieving exact unlearning without retraining (Gong et al., 2021).

6. Empirical Performance and Limitations

Numerical and real-world experiments consistently show that decentralized Bayesian inference protocols:

Match the predictive accuracy and uncertainty quantification of centralized Bayesian inference, up to penalties dictated by the network spectral gap and communication frequency (Parayil et al., 2020, Wu et al., 2023, Lalitha et al., 2020).
Scale to large networks and high dimensions with moderate per-agent memory and communication costs, often reducing global communication by multiple orders of magnitude via partitioned, marginal, or factorized computations (Dagan et al., 2021, Paritosh et al., 2023, Llorente et al., 7 Feb 2025).
Exhibit resilience to network sparsity, agent/node failure, and asynchronous operation, with only a quantifiable slowdown in consensus and convergence (Barbieri et al., 2022, Anwar et al., 2018, Wu et al., 2023).
In multi-agent collaboration and theory-of-mind tasks, outperform heuristic methods in both cooperative and ad-hoc settings and produce human-like inference patterns (Wang et al., 2020).

However, certain limitations remain:

Graph topology and spectral gap fundamentally bound the rate of convergence and variance shrinkage.
Non-convex models and nonlinear likelihoods introduce nontrivial approximation errors absent in centralized or convex regimes.
Some protocols require non-negligible communication per iteration, particularly those based on full-posterior exchange or per-dimension parameter averaging.
In multimodal or permutation-symmetric models, naive merging of posteriors without symmetrization can result in predictive degeneracy (Campbell et al., 2014).

7. Outlook and Research Directions

Open research questions and future directions in decentralized Bayesian inference include:

Designing scalable protocols for online, non-convex, and deep probabilistic models, with provable non-asymptotic guarantees.
Developing message-passing or consensus algorithms robust to malicious, adversarial, or Byzantine agents, incorporating privacy-by-design mechanisms.
Extending market-based and game-theoretic frameworks to support mechanism design, belief integration, and resource allocation beyond standard inference (Pennock et al., 2013).
Incorporating communication-efficient quantization, adaptive bandwidth allocation, and sparsification into high-dimensional decentralized Bayesian learning (Kungurtsev et al., 2021, Barbieri et al., 2022).
Exploring principled methods for decentralized Bayesian unlearning, continual learning, or intent inference in rapidly changing environments (Gong et al., 2021, Wang et al., 2020).
Establishing tighter lower bounds and uncertainty quantification trade-offs under general network and data partitioning constraints.

Decentralized Bayesian inference is thus a unifying paradigm at the intersection of statistics, control, networking, and machine learning, providing rigorous, interpretable, and robust frameworks for inference and decision-making in large-scale, distributed, and heterogeneous systems.