Decentralized Machine Learning with Centralized Performance Guarantees via Gibbs Algorithms

Abstract: In this paper, it is shown, for the first time, that centralized performance is achievable in decentralized learning without sharing the local datasets. Specifically, when clients adopt an empirical risk minimization with relative-entropy regularization (ERM-RER) learning framework and a forward-backward communication between clients is established, it suffices to share the locally obtained Gibbs measures to achieve the same performance as that of a centralized ERM-RER with access to all the datasets. The core idea is that the Gibbs measure produced by client~$k$ is used, as reference measure, by client~$k+1$. This effectively establishes a principled way to encode prior information through a reference measure. In particular, achieving centralized performance in the decentralized setting requires a specific scaling of the regularization factors with the local sample sizes. Overall, this result opens the door to novel decentralized learning paradigms that shift the collaboration strategy from sharing data to sharing the local inductive bias via the reference measures over the set of models.

• The paper proposes a novel decentralized learning paradigm by exchanging Gibbs measures, proving performance parity with centralized ERM-RER solutions.
• It derives closed-form expressions and scaling conditions that enable recursive update of reference measures to fully incorporate all local datasets.
• The approach balances privacy and performance, although practical challenges remain for efficiently communicating high-dimensional probability measures.

Decentralized Learning with Centralized Guarantees via Gibbs Algorithms

Problem Statement and Context

The paper addresses a longstanding challenge in decentralized machine learning: how to close the gap in performance versus centralized learning when data sharing is infeasible due to privacy, bandwidth, or regulatory constraints. Traditionally, decentralized approaches fail to achieve the same empirical risk minimization (ERM) performance as centralized ones, since clients cannot access each other's raw datasets. The work proposes an alternative collaboration paradigm by shifting the communication from datasets or model parameters to probability measures—specifically, Gibbs measures derived from ERM problems with relative-entropy regularization (ERM-RER).

Technical Development

The authors formalize the decentralized learning scenario with $K$ clients, each possessing a private local dataset, its own loss function, and model space. Core to the proposed approach is defining distributed learning algorithms via conditional probability measures (Gibbs measures) on the model space. Each client's learning algorithm solves an ERM-RER problem:

$$\min_{P} \mathbb{E}_{\theta \sim P}[\text{loss}(\theta)] + \lambda D(P \| Q),$$

where the client-specific reference measure $Q$ and regularization parameter $\lambda$ structure the solution as a Gibbs probability measure.

The key technical insight is that, through a specific forward-backward peer-to-peer communication protocol, clients transmit their locally computed Gibbs measures instead of raw data. Client $k$ provides its posterior Gibbs measure to client $k+1$, which uses it as the reference measure in its own regularized ERM problem. This construction embeds the inductive bias and information from previous clients without explicit data sharing, establishing a nested structure where each reference measure recursively encodes the history of all prior clients.

A closed-form expression for the decentralized Gibbs measure is derived and proved, up to normalization, to match the centralized Gibbs measure that would be computed if all datasets were known—a fact dependent on precise scaling of the regularization factors in direct proportion to the local and global sample sizes.

Theoretical Results

The paper's primary contribution is the rigorous establishment of conditions under which decentralized systems can guarantee centralized performance using only Gibbs measure exchange, not data. The main theorems assert the following:

• If each client $k$ uses as regularization parameter $\lambda_k = n_k / n_0 \cdot \lambda_0$ (with $n_k$ the local, $n_0$ the total number of samples), and the reference measures are set recursively as described, then the final decentralized Gibbs measure equals the centralized Gibbs measure obtained by joint ERM-RER on the aggregate dataset.
• The equality holds pointwise for all models within the support of the base reference measure $$\min_{P} \mathbb{E}_{\theta \sim P}[\text{loss}(\theta)] + \lambda D(P \| Q),$$0.

The theorems are justified via a careful chain of results involving variational principles, change-of-measure techniques, and Radon-Nikodym derivative calculus. In particular, the proof establishes that the iterative update of reference measures propagates all necessary dataset-dependent information, matching the centralized optimization trajectory.

Numerical Implications and Contrasts

The paper claims that, for any number of clients and for arbitrary local dataset divisibility, decentralized algorithms using the prescribed protocol and regularization scaling achieve loss and risk minimization identical to the centralized ERM-RER solution given all the data. This is a strong claim, in contradiction with prior assertions that decentralized and federated learning inherently incur a performance gap versus centralized regimes due to information fragmentation.

Unlike parameter averaging or traditional federated averaging—which only loosely couple client updates and almost always underperform relative to the pooled-data solution—the Gibbs-measure-based approach encodes not only point estimates but the distributional effects of local data, facilitating full aggregation of inductive biases.

However, practical limitations are noted: communicating exact probability measures may be expensive in terms of bandwidth and encoding complexity, especially when the model space is high-dimensional; finite-rate constraints induce distortions not analyzed in the current work.

Practical and Theoretical Implications

This methodology redefines the boundary between privacy and utility in collaborative machine learning. By requiring only Gibbs measure sharing, it offers enhanced privacy by not exposing raw data, and it is also compatible with frameworks ensuring information-theoretic privacy (e.g., differential privacy via measure perturbations).

From a theoretical standpoint, the equivalence between decentralized and centralized learning under this paradigm opens opportunities for redesigning collaborative protocols in domains where communication constraints or local data governance prohibit centralized pooling. It also deepens the connection between variational inference, PAC-Bayesian theory, and decentralized optimization by framing the entire collaborative process as information transport via probability measures rather than direct transmission of input variables.

This perspective facilitates future extensions, such as supporting heterogeneous loss functions, integrating measure quantization for practical transmission, designing robust protocols against adversarial measure distortion, and exploring trade-offs between communication efficiency and performance guarantees.

Conclusion

The paper provides a rigorous framework whereby decentralized learning can attain centralized ERM-RER performance by exchanging Gibbs probability measures. The results hinge on the recursive construction of reference measures and specific regularization scaling, enabling each client to act on local data while fully propagating dataset information throughout the network. This positions Gibbs-measure-based protocols as a new paradigm for privacy-preserving, high-utility decentralized machine learning, with open challenges remaining in efficient measure communication and robustness to finite-rate distortions (2604.20492).