Federated Multi-Agent Reinforcement Learning

• FMARL is a paradigm that integrates federated learning and multi-agent reinforcement learning to collaboratively solve distributed sequential decision problems without sharing raw data.
• It employs aggregation techniques such as FedAvg, clustering, and gradient consensus to handle non-i.i.d. data and improve convergence rates across agents.
• FMARL incorporates privacy-preserving mechanisms, including encryption and differential privacy, to secure local updates against adversarial threats while maintaining scalability.

Federated Multi-Agent Reinforcement Learning (FMARL) is a paradigm that integrates principles from federated learning and multi-agent reinforcement learning. It enables multiple agents, each controlling its own environment or subsystem and possessing only local observations and interactions, to collaboratively solve a global sequential decision problem via distributed and communication-efficient protocols. FMARL is defined by the absence of raw data or trajectory sharing among agents; instead, learning occurs through local RL updates and periodic exchange of model parameters or summaries, often coordinated by a central server or through decentralized consensus. This approach is designed to address privacy, scalability, and communication bottlenecks inherent in large-scale, distributed, and heterogeneous systems.

1. Mathematical Formulation and Foundational Principles

The canonical FMARL setup considers $N$ agents, each solving an individual (typically nonidentical) MDP or Dec-POMDP $$\mathcal{M}_i=(\mathcal{S}_i,\mathcal{A}_i,P_i,R_i,\gamma)$$; agents have private transitions and reward functions. Each agent maintains a local policy $\pi_{\theta_i}$ parameterized by $\theta_i$ and aims to maximize its expected discounted return via local RL. A global objective frequently takes the form

$\min_{\theta}~\sum_{i=1}^N w_i L_i(\theta)$

where $L_i$ denotes the local RL loss for agent $i$, and the aggregation operator (e.g., FedAvg) synchronizes $\theta_i$ into a global parameter vector $\bar{\theta}$ (Cheruiyot et al., 8 Jul 2025).

In cooperative Dec-POMDP settings, agents seek to maximize a joint return over global state transitions, but each observes only local observations $o_i^t$ and takes actions $$\mathcal{M}_i=(\mathcal{S}_i,\mathcal{A}_i,P_i,R_i,\gamma)$$0. The communication protocol is designed to support local updates and periodic aggregation without requiring transmission of raw samples, thereby preserving privacy and reducing bandwidth use (Fu et al., 7 Apr 2025, Zhu et al., 2020).

2. Frameworks and Aggregation Algorithms

FMARL methodologies encompass several principal aggregation and communication frameworks:

• FedAvg-style Aggregation: Local agents perform RL updates on neural or tabular models and periodically average their parameters, typically in a star topology with a central server. This setup generalizes to deep RL (DQNAvg, DDPGAvg) (Cheruiyot et al., 8 Jul 2025).
• Clustered or Personalized Aggregation: Techniques such as dynamic clustering (FedClusterLight) or personalized weighting (FedFomoLight) are used to resolve non-i.i.d. heterogeneity, clustering agents with similar environments and aggregating within clusters (Fu et al., 7 Apr 2025). Personalization can also take the form of separate embedding vectors retained locally while sharing a backbone globally.
• Gradient Consensus and Decay: To further reduce communication and noise from stale updates, decay-based weighting of local gradients (Xu et al., 2021) and local peer-to-peer consensus steps before federated aggregation are used. These strategies enable improved convergence with reduced communication frequency.
• Graph-based and Decentralized Federated RL: Approaches like AirFed (Wang et al., 27 Oct 2025) deploy decentralized, reputation-based aggregation, with adaptive quantization and dynamic neighbor selection.

A common theme is the non-transmission of raw trajectories or environment states, instead sharing only model parameters, local value functions, or encrypted statistics to achieve privacy and communication efficiency (Lin et al., 2023, Andong et al., 12 Sep 2025).

3. Theoretical Guarantees, Convergence, and Scalability

Convergence analyses for FMARL, both in tabular and function-approximation settings, rely on assumptions of bounded variance, smoothness, and (for stochastic approximation) ergodicity or Markov mixing (Khodadadian et al., 2022, Doan, 2020). Notable results include:

• Linear Speedup: Under appropriate mixing and contractivity assumptions, federated stochastic approximation achieves $$\mathcal{M}_i=(\mathcal{S}_i,\mathcal{A}_i,P_i,R_i,\gamma)$$1 mean-squared error in $$\mathcal{M}_i=(\mathcal{S}_i,\mathcal{A}_i,P_i,R_i,\gamma)$$2 rounds for $$\mathcal{M}_i=(\mathcal{S}_i,\mathcal{A}_i,P_i,R_i,\gamma)$$3 agents, representing "linear speedup" relative to single-agent RL (Khodadadian et al., 2022).
• Gradient Norm Convergence: In multi-agent actor-critic settings, federated averaging (with tuning for aggregation mixing) yields a uniform bound on the time-average squared gradient norm; full independence (no federation) leads to unbounded gradients (Zhu et al., 2020).
• Regret Bounds: With limited communication (rare synchronization), regret scales as $$\mathcal{M}_i=(\mathcal{S}_i,\mathcal{A}_i,P_i,R_i,\gamma)$$4 for $$\mathcal{M}_i=(\mathcal{S}_i,\mathcal{A}_i,P_i,R_i,\gamma)$$5 parallel agents and communication threshold $$\mathcal{M}_i=(\mathcal{S}_i,\mathcal{A}_i,P_i,R_i,\gamma)$$6; communication rounds can remain $$\mathcal{M}_i=(\mathcal{S}_i,\mathcal{A}_i,P_i,R_i,\gamma)$$7 (Dubey et al., 2021).

Trade-offs exist between communication cost (controlled by aggregation frequency and method), heterogeneity (addressed by clustering or weighting), and convergence speed. Mean-field/FedAvg and parameter decay weighting can mitigate staleness and balance local specialization versus global alignment (Hu et al., 2021, Xu et al., 2021).

4. Heterogeneity, Personalization, and Non-IID Data

Handling agent heterogeneity and non-i.i.d. data is critical for FMARL in real-world deployments. Approaches include:

• Clustering and Optimization-Based Aggregation: HFRL schemes (FedClusterLight, FedFomoLight) dynamically cluster agents or personalize aggregation weights based on compatibility of learned parameters and local reward gains, enabling robust performance under highly non-i.i.d. conditions (Fu et al., 7 Apr 2025).
• Adaptive Weighting and Fairness: Aggregation weights can be adapted through server-side mechanisms such as self-organizing maps (SOMs), which modulate contributions from clients based on similarity metrics, better handling distribution shifts and promoting fairness (Sahoo et al., 2024).
• Personalized Regularization: Adaptive proximal terms, tuned by federated MARL (e.g. via QMIX), stabilize local updates and promote convergence to global optima even when data and objective functions diverge (Sahoo et al., 2024).

Empirical results demonstrate significant performance improvements (e.g., 7–10% latency reduction and higher accuracy) over naive FedAvg in imaging, wireless networking, and traffic management under heterogeneous workloads (Fu et al., 7 Apr 2025, Sahoo et al., 2024).

5. Privacy, Security, and Adversarial Robustness

FMARL is motivated by strong privacy requirements, particularly in edge, IoT, and cross-silo FL scenarios. Privacy-preserving mechanisms include:

• Encryption and Differential Privacy: Local Q-values or parameters are encrypted with Gaussian noise or aggregated via cryptographic protocols (e.g., elliptic curve Diffie–Hellman masking in FERMI-6G) to protect agent data against semi-honest or adversarial aggregators (Lin et al., 2023, Andong et al., 12 Sep 2025).
• Adversarial Robustness: FMARL protocols are susceptible to policy manipulation or model poisoning by adversarial actors. Communication-adaptive aggregation methods (e.g., ComA-FedRL) identify and throttle suspected adversarial agents by dynamically changing their aggregation frequency, retaining near-optimal performance for honest agents (Anwar et al., 2021).
• Decentralized Defense and Reputation: In reputation-based decentralized FMARL (e.g. AirFed), aggregation weights are modulated by peer performance, punishing noncooperative or low-credibility neighbors, further reducing adversarial impact (Wang et al., 27 Oct 2025).

These mechanisms enable FMARL to trade off privacy, security, and utility, though formal privacy–utility tradeoff analyses remain an open subject.

6. Practical Applications and Empirical Results

FMARL has been deployed in numerous domains requiring distributed, privacy-preserving, or communication-limited multi-agent intelligence:

Application Domain	Key FMARL Techniques	Performance Gains / Characteristics
Adaptive Traffic Signals	HFRL (FedClusterLight/FedFomoLight)	7–15% latency/wait reduction, scalable
UAV MEC/Trajectory Opt.	QMIX, AirFed, secure federated QMIX	42.9% cost cut, >99% deadline, privacy
Large-Scale Wireless	Mean-field FMARL, FedAvg, CORAL	30–40% higher QoS, scalable to 20+ AP
Medical Imaging	MARL-tuned per-client regularization, SOM	≥0.9% accuracy gain under dirichlet skew
Edge/IoV Resource Mgmt.	FRL with Dec-POMDP, federated/Central MLP	Higher trade-off, privacy, lower HO
Cross-silo FL Incentives	Actor–critic MARL for provable NE/utility	12–18% long-term payoff improvement

In addition, model-aided FMARL leverages environmental simulation to reduce real-world data requirements by up to three orders of magnitude, matching standard deep MARL performance at a fraction of sample cost (Chen et al., 2023).

7. Limitations, Challenges, and Open Research Questions

Open problems in FMARL include:

• Scalability and Communication Efficiency: Communicating high-dimensional parameters at scale imposes significant overhead. Compression, sparsification, adaptive or event-triggered communication, and peer-to-peer protocols are active areas (Hu et al., 2021, Gatsis, 2021).
• Convergence in Deep Nonlinear Settings: Most theoretical guarantees hold under linearity, convexity, or bounded noise; extending convergence, regret, and privacy guarantees to deep FMARL remains challenging (Khodadadian et al., 2022, Cheruiyot et al., 8 Jul 2025).
• Dynamic and Adversarial Environments: Mechanisms for robust resilience against more sophisticated adversaries and in highly dynamic, nonstationary, or partially observed environments require further development (Anwar et al., 2021, Wang et al., 27 Oct 2025).
• Personalization vs. Aggregation: The balance between global alignment (shared model) and local adaptation (personalized weights/embeddings) is not fully resolved in theory or empirical practice (Cheruiyot et al., 8 Jul 2025, Sahoo et al., 2024).
• Privacy-Utility Tradeoff: Quantitative understanding of how privacy mechanisms (differential privacy, encryption) interact with FMARL utility and sample efficiency is incomplete.

FMARL spans the intersection of distributed optimization, game theory, sequential decision making, and applied privacy, and is an active area of algorithmic, theoretical, and empirical research across communications, control, vision, and management science.