Fed-MARL: Federated Multi-Agent Reinforcement Learning

• Fed-MARL is a distributed reinforcement learning framework in which multiple agents perform local updates and periodically synchronize through model averaging.
• It encompasses architectures like federated RL, cooperative federated MARL, MARL-assisted FL, and privacy-compatible components to address system heterogeneity and communication constraints.
• The approach emphasizes communication design, aggregation strategies, and privacy-preserving mechanisms to achieve robust, personalized control across diverse application domains.

Federated Multi-Agent Reinforcement Learning (Fed-MARL) denotes a family of architectures in which multiple agents or clients perform local reinforcement-learning updates from decentralized experience, exchange model information rather than raw trajectories, and periodically aggregate that information to improve a shared or partially shared control policy. The term does not identify a single universally accepted formalism. Recent survey treatment places “Federated RL” alongside cooperative decentralized RL and noncooperative MARL, and uses it to denote distributed RL under privacy preservation, parameter or gradient sharing, and explicit heterogeneity across environments or agents (Cheruiyot et al., 8 Jul 2025).

1. Scope, taxonomy, and definitional boundaries

The contemporary Fed-MARL literature spans several distinct technical regimes that are often conflated. In the narrowest sense, it refers to federated optimization of RL policies or value functions across multiple agents or environments. In a broader sense, it also includes cooperative MARL systems whose training is periodically synchronized by federated averaging, and adjacent methods in which MARL is used to control federated optimization itself rather than to solve a shared environment task (Zhu et al., 2024).

A useful synthesis is that the literature contains at least four recurring patterns. First, there are federated RL formulations in which multiple agents interact with separate MDPs and optimize an average return or shared fixed point through periodic parameter averaging, as in federated stochastic approximation, federated policy gradient, and federated value learning (Khodadadian et al., 2022). Second, there are cooperative federated MARL systems in which each client trains a decentralized multi-agent policy locally and synchronizes parameters, as in federated QMIX, DRQN-based edge control, and hierarchical federated actor-critic for traffic signals (Chen et al., 2023). Third, there are MARL-assisted federated-learning methods where multi-agent control is embedded inside the FL optimizer, for example to adapt client-specific proximal terms under data heterogeneity (Sahoo et al., 2024). Fourth, there are privacy-compatible MARL components that are not themselves federated optimization algorithms but are explicitly motivated by federated or privacy-constrained settings, such as local-information-only agent modeling (Fang et al., 2023).

Subfamily	Core idea	Representative papers
Federated RL	Periodic aggregation of local value or policy updates across agents/environments	(Gatsis, 2021, Khodadadian et al., 2022, Zhu et al., 2024, Jin et al., 2024)
Cooperative federated MARL	Local training of multi-agent policies with federated synchronization	(Chen et al., 2023, Fu et al., 7 Apr 2025, Andong et al., 12 Sep 2025, Hu et al., 2021)
MARL for FL control	Multi-agent controller adjusts FL optimization variables	(Sahoo et al., 2024)
Federated-compatible MARL components	MARL modules designed for privacy-limited information access	(Fang et al., 2023)

The boundary of the field is also shaped by terminology mismatches. “Federated control” in early hierarchical MARL work refers to semi-decentralized coordination with private local information, not federated optimization in the modern FL sense (Kumar et al., 2017). The supplied material also explicitly identifies the arXiv entry “A Multi-agent Reinforcement Learning Approach for Efficient Client Selection in Federated Learning” as irrelevant to technical Fed-MARL because the provided document is an AAAI formatting template rather than a scientific contribution and should therefore be excluded from a technical corpus (Zhang et al., 2022).

2. Canonical architectural patterns

A recurring architectural template is local RL training plus periodic model averaging. In federated stochastic approximation, each agent maintains a local iterate $x_t^i$, performs $K$ local updates under Markovian noise, and then synchronizes through

$$x_{t+1}^i = \bar x_{t+1} \triangleq \frac{1}{N}\sum_{j=1}^N x_{t+1}^j,\qquad \forall i\in[N].$$

This pattern subsumes federated on-policy TD, off-policy TD, and $Q$-learning under a common fixed-point view (Khodadadian et al., 2022). A related policy-optimization instance defines

$$J(\theta) \triangleq \frac{1}{N}\sum_{i=1}^N J_i(\theta),$$

and performs $H$ local policy-gradient steps between communication rounds, with a server-side correction term to mitigate client drift before aggregation (Zhu et al., 2024).

In cooperative Fed-MARL proper, local models are themselves multi-agent value decompositions or recurrent controllers. In model-aided FedQMIX for multi-UAV data harvesting, each UAV client trains a full local QMIX model in its own learned simulator and synchronizes by uniform parameter averaging,

$$\theta \leftarrow \frac{1}{I}\sum_{i\in\mathcal I}\theta^i,$$

every $N_{\text{freq}}=50$ local episodes (Chen et al., 2023). In 6G edge resource management, each edge agent trains a local DRQN under partial observability and periodically replaces its local and target parameters with a global aggregate after every $K$ episodes (Andong et al., 12 Sep 2025). In adaptive traffic signal control, local A2C policies $w_n=(\theta_n,\phi_n)$ are trained per intersection and then combined either by global FedAvg, by cluster-wise FedAvg, or by personalized agent-specific mixtures (Fu et al., 7 Apr 2025).

A second pattern is server-side optimization control within federated learning. FedMRL places one cooperative QMIX-associated agent per client on the server, uses local client descriptors $K$0, and outputs client-specific proximal coefficients $K$1 that modify each client’s local FL objective. In that design, MARL does not train decentralized environment policies; it tunes optimizer hyperparameters inside the federated loop (Sahoo et al., 2024).

A third pattern is region- or neighborhood-restricted federation. In federated control over a single MDP, each learner accesses only a subset $K$2 of the state space, and communication transmits boundary-value information rather than raw trajectories. In large wireless networks, APs learn only from local effective regions and overlapping neighborhoods, while federated parameter sharing is used to create consistency across agents without centralized real-time control (Jin et al., 2024).

3. Communication, aggregation, privacy, and personalization

Communication design is a central organizing principle of Fed-MARL. A communication survey frames practical MARL systems around the questions of who communicates, when, and what/rate to communicate, and treats federated learning as a domain where repeated model exchange, limited bandwidth, staleness, and privacy trade-offs fundamentally determine system behavior (Liu et al., 14 Nov 2025). This perspective aligns closely with federated RL at the edge, where communication cost is placed directly into the performance criterion,

$K$3

and transmission is triggered only when a local update is sufficiently informative (Gatsis, 2021).

Aggregation is not limited to uniform averaging. FedMRL augments standard size-based FL aggregation with a server-side self-organizing map and cosine-similarity weighting, and couples that with a fairness term

$K$4

in the local objective to reduce bias toward dominant clients (Sahoo et al., 2024). Hierarchical federated traffic control replaces one global average by either hard grouping,

$K$5

or personalized mixtures,

$K$6

with $K$7 derived from local reward comparisons and parameter distances (Fu et al., 7 Apr 2025). In large wireless networks, plain FedAvg is reported to degrade because full averaging erases local “personality,” and a transfer-based variant with CORAL is introduced to preserve specialization (Hu et al., 2021).

Privacy mechanisms range from architectural data locality to explicit secure aggregation. FERMI-6G uses pairwise masks derived from ECDH-based shared secrets: $K$8 so that

$K$9

while individual updates remain hidden from a semi-honest aggregator (Andong et al., 12 Sep 2025). At the MARL-module level, FAM removes the need to access other agents’ local observations, actions, rewards, hidden states, or policy parameters and instead learns latent representations of others from local “facts,” namely next observations and rewards (Fang et al., 2023).

Communication frequency is itself a control variable. In federated wireless coordination, synchronization every $$x_{t+1}^i = \bar x_{t+1} \triangleq \frac{1}{N}\sum_{j=1}^N x_{t+1}^j,\qquad \forall i\in[N].$$0 local steps trades backbone traffic against faster reduction of inter-agent uncertainty (Hu et al., 2021). In Fast-FedPG, communication happens after $$x_{t+1}^i = \bar x_{t+1} \triangleq \frac{1}{N}\sum_{j=1}^N x_{t+1}^j,\qquad \forall i\in[N].$$1 local policy-gradient steps, and the method explicitly uses a round-start global gradient estimate to correct local drift (Zhu et al., 2024). In asynchronous federated settings, the communication survey emphasizes trust-aware scheduling, bounded staleness, compressed or sketched updates with error feedback, and hierarchical edge aggregation as major design directions rather than afterthoughts (Liu et al., 14 Nov 2025).

4. Mathematical formulations and theoretical guarantees

The strongest theory in the area is concentrated in federated RL rather than in fully interactive stochastic-game MARL. A common federated objective for multi-task RL is

$$x_{t+1}^i = \bar x_{t+1} \triangleq \frac{1}{N}\sum_{j=1}^N x_{t+1}^j,\qquad \forall i\in[N].$$2

where each $$x_{t+1}^i = \bar x_{t+1} \triangleq \frac{1}{N}\sum_{j=1}^N x_{t+1}^j,\qquad \forall i\in[N].$$3 is the long-term discounted objective of a local MDP, and the goal is a single shared policy that performs well on average across environments (Zhu et al., 2024). Under a gradient-domination condition, Fast-FedPG proves linear convergence with exact gradients and noisy-gradient rates with linear speedup in the number of agents, yielding

$$x_{t+1}^i = \bar x_{t+1} \triangleq \frac{1}{N}\sum_{j=1}^N x_{t+1}^j,\qquad \forall i\in[N].$$4

under the stated parameter choices and no heterogeneity-induced bias in the limit point (Zhu et al., 2024).

A second theoretical line studies fixed-point learning with Markovian noise. FedSAM defines local stochastic approximation recursions with periodic averaging and proves a bound of the form

$$x_{t+1}^i = \bar x_{t+1} \triangleq \frac{1}{N}\sum_{j=1}^N x_{t+1}^j,\qquad \forall i\in[N].$$5

which leads to

$$x_{t+1}^i = \bar x_{t+1} \triangleq \frac{1}{N}\sum_{j=1}^N x_{t+1}^j,\qquad \forall i\in[N].$$6

for federated on-policy TD, off-policy TD, and tabular $$x_{t+1}^i = \bar x_{t+1} \triangleq \frac{1}{N}\sum_{j=1}^N x_{t+1}^j,\qquad \forall i\in[N].$$7-learning when $$x_{t+1}^i = \bar x_{t+1} \triangleq \frac{1}{N}\sum_{j=1}^N x_{t+1}^j,\qquad \forall i\in[N].$$8 (Khodadadian et al., 2022). The conceptual significance is that the $$x_{t+1}^i = \bar x_{t+1} \triangleq \frac{1}{N}\sum_{j=1}^N x_{t+1}^j,\qquad \forall i\in[N].$$9 variance reduction survives Markovian trajectory noise rather than requiring i.i.d. gradients.

A third line makes communication explicit in the learning criterion. Federated RL at the edge studies approximate value-function learning for a fixed policy and an event-triggered communication rule,

$Q$0

with a theorem bounding the combined objective of communication rate and final learning error in the two-agent case (Gatsis, 2021).

State-access heterogeneity has its own theory. Federated control in MDPs defines leakage probabilities

$Q$1

for agents restricted to regions $Q$2, proves that the induced federated Bellman operator has unique fixed point $Q$3, and derives per-agent sample-complexity reductions that become linear in $Q$4 under uniform workload (Jin et al., 2024). At a more modeling-oriented level, mean-field control with a doubly stochastic interaction matrix $Q$5 approximates cooperative non-uniform MARL with error

$Q$6

and the associated NPG algorithm attains $Q$7 accuracy with sample complexity $Q$8; this is not a federated optimization algorithm, but it is directly relevant to graph-structured distributed interaction (Mondal et al., 2022).

Two caveats recur throughout the literature. First, a theorem summarized in the 2025 survey states that there need not exist a universally optimal single global policy across all initial-state distributions in heterogeneous federated RL. Second, the same survey states that naive QAvg and PAvg converge to suboptimal solutions and that the suboptimality depends on environment heterogeneity (Cheruiyot et al., 8 Jul 2025).

5. Application domains and empirical patterns

The application literature is broad, but its empirical lessons are consistent: heterogeneity-aware aggregation and privacy-preserving synchronization can materially improve distributed control.

In medical imaging federated learning with embedded MARL control, FedMRL addresses severe non-IIDness by combining a fairness-aware local objective, server-side QMIX-based proximal adaptation, and SOM-based adaptive aggregation. On ISIC-2018 it reports ACC $Q$9 for FedMRL versus $$J(\theta) \triangleq \frac{1}{N}\sum_{i=1}^N J_i(\theta),$$0 for FedAvg, $$J(\theta) \triangleq \frac{1}{N}\sum_{i=1}^N J_i(\theta),$$1 for FedProx, $$J(\theta) \triangleq \frac{1}{N}\sum_{i=1}^N J_i(\theta),$$2 for FedNova, and $$J(\theta) \triangleq \frac{1}{N}\sum_{i=1}^N J_i(\theta),$$3 for FedBN; on Messidor it reports ACC $$J(\theta) \triangleq \frac{1}{N}\sum_{i=1}^N J_i(\theta),$$4 versus $$J(\theta) \triangleq \frac{1}{N}\sum_{i=1}^N J_i(\theta),$$5, $$J(\theta) \triangleq \frac{1}{N}\sum_{i=1}^N J_i(\theta),$$6, $$J(\theta) \triangleq \frac{1}{N}\sum_{i=1}^N J_i(\theta),$$7, and $$J(\theta) \triangleq \frac{1}{N}\sum_{i=1}^N J_i(\theta),$$8, respectively (Sahoo et al., 2024).

In multi-UAV IoT data harvesting, model-aided FedQMIX alternates between sparse real-world measurement collection, learned simulator construction, and federated QMIX training. The reported headline result is that the method attains similar data-collection performance to standard QMIX trained on real-world samples while requiring around three orders of magnitude fewer real-world training experiences (Chen et al., 2023).

In 6G edge resource management, FERMI-6G combines decentralized DRQN control with periodic secure aggregation and a multi-objective reward over latency, energy, fairness, reliability, spectral efficiency, energy efficiency, and MAC success rate. The reported metrics are reward $$J(\theta) \triangleq \frac{1}{N}\sum_{i=1}^N J_i(\theta),$$9, reliability $H$0, throughput $H$1 Gbps, energy efficiency $H$2 bits/J, and 50-agent reliability $H$3, outperforming the compared centralized and simpler federated baselines on overall reward, reliability, throughput, and scalability, although not on every individual metric such as spectral efficiency (Andong et al., 12 Sep 2025).

In adaptive traffic signal control, HFRL replaces one global federated model by cluster-wise or personalized aggregation. Across synthetic and real-world networks, the paper reports that FedFomoLight achieves the best travel time and waiting time, that FedClusterLight is typically slightly better than standard FedAvg, and that the induced grouping patterns correlate with topology or demand structure (Fu et al., 7 Apr 2025).

In large-scale wireless coordination, federated actor-critic methods outperform isolated local RL under PCP user distributions, and personalization is again decisive. The paper reports that ACER FedAvg and especially ACER FedTrans achieve high QoS and faster convergence in JT-CoMP clustering, while FedAvg can eventually degrade because full averaging destroys local personality; FedTrans is introduced precisely to retain common knowledge without erasing local specialization (Hu et al., 2021).

6. Limitations, controversies, and open directions

A central controversy is that many papers cited under the Fed-MARL label are not, in the strict sense, federated training of interacting multi-agent policies in a shared Markov game. Federated RL at the edge studies distributed value estimation for a fixed policy rather than multi-agent policy learning (Gatsis, 2021). Fast-FedPG studies multiple agents on separate MDPs with heterogeneous rewards, not joint interaction in a shared environment (Zhu et al., 2024). FedMRL uses server-side MARL to control federated optimization rather than federated optimization to train MARL agents (Sahoo et al., 2024). Federated control in MDPs studies multiple learners solving one global MDP under region restrictions rather than a decentralized stochastic game (Jin et al., 2024). The term therefore functions more as an umbrella than as a settled formal category.

A second limitation is uneven methodological rigor. Some systems papers provide strong engineering pipelines but leave core RL details underspecified. FedMRL does not provide the actual QMIX monotonic mixing equation, Bellman target, replay-buffer details, or a documented reconciliation of QMIX with its continuous proximal action $H$4, and the visible experiments do not isolate the separate effects of MARL-based proximal adaptation, fairness loss, and SOM weighting (Sahoo et al., 2024). FERMI-6G omits the explicit Bellman target, DRQN TD loss, and prioritized-replay priority rule, and offers no theoretical guarantees beyond the secure aggregation algebra (Andong et al., 12 Sep 2025). Model-aided FedQMIX does not specify exact utility-network and mixing-network architectures, replay size, batch size, or exact communication payload sizes in the supplied text (Chen et al., 2023).

A third limitation is the persistence of heterogeneity. The 2025 survey emphasizes that no single global policy is universally optimal across all initial-state distributions, and that naive federated averaging is suboptimal under environment heterogeneity (Cheruiyot et al., 8 Jul 2025). This has driven a shift toward grouped, similarity-aware, and personalized aggregation: SOM-based weights in FL, cluster-wise traffic-signal federation, FedTrans-style personality preservation in wireless networks, and personalized local embeddings in survey coverage of federated RL (Sahoo et al., 2024).

The main open directions are correspondingly clear. The communication survey advocates event-triggered uploads, synchronization schedules tailored to different components, trust-aware and Byzantine-resilient aggregation, compressed or sketched updates with error feedback, hierarchical edge aggregation, and bounded staleness (Liu et al., 14 Nov 2025). The MARL survey highlights stronger theory for deep federated RL, heterogeneity-aware aggregation beyond naive averaging, theoretical foundations for personalization, privacy-utility trade-offs, adaptive communication, and standardized benchmarks that jointly capture communication cost, convergence, privacy, and robustness (Cheruiyot et al., 8 Jul 2025). A plausible implication is that the next phase of Fed-MARL research will be defined less by the mere presence of federation and more by how communication control, personalization, robustness, and partial observability are co-designed inside the learning system.