Federated Reinforcement Learning

• Federated reinforcement learning is a framework where multiple agents learn collaboratively by sharing model-side information (e.g., Q-functions, policies) while keeping local interaction data private.
• It incorporates diverse aggregation strategies such as soft updates, consensus-based gradient sharing, and function-space averaging to handle heterogeneous transitions and environments.
• Key applications span swarm robotics, autonomous driving, and teleoperated systems, emphasizing communication efficiency, robustness under client drift, and privacy preservation.

Federated reinforcement learning (FRL) denotes a family of methods in which multiple agents or clients interact with local environments and collaboratively improve a shared policy or value estimate without sharing raw trajectories. In the published literature, federation occurs over model-side objects rather than over experience itself: tabular $Q$-functions, policies, neural-network parameters, gradients, dynamics models, value outputs, or anchor-state predictions are exchanged or aggregated, while local interaction data remain private. The central technical issues are no longer only those of reinforcement learning or federated learning in isolation, but their combination: Markovian sampling, client drift, environment heterogeneity, communication constraints, robustness to failures, and the mismatch between parameter-space averaging and function-space consensus (Jin et al., 2022, Khodadadian et al., 2022, Zhu et al., 2024).

1. Conceptual scope

FRL is not a single algorithmic template. One prominent line studies $n$ agents that share $\mathcal{S}$, $\mathcal{A}$, $R$, and $\gamma$, but differ in transition kernels $\{\mathcal{P}_i\}_{i=1}^n$; the federated objective is then the average discounted return across environments rather than the return of any one client MDP (Jin et al., 2022). A second line keeps agents in the same radio or control environment but lets them have different state spaces, action spaces, and local value functions, with federation implemented through a coordination model that fuses local Q-values rather than through ordinary FedAvg (Zhang et al., 2022). A third line makes the federation target itself explicit in function space: the global Q-function is the weighted average $$Q_{\mathrm{glob}}(s,a)=\sum_{i=1}^N \tau_i Q_i(s,a)$$, and the aggregation mechanism is chosen so that this equation is exact or can be compiled in closed form into each client representation (Hou et al., 27 May 2026).

These patterns are usefully distinguished because they optimize different objects. In transition-heterogeneous FRL, the question is how to learn one policy that performs well across non-identical MDPs. In shared-environment coordination, the question is how to combine heterogeneous local control modules whose decisions interact. In function-space FRL, the question is how to aggregate value functions when parameter averaging is not semantically meaningful.

FRL pattern	Federated object	Representative formulation
Same task, heterogeneous transitions	Averaged $Q$-tables, policies, or network weights	$$\mathcal{M}_i=\langle \mathcal{S},\mathcal{A},R,\mathcal{P}_i,\gamma\rangle$$ with average-return objective (Jin et al., 2022)
Heterogeneous agents in one environment	Local Q-value outputs fused by a global DNN	$n$0 (Zhang et al., 2022)
Function-space federation	Weighted average of local Q-functions	$n$1 (Hou et al., 27 May 2026)

The scope of FRL therefore extends from federated policy evaluation and control to federated coordination and representation fusion. What remains common is that local sequential interaction is private and model-side collaboration is explicit.

2. Canonical training architectures

The simplest FRL architecture is local RL plus periodic server aggregation. In swarm robotics, FLDDPG instantiates this pattern with local actor and critic networks, local replay buffers, periodic weight averaging every $n$2 episodes, and a soft update

$n$3

so that only model weights are exchanged and local specialization is partially preserved (Na et al., 2022). In the reported experiments, the soft update is used with $n$4, and the design choice is justified by the observation that hard replacement can temporarily degrade local performance when local environments differ.

A communication-efficient variant augments periodic server averaging with local consensus. In the independent-RL setting studied for multi-agent control, agents periodically synchronize with a virtual server but also exchange local gradients with neighbors through a consensus step before local updates. The paper formalizes the systems trade-off through a utility

$n$5

where $n$6 is a convergence bound and $n$7 is communication-computation cost, so FRL is cast explicitly as joint optimization of learning quality and resource expenditure rather than of return alone (Xu et al., 2022).

A more radical design is to federate a world model instead of a policy. FEMRL trains local dynamics models on clients, uploads those models to the server, forms an ensemble, distills the ensemble into a student model, and then performs policy optimization with TRPO using fictitious rollouts generated by the learned dynamics rather than additional real-environment interaction (Wang et al., 2021). In this architecture the federated object is the transition model $n$8, not the policy parameters directly. The paper’s central claim is that this redesign is substantially more sample efficient than classic model-free FRL baselines.

Another departure from FedAvg is contextual aggregation. FedFormer keeps local replay and local SAC updates, but replaces ordinary parameter averaging with attention over local and external encoder embeddings. Each agent receives encoder weights from peers, applies all encoders to its own $n$9, forms

$\mathcal{S}$0

passes the resulting token set through a Transformer encoder, and decodes a contextual federated Q-function

$\mathcal{S}$1

Federation therefore occurs through state-conditional representation mixing rather than through a single global average model (Hebert et al., 2022).

3. Heterogeneity, robustness, and theory

Environment heterogeneity is the dominant theoretical theme in modern FRL. In the tabular transition-heterogeneous setting, QAvg and PAvg make the heterogeneity explicit through $\mathcal{S}$2 and $\mathcal{S}$3, where $\mathcal{S}$4 measures transition-kernel deviation and $\mathcal{S}$5 measures gradient disagreement across environments. QAvg converges to the optimal $\mathcal{S}$6-function of an “imaginary environment” with averaged kernel $\mathcal{S}$7, while PAvg obtains a policy whose value is within $\mathcal{S}$8 of the optimum for the federated objective. The same work also shows that a single globally shared policy is inherently suboptimal for individual heterogeneous environments and introduces private environment embeddings as a personalization heuristic that improves both local specialization and few-shot adaptation to unseen environments (Jin et al., 2022).

Actor-critic theory extends this line. In Single-loop Federated Actor Critic, clients interact with heterogeneous MDPs

$\mathcal{S}$9

and optimize the mixture-environment objective

$\mathcal{A}$0

The analysis proves convergence to a near-stationary point rather than exact optimality, with a residual that depends on environment heterogeneity, critic approximation error, and Markovian sampling. It also identifies linear speed-up from federation through $\mathcal{A}$1-type variance reduction terms, while emphasizing that actor-critic coupling and client-specific critic fixed points make FRL actor-critic substantially harder than federated supervised optimization (Zhu et al., 2024).

For value-based FRL under Markovian sampling, the federated stochastic approximation framework of (Khodadadian et al., 2022) proves linear speedup for federated on-policy TD, off-policy TD, and $\mathcal{A}$2-learning. The general bound has the form “bias decay + $\mathcal{A}$3 + synchronization drift + higher-order Markov term”, making explicit that the $\mathcal{A}$4 gain from parallel clients competes with the extra error induced by multiple local updates between synchronization rounds. This is one of the clearest formulations of FRL as stochastic approximation under Markov noise rather than as static-data federated optimization.

Robustness to failures and attacks appears in federated policy gradient. FedPG-BR assumes a trusted server and fewer than half Byzantine clients, i.e. $\mathcal{A}$5, uses a two-stage robust aggregation rule on client gradient estimates, and proves that per-agent trajectory complexity improves from $\mathcal{A}$6 in the single-agent case to

$\mathcal{A}$7

when $\mathcal{A}$8 and $\mathcal{A}$9, with an additional Byzantine penalty when $R$0 (Fan et al., 2021). In this setting, FRL is analyzed not only for privacy and sample efficiency but also for adversarial resilience.

A more recent robust formulation changes the global objective itself. In FRL with Environment Heterogeneity, the robust objective is

$R$1

where $R$2 is a covering set of transition kernels containing the observed local environments and their plausible perturbations. FedRQ implements this idea with a robust local Bellman update that adds a pessimistic next-state term weighted by $R$3, and the tabular theory proves convergence of the averaged global iterate to $R$4 with

$R$5

The same paper extends the construction to continuous state spaces through expectile regression for a degree function $R$6, then instantiates the deep variants as FedRDQN and FedRDDPG (Hwang et al., 19 Jul 2025).

4. Communication, aggregation, and systems efficiency

Communication is not a secondary implementation detail in FRL; in several papers it defines the problem. FLDDPG makes this explicit by imposing roughly matched total communication budgets across baselines: one full weight exchange costs $R$7 MB round-trip, so $R$8 weight updates imply $R$9 MB total; under the same budget, replay-sharing SEDDPG uses $\gamma$0 MB and shared-network SNDDPG uses $\gamma$1 MB. Under this communication-limited protocol, FLDDPG yields the best final reward profile and the highest simulated success rate, $\gamma$2, versus $\gamma$3 for IDDPG, $\gamma$4 for SEDDPG, and $\gamma$5 for SNDDPG (Na et al., 2022).

Communication-efficient FRL can also be formulated analytically. In the consensus-based method of (Xu et al., 2022), increasing the server synchronization period $\gamma$6 lowers the communication term in the resource cost but worsens the convergence bound, while the local consensus factor

$\gamma$7

improves with more local interaction rounds $\gamma$8, larger interaction step size $\gamma$9, and stronger graph connectivity $\{\mathcal{P}_i\}_{i=1}^n$0. The resulting picture is that FRL can trade expensive server communication for cheaper neighbor communication without abandoning global coordination.

In teleoperated-driving networks, federation is used to average action-value representations rather than trajectories, with local models weighted by the number of local learning steps. The main empirical conclusion is notable because it runs against the common assumption that deeper models are automatically preferable: federated Q-learning with a linear approximator reaches reward $\{\mathcal{P}_i\}_{i=1}^n$1, regret $\{\mathcal{P}_i\}_{i=1}^n$2, $\{\mathcal{P}_i\}_{i=1}^n$3 learnable parameters, and $\{\mathcal{P}_i\}_{i=1}^n$4 ms average time per learning step, matching or exceeding the heavier deep variants in average reward while remaining far cheaper computationally (Bragato et al., 2024).

Smart-eyewear runtime optimization adds another dimension: asynchronous federation. In that system, local DQN agents optimize runtime DNN partitioning across smart eyewear, smartphone, and cloud. Synchronous aggregation averages local weights every $\{\mathcal{P}_i\}_{i=1}^n$5 steps, while asynchronous aggregation first averages fast agents and then incrementally folds in slow agents through

$\{\mathcal{P}_i\}_{i=1}^n$6

The reported result is not a dramatic convergence-speed gain but lower training and validation variability, with a latency-violation rate around $\{\mathcal{P}_i\}_{i=1}^n$7 after $\{\mathcal{P}_i\}_{i=1}^n$8 steps, and similar behavior under moderate fractions of slow agents (Sedghani et al., 24 Aug 2025).

5. Representative application domains

The application range of FRL is broad, but the literature is dominated by systems where privacy, communication, and deployment heterogeneity are intrinsic rather than incidental.

Domain	FRL formulation	Representative finding
Swarm robotics	FLDDPG with periodic soft averaging of actor and critic weights	Success rates: $\{\mathcal{P}_i\}_{i=1}^n$9 for IDDPG, $$Q_{\mathrm{glob}}(s,a)=\sum_{i=1}^N \tau_i Q_i(s,a)$$0 for SEDDPG, $$Q_{\mathrm{glob}}(s,a)=\sum_{i=1}^N \tau_i Q_i(s,a)$$1 for SNDDPG, and $$Q_{\mathrm{glob}}(s,a)=\sum_{i=1}^N \tau_i Q_i(s,a)$$2 for FLDDPG (Na et al., 2022)
Autonomous driving	Online federated transfer RL over AirSim and JetsonTX2 RC cars	FTRL-DDPG-SIM achieves $$Q_{\mathrm{glob}}(s,a)=\sum_{i=1}^N \tau_i Q_i(s,a)$$3 increase in average obstacle distance and $$Q_{\mathrm{glob}}(s,a)=\sum_{i=1}^N \tau_i Q_i(s,a)$$4 decrease in collisions versus DDPG (Liang et al., 2019)
O-RAN slicing	Two heterogeneous xAPPs whose local Q-values are fused by a global DNN	At eMBB load $$Q_{\mathrm{glob}}(s,a)=\sum_{i=1}^N \tau_i Q_i(s,a)$$5 Mbps, FRL gives $$Q_{\mathrm{glob}}(s,a)=\sum_{i=1}^N \tau_i Q_i(s,a)$$6 higher eMBB throughput and $$Q_{\mathrm{glob}}(s,a)=\sum_{i=1}^N \tau_i Q_i(s,a)$$7 lower URLLC delay than IRL (Zhang et al., 2022)
Teleoperated driving	Federated RL for adaptive LiDAR compression in 5G TD	Federated Q-learning reaches reward $$Q_{\mathrm{glob}}(s,a)=\sum_{i=1}^N \tau_i Q_i(s,a)$$8 with $$Q_{\mathrm{glob}}(s,a)=\sum_{i=1}^N \tau_i Q_i(s,a)$$9 parameters and $Q$0 ms per learning step (Bragato et al., 2024)
Multi-platform recommendation	FedSlate fuses platform-local SlateQ outputs through a federated Q-network	For $Q$1, platform $Q$2 reaches $Q$3 versus $Q$4 for random recommendation (Deng et al., 2024)

These applications are methodologically diverse. Swarm robotics and autonomous driving use periodic model averaging to transfer control knowledge while keeping local sensory streams private (Na et al., 2022, Liang et al., 2019). O-RAN slicing uses “vertical” FRL in the paper’s own terminology: two xAPPs with different state and action spaces coordinate by sending local Q-tables to a global fusion model, which returns calibrated Q-values for final action selection (Zhang et al., 2022). Recommendation extends FRL from the conventional single-user single-platform setting to a single-user multi-platform setting, and one of its most distinctive results is that federation enables learning on a platform that lacks direct reward feedback (Deng et al., 2024).

Edge systems supply a complementary perspective. In smart eyewears, the policy does not control a robot or recommender but a runtime AI stack: the action is DNN partitioning across eyewear, smartphone, and cloud, and the reward is replaced by a cost that includes energy, 5G transmission cost, latency violations, and reconfiguration (Sedghani et al., 24 Aug 2025). This suggests that FRL is as much a systems-control paradigm as a control-theory paradigm.

6. Relation to adjacent areas and open directions

A persistent boundary issue is that not every paper combining RL and FL is FRL in the strict sense. Dynamic q-fair federated learning uses a centralized policy-gradient controller to choose a fairness parameter $Q$5 for supervised federated aggregation; the federated object is a supervised model, while RL serves as a server-side hyperparameter controller (Chen et al., 2023). FedDRL likewise applies A2C and TD3 on the server side to select trustworthy client models and aggregation weights for supervised image classification; the task being federated is not an RL policy or value function (Chen et al., 2023). These papers are relevant to RL-assisted federation, but they are not examples of federated training of RL decision-makers.

Another boundary issue is function-space consistency. FedQHD argues that FedAvg-style parameter averaging is not function-space consistent: even with identical nonlinear networks, averaging parameters need not equal averaging client Q-functions. With a shared random-feature encoder and linear readout, however,

$Q$6

is exactly the weighted average in function space. With heterogeneous encoders, the server instead averages client Q-values on an anchor-state set and each client compiles the resulting teacher through a ridge projection. The associated federation gap is decomposed into subspace misalignment, anchor-set conditioning, and regularization bias, and the well-conditioned regime is identified as $Q$7 (Hou et al., 27 May 2026). This is a direct challenge to the assumption that FRL aggregation should always occur in parameter space.

Open problems recur across the literature. The theory papers repeatedly note the need for sharper results for deep actor-critic and personalized FRL, as well as for asynchronous aggregation, client sampling, richer heterogeneity in rewards and observation spaces, and stronger privacy or security mechanisms than simple data locality (Jin et al., 2022, Zhu et al., 2024). Robust FRL-EH additionally leaves open how to choose the robustness level $Q$8 and how to relax assumptions such as common support of nonzero transitions across clients (Hwang et al., 19 Jul 2025). Systems papers raise related concerns from the opposite direction: communication cadence, stragglers, stale updates, and task transfer matter as much as asymptotic return in real deployments (Sedghani et al., 24 Aug 2025).

Taken together, these works present FRL as a distinct research area rather than as a minor variant of federated learning or reinforcement learning. Its defining question is not simply how to average decentralized learners, but how to preserve the semantics of sequential decision-making under private local interaction, heterogeneous environments, and limited communication.