Federated Reinforcement Learning

• Federated Reinforcement Learning is a distributed paradigm where local RL agents collaborate by sharing model parameters instead of raw trajectories.
• It employs diverse methods such as FedAvg, reward sharing, and asynchronous updates to address heterogeneity and ensure robust performance.
• Applications in wireless communications, edge computing, and autonomous control demonstrate FRL’s ability to improve stability, fairness, and privacy protection.

Federated Reinforcement Learning (FRL) is a distributed learning paradigm that combines reinforcement learning with federated learning: multiple agents interact with their own local environments, keep trajectories and other raw experience data local, and cooperate by exchanging model-related information to learn a shared decision policy or value representation. In the most common server-based form, clients periodically upload parameters, gradients, or parameter differences to a central coordinator, which aggregates them into a global model and broadcasts the result back; other instantiations replace parameter aggregation with collaborative reward sharing or federated representation learning (Li et al., 2022, Metelo et al., 3 Jun 2025, Ali et al., 2023). FRL differs from centralized RL because state–action–reward trajectories are not centralized, and it differs from standard federated learning because the data are sequential, policy-dependent, and generated under Markovian interaction, often with partial observability and environment heterogeneity (Zhang et al., 2024, Ganesh et al., 2024).

1. Formal setting and problem formulations

A standard FRL formulation associates each client $i$ with its own MDP $\mathcal{M}_i$, possibly sharing state and action spaces while differing in transition kernels, reward functions, or initial-state distributions. A common global objective is to optimize a single policy parameter vector $\theta$ for average performance across clients,

$J(\theta) = \frac{1}{N}\sum_{i=1}^N J_i(\theta),$

where $J_i(\theta)$ is the expected discounted return of $\pi_\theta$ in environment $i$ (Wang et al., 2024, Zhu et al., 2024). This average-objective view underlies several policy-gradient and actor–critic formulations, including settings with arbitrarily heterogeneous environments and settings where each agent interacts with an independent but identically distributed copy of a common MDP (Wang et al., 2024, Ganesh et al., 2024).

Environment heterogeneity is a central complication. In on-policy heterogeneous FRL, each client may face different transition kernels $P^{(i)}$ and reward functions $r^{(i)}$, so a universal policy cannot in general be simultaneously optimal for all agents. The finite-time analysis of FedSARSA shows that collaboration can still yield a policy that is near-optimal for all agents, with the extent of near-optimality proportional to the level of heterogeneity, while preserving linear speedups as the number of agents increases (Zhang et al., 2024). The single-loop federated actor–critic setting reaches a similar conclusion: convergence is to a near-stationary point, and the asymptotic error scales with environment heterogeneity (Zhu et al., 2024).

A more robust formulation replaces the average objective by a worst-case one. In the FRL-EH framework, the global objective is

$$J^{\rm FRL}(\pi) := \inf_{P \in \mathcal{P}} J(\pi \mid P),$$

where $\mathcal{M}_i$0 is a covering set of plausible transition kernels containing all local environments and perturbations around them (Hwang et al., 19 Jul 2025). This formulation explicitly targets robust performance across heterogeneous environments rather than performance on a virtual averaged environment.

Offline FRL changes the data model again. Instead of interacting with environments during training, each client holds a static dataset $\mathcal{M}_i$1 collected by a local behavior policy. In that case, the global objective is still an average over clients, but optimization must account for coverage limitations and mixed-quality logged data. FOVA addresses precisely this setting by introducing a vote mechanism and advantage-weighted regression to mitigate the effect of low-quality local behavior policies (Qiao et al., 2 Dec 2025).

2. Architectural patterns and communication schemes

The canonical FRL architecture is server-mediated. In indoor small-cell power control, each room acts as a client with its own DQN agent, the central server aggregates parameter vectors with FedAvg,

$\mathcal{M}_i$2

and the resulting global model is rebroadcast for further local training or adaptation in new rooms (Li et al., 2022). The same basic pattern appears in V2X resource allocation, where V2V agents run local policy-gradient updates and periodically synchronize via a central server, and in smart-eyewear runtime optimization, where DQN parameters are aggregated either synchronously or asynchronously (Xu et al., 2024, Sedghani et al., 24 Aug 2025).

FRL is not restricted to full-policy averaging. FAuNO uses a federated actor–critic structure in which local PPO actors remain personalized while only the critic is federated; the global manager performs a buffered asynchronous aggregation of critic updates inspired by FedBuff, so faster agents do not wait for stragglers (Metelo et al., 3 Jun 2025). FedRAG pushes this specialization further by federating only a behavioral metric-based state projection function $\mathcal{M}_i$3, leaving policy and value optimization local while sharing a representation that improves cross-client generalization and provides a degree of privacy protection (2505.09959).

Peer-to-peer FRL variants also exist. In multi-link Wi-Fi networks, the “global” component is implemented not by a parameter server but by collaborative reward sharing. Each access point runs a local multi-armed bandit, broadcasts locally computed rewards to neighbors, and derives a global reward from the minimum achieved reward among nearby BSSs to enforce a max–min fairness criterion (Ali et al., 2023). This design shows that FRL can be realized through shared learning signals rather than explicit model averaging.

Communication schedules matter because they determine both statistical coupling and systems cost. FAuNO uses semi-asynchronous buffered aggregation to avoid straggler blocking in heterogeneous edge systems (Metelo et al., 3 Jun 2025). Smart-eyewear FRL implements both synchronous and asynchronous federation strategies and reports that federated agents exhibit significantly lower performance variability in both modes (Sedghani et al., 24 Aug 2025). AV platooning varies update delay and cutoff ratio, showing that early global sharing followed by later local specialization can be beneficial (Boin et al., 2022).

3. Algorithmic families

Value-based FRL remains a foundational strand. DQN-based FRL appears in indoor small-cell transmit-power control, where a global model aggregated from multiple rooms attains almost identical performance to specialized per-room RL agents and improves convergence speed when used as a base model in an unseen room (Li et al., 2022). In heterogeneous on-policy settings, FedSARSA combines linear function approximation, local SARSA updates, and periodic averaging to obtain finite-time guarantees under Markovian sampling and multiple local steps (Zhang et al., 2024). FedRQ extends tabular federated Q-learning to robust heterogeneous environments by adding a pessimistic transition term and proves asymptotic convergence to the optimal policy for the robust global objective (Hwang et al., 19 Jul 2025).

Policy-gradient FRL emphasizes direct optimization of $\mathcal{M}_i$4. Res-NHARPG treats each honest client as computing recursive Hessian-aided policy-gradient estimators and the server as applying an $\mathcal{M}_i$5-resilient aggregation rule, which yields the first global convergence guarantees with general parametrization in the presence of Byzantine adversaries (Ganesh et al., 2024). FedSVRPG-M and FedHAPG-M incorporate momentum together with variance reduction or Hessian approximation, and exactly converge to a stationary point of the average performance function regardless of the magnitude of environment heterogeneity, with per-agent sample complexity $\mathcal{M}_i$6 (Wang et al., 2024).

Actor–critic FRL covers both practical and theoretical variants. FAuNO uses local PPO actors with a federated critic for decentralized task offloading in edge systems, and empirically matches or exceeds heuristic and federated multi-agent RL baselines in reducing task loss and latency (Metelo et al., 3 Jun 2025). SFAC develops a single-loop federated actor–critic with two-level federation of actor and critic, and proves convergence to a near-stationary point with linear speed-up under heterogeneous environments and Markovian sampling (Zhu et al., 2024). In continuous-control platooning, AVDDPG federates actor and critic parameters or gradients across vehicles, finding that weight aggregation outperforms gradient aggregation, especially in intra-platoon sharing (Boin et al., 2022).

Model-based FRL addresses sample efficiency more directly. FEMRL trains local dynamics models on each client, constructs an ensemble of these models at the server, distills the ensemble into a global student model, and then trains the policy solely on the learned dynamics without further interaction with the real environment. It provides a monotonic improvement guarantee and obtains much higher sample efficiency than classic model-free FRL algorithms in continuous-control benchmarks (Wang et al., 2021).

Offline FRL introduces additional algorithmic structure because no new trajectories can be collected. FOVA addresses mixed-quality logged data with a vote mechanism that identifies high-return actions during local policy evaluation and with consistent local and global objectives built on advantage-weighted regression. Its theoretical analysis shows strict policy improvement over the behavioral policy (Qiao et al., 2 Dec 2025).

Representation-centric FRL forms another category. FedRAG learns state projection functions that approximate a behavioral metric and federates only the projection parameters. This avoids sharing task-specific trajectories or policies while improving transfer across heterogeneous environments and distracting visual conditions (2505.09959). RIFRL, by contrast, exploits the node-wise invariance of ReLU networks and applies a backward rescale-invariant operation before and after aggregation to reduce parameter-scale discrepancy across V2X agents (Xu et al., 2024).

4. Theoretical guarantees, heterogeneity, and robustness

A major development in FRL has been the shift from heuristic federation to finite-time and non-asymptotic theory. FedSARSA establishes the first comprehensive finite-time analysis for an on-policy, heterogeneous, value-based FRL scheme with continuous state–action spaces and linear function approximation. Its bounds separate variance terms, which scale as $\mathcal{M}_i$7, from heterogeneity-induced bias, which scales with explicit transition and reward discrepancy measures $\mathcal{M}_i$8 and $\mathcal{M}_i$9 (Zhang et al., 2024). SFAC extends this line to actor–critic, showing that the federation of agents yields linear sample-complexity speed-up even with Markovian sampling and multiple local updates (Zhu et al., 2024).

Robustness to adversaries adds another layer. Res-NHARPG proves global convergence for federated policy gradient with Byzantine agents under general parametrization, and achieves optimal sample complexity of order

$\theta$0

where $\theta$1 is the total number of agents and $\theta$2 is the number of adversarial agents (Ganesh et al., 2024). A separate robustness line shows that existing Byzantine-robust FRL aggregators are vulnerable to the Normalized attack, which maximizes angular deviation rather than raw distance between benign and poisoned global updates. To counter this, an ensemble FRL method trains multiple global policies and chooses actions by majority vote in discrete action spaces or geometric median in continuous ones, yielding substantial resistance against poisoning attacks (Fang et al., 12 Feb 2025).

Robustness can also target environment uncertainty rather than malicious agents. FedRQ formalizes this through a covering set $\theta$3 of transition kernels and proves that its tabular Q-function converges to the unique fixed point of a robust Bellman operator, i.e., the optimal policy for the worst-case objective over $\theta$4 (Hwang et al., 19 Jul 2025). In deep settings, the same paper uses expectile loss to approximate the “min over neighboring states” term required by the robust objective, enabling robust extensions of DQN and DDPG (Hwang et al., 19 Jul 2025).

Hardware reliability is an additional theoretical and systems concern. FRL-FI studies transient faults in FRL-based navigation systems and finds that multi-agent FRL is more resilient than single-agent RL because corrupted updates from one agent can be diluted by the others. It also proposes server checkpointing and range-based anomaly detection, achieving up to $\theta$5 improvement in resilience with $\theta$6 overhead (Wan et al., 2022).

5. Application domains and empirical behavior

Wireless communications has been a particularly active application domain. In indoor 5G small-cell power control, FRL addresses the observation that an RL agent trained in one room is room-dependent and does not transfer well to new heterogeneous layouts; the federated model matches specialized single-room RL performance, outperforms random power allocation and exhaustive search, and serves as a better initialization for adaptation in new rooms (Li et al., 2022). In multi-link Wi-Fi, FRL-based link allocation improves throughput fairness and higher reliability by maximizing the minimum achieved data rate relative to fixed, random, and non-federated RL schemes (Ali et al., 2023). In V2X resource allocation, the rescale-invariant FRL algorithm improves both convergence speed and convergent performance by reducing data discrepancy among agents (Xu et al., 2024).

Edge and distributed systems provide a second major cluster of applications. FAuNO targets decentralized task offloading in edge systems and uses a federated critic to coordinate agents under partial observability, buffered asynchrony, and variable connectivity (Metelo et al., 3 Jun 2025). FEMRL addresses edge-computing continuous control with model-based RL and ensemble distillation, obtaining much higher sample efficiency than model-free FRL baselines (Wang et al., 2021). Smart-eyewear runtime optimization uses FRL to choose DNN partitioning and offloading decisions across SEW, phone, and cloud; synchronous and asynchronous federation both reduce performance variability, which is critical for real-time object detection workloads (Sedghani et al., 24 Aug 2025).

Autonomous control applications show that FRL can also exploit topology. AVDDPG studies inter-platoon and intra-platoon federation for continuous platoon control and concludes that intra-platoon weight aggregation provides the best performance, with weight aggregation outperforming gradient aggregation and largely outperforming non-federated training for platoons of length 3, 4, and 5 vehicles (Boin et al., 2022). FRL-FI considers swarm-style navigation and demonstrates that FRL-based navigation systems can maintain better task performance under transient faults than comparable single-agent systems (Wan et al., 2022).

Across these application classes, a recurring empirical pattern is that FRL often improves stability, transfer, or generalization more reliably than it improves asymptotic peak reward. This suggests that the practical value of federation is frequently in reducing variance, accelerating warm-start adaptation, or maintaining fairness and reliability across environments, rather than in strictly dominating all standalone RL baselines on every local task.

6. Privacy, limitations, and research directions

A central motivation for FRL is privacy: clients generally exchange parameters, gradients, or compact statistics rather than raw trajectories, UE measurements, video streams, or task logs (Li et al., 2022, Sedghani et al., 24 Aug 2025). Yet privacy is not automatic. FedRAG explicitly argues that sharing an approximated behavior metric-based state projection function provides information gain while sharing no sensitive task-specific information (2505.09959). Offline FRL makes the issue sharper because logged trajectories may be heterogeneous and sensitive; FOVA therefore emphasizes consistency between local and global objectives while avoiding raw-data sharing (Qiao et al., 2 Dec 2025).

Several limitations recur across the literature. First, non-IID client data and heterogeneous environments remain the dominant statistical difficulty. Simple FedAvg can be suboptimal under strong heterogeneity, motivating personalized FRL, client weighting, clustered aggregation, robust objectives, and local specialization (Li et al., 2022, Zhang et al., 2024). Second, communication and synchronization costs remain significant: aggregation period $\theta$7, buffer size, partial participation, and staleness all affect convergence and systems efficiency (Metelo et al., 3 Jun 2025, Sedghani et al., 24 Aug 2025). Third, action-space growth can become prohibitive in wireless control and multi-agent resource allocation, where the joint action space scales combinatorially (Li et al., 2022).

A common misconception is that FRL is merely “FedAvg applied to RL.” The surveyed systems contradict this. Critics alone can be federated while actors remain local; only state encoders can be shared; reward-sharing can induce a federated signal without any model averaging; and robust ensemble methods can replace a single global policy altogether (Metelo et al., 3 Jun 2025, 2505.09959, Ali et al., 2023, Fang et al., 12 Feb 2025). Another misconception is that a single averaged policy is always the right target. Robust FRL-EH instead optimizes worst-case return over a covering set of perturbed environments, and offline FRL may explicitly privilege high-return actions through vote mechanisms when client data quality is mixed (Hwang et al., 19 Jul 2025, Qiao et al., 2 Dec 2025).

Current research directions therefore cluster around five themes: stronger robustness to Byzantine agents and transient faults, principled handling of heterogeneity, more communication-efficient and asynchronous federation, richer model-based and representation-based sharing schemes, and extensions from tabular or linear theory to deep actor–critic and offline settings with rigorous guarantees. Taken together, these developments position FRL less as a single algorithm than as a family of distributed RL designs for learning under privacy, decentralization, and environment diversity.