---
title: Federated Q-Learning
url: https://www.emergentmind.com/topics/federated-q-learning
type: topic
---

# Federated Q-Learning

Federated Q-Learning (FedRL) is a paradigm within Federated Reinforcement Learning in which multiple agents, distributed across computational nodes or physical locations, collaboratively learn or refine action-value functions (Q-functions). These agents do so without sharing raw experience data, but instead coordinate via aggregation and synchronization of Q-function parameters or value estimates under the orchestration of a central server. FedRL generalizes classic Q-learning to decentralized settings where privacy, communication constraints, or structural heterogeneity preclude centralized data collection.

## 1. Formulation and Algorithmic Foundations

FedRL assumes a set of $N$ (or $K$) agents, each interacting with its own instance of an environment formulated as a Markov Decision Process (MDP) or with a local sampler. The canonical objective is estimation of the optimal Q-function $Q^*$ that satisfies the Bellman optimality condition
$$
Q^*(s,a) = r(s,a) + \gamma \mathbb{E}_{s' \sim P(\cdot|s,a)} [\max_{a'} Q^*(s',a')],
$$
where $r(s,a)$ is the reward, $P$ is the transition kernel, and $\gamma < 1$ is the discount factor.

A typical synchronous protocol proceeds as follows:

1. **Local Q-update:** Each agent $k$ executes local Q-learning, using its data or samples, performing
   $$
   Q^{k}_{t+\frac{1}{2}}(s,a) = (1-\eta) Q^k_t(s,a) + \eta \left[ r(s,a) + \gamma \max_{a'} Q^k_t(s',a') \right],
   $$
   where $s' \sim P^k(\cdot|s,a)$ is agent-specific.

2. **Periodic aggregation:** Every $E$ local steps, the server aggregates local Q-estimates, commonly via uniform averaging:
   $$
   Q^{k}_{t+1}(s,a) = \frac{1}{N} \sum_{j=1}^N Q^j_{t+\frac{1}{2}}(s,a).
   $$
   Agents then replace or combine their local Q with the aggregated model.

3. **Termination:** The process continues until desired accuracy is achieved or resource budgets are exhausted.

Variants include asynchronous protocols, intermittent communication schemes, and importance-weighted aggregation [2305.10697][2408.16981].

## 2. Theoretical Guarantees and Communication-Sample Complexity

Early analysis focused on the trade-off between communication complexity (frequency and cost of aggregation) and per-agent sample complexity. For homogeneous agents (identical dynamics and reward), linear speedup ($1/N$ scaling) in sample complexity is achievable by distributing the global sample budget across agents [2305.10697]. The following sample complexity bounds are representative:
$$
T = \widetilde{O}\left( \frac{|S||A|}{N (1-\gamma)^5 \varepsilon^2} \right)
$$
for achieving $\| Q - Q^* \|_\infty < \varepsilon$ [2305.10697][2408.16981].

Critical lower bounds establish that any protocol obtaining this speedup must incur at least $\Omega(1/(1-\gamma))$ communication rounds (up to logarithmic factors), and a total communication volume proportional to the state-action space [2408.16981]. Algorithms such as Fed-DVR-Q are order-optimal in both sample and communication complexity:
- Sample: $O\left( \frac{|S||A|}{N (1-\gamma)^3 \varepsilon^2} \right)$,
- Rounds: $O\left( \frac{1}{1-\gamma} \log \frac{1}{(1-\gamma)\varepsilon} \right)$,
- Bits: $O\left(\frac{|S||A|}{1-\gamma} \log \frac{1}{(1-\gamma)\varepsilon}\right)$ [2408.16981].

Achieving optimality requires careful design (variance reduction, large batch local updates, quantized aggregation) and is fundamentally limited by the contraction properties of the Bellman operator and the non-i.i.d. nature of reinforcement learning data.

## 3. Heterogeneous Agents and BLack-Box Architectures

In practical scenarios, participating agents are often heterogeneous with respect to their MDPs, policy parameterizations, data distributions, or computational budgets. The "FedRL-HALE" setting (Federated RL with Heterogeneous And bLack-box agEnts) captures cases where each agent $n$ maintains a private Q-network $Q(\cdot, \cdot; \theta_n)$ (possibly distinct architecture, optimizer, data buffer), and only exposes value estimates for prescribed queries [2301.11135].

The Federated Heterogeneous Q-Learning (FedHQL) framework decouples agent architectures and permits "black-box" participation. The protocol features:
- **Federated UCB exploration:** The server computes an action $a_t$ via UCB over the agents' $Q_n(s_t, a)$ vectors.
- **Federated temporal-difference (TD) update:** Aggregated $Q$-values are updated via server-side Bellman backup, and broadcast to agents.
- **Local knowledge distillation:** Agents locally align their $Q_n$ to the federated $Q$ via loss minimization.

This architecture:
- Enables collaboration among disparate agents;
- Preserves data/model privacy (no gradients or raw samples shared);
- Empirically yields $2$–$3\times$ sample-efficiency gains, robust consensus learning, and improved performance for weak agents [2301.11135].

FedHQL and analogous approaches handle both parametric heterogeneity and dynamics heterogeneity but pose open challenges for global convergence analysis and theoretical optimality beyond value-bounded stability.

## 4. Impact of Heterogeneity and Adaptive Aggregation

In the presence of heterogeneous dynamics (distinct $P^k$ across agents), convergence behavior can be fundamentally altered. In synchronous averaging schemes with interval $E>1$, heterogeneity induces a bias that does not vanish even with infinite samples:
$$
\| Q_T - Q^* \|_\infty \geq \Omega\left(\frac{E}{T}\right),
$$
where $E$ is the local update interval and $T$ the total updates. The bias scales with the degree of transition kernel mismatch $\kappa = \sup_{k,s,a} \| \bar P(\cdot|s,a)-P^k(\cdot|s,a) \|_1$ [2409.03897]. Hence,
- Frequent aggregation ($E=1$) is necessary in highly heterogeneous settings;
- Multi-phase, adaptive step-size schedules can mitigate bias: aggressive exploration early, reduced learning rate post phase-transition;
- Estimating $\kappa$ in practice guides the choice of $E$ and step-size [2409.03897].

An important positive result is the "blessing of heterogeneity": even when individual agents’ trajectories do not cover the entire state-action space, learning is possible as long as the **union** of their behaviors ensures collective coverage. Robust linear speedup is possible with correct aggregation weights (visit-count-based importance averaging) over randomly and heterogeneously sampled data [2305.10697].

## 5. Advances in Communication-Efficient and Gap-Dependent Protocols

Modern FedRL research aims at protocols that simultaneously minimize communication cost and achieve optimal regret/sample rates:
- **Event-triggered communication:** Synchronization and aggregation are invoked only upon certain data-driven events (e.g., visit-count thresholds), leading to $O(\log T)$ communication rounds while maintaining minimax regret rates for episodic MDPs [2312.15023][2405.18795].
- **Gap-dependent analysis:** In favorable MDPs where suboptimal actions have positive gaps $\Delta_{\min}>0$, FedRL can achieve $\log T$-type regret instead of $\sqrt{T}$, and further reduce the communication cost, removing dependence on agent/state/action number inside logarithms [2502.02859].
- **Variance reduction and reference-advantage decomposition:** Using fixed reference values and advantage estimates for Q-updates reduces variance and lowers the leading constant in regret and communication bounds [2405.18795].
- **Compression:** Incorporating unbiased and error-feedback-based compression operators (e.g., Top-$K$, Sparsified-$K$) at the per-round aggregation step can reduce bandwidth up to $90\%$ over standard uncompressed aggregation, without statistically significant loss in solution quality [2404.10635].

These techniques allow scaling FedRL to high-dimensional settings and stringent communication constraints.

## 6. Practical Applications and Empirical Evaluations

FedRL has been deployed for real-world networked and multi-agent control systems:
- **5G and Open-RAN load balancing:** Devices locally train DQNs for handover decisions and aggregate models at the near-RT controller via FedAvg. This achieves $12\%$ higher throughput and $30\%$ lower load variance than MAX-SINR baselines [2403.08813].
- **IoT and edge networks:** Federated DDQN and DQN schemes for offloading and energy-delay minimization achieve up to $3\times$ faster convergence compared to non-federated baselines, with observed acceleration attributable to knowledge transfer via model aggregation [2104.11320].
- **Teleoperated and vehicular networks:** Tabular and neural federated Q-learning outperform both centralized and stateless RL for predictive QoS, balancing end-to-end delay and compression fidelity in strict real-time environments [2410.02312].
- **Edge cloud task allocation/routing:** Federated Deep Q-Learning enables decentralized task routing with reduced fronthaul delay and improved global reward [2308.07258].

Empirically, federated protocols enable privacy-preserving, distributed RL with lowered computation and communication burdens on central controllers and enhanced robustness to agent failures or stragglers.

## 7. Limitations, Open Problems, and Future Directions

Despite the significant advances, FedRL remains subject to several open challenges:
- Theoretical characterization of convergence and optimality for deep, heterogeneous, and non-tabular Q-networks remains incomplete [2301.11135].
- Handling severe dynamics heterogeneity, non-stationary environments, and partial coverage without loss of statistical or communication efficiency is an area of active research [2409.03897].
- Practical security and privacy constraints (e.g., differential privacy, secure aggregation) for Q-models and gradients are not fully addressed in most protocols [2403.08813].
- Extending federated methodologies to actor-critic, policy-gradient, or value-prediction settings, and developing communication-adaptive and asynchrony-robust designs, represent current frontiers [2301.11135][2405.04026].

*In summary, federated Q-learning provides rigorous, scalable mechanisms for distributed RL under privacy, heterogeneity, and communication constraints. A growing body of theory quantifies the conditions under which linear speedup, minimal communication, and robustness to agent and data non-uniformity can be achieved. Ongoing research seeks to generalize these guarantees to broader classes of RL algorithms and to more severe heterogeneity and limited communication scenarios.*

Source: https://www.emergentmind.com/topics/federated-q-learning