FedRQ: Robust Federated Learning Approaches

• FedRQ is a federated learning paradigm combining robust reinforcement learning and residual-quantized recommendation to tackle heterogeneity, noise, and communication challenges.
• In reinforcement learning, it employs a min-max robust Q-update with expectile regression to ensure policy performance under worst-case environmental perturbations.
• In recommendation, it leverages feature-indexed residual quantization to reduce communication costs and enhance noise resilience in privacy-preserving settings.

FedRQ refers to multiple, independently developed federated learning algorithms that incorporate the "RQ" abbreviation, including: (1) a robust federated reinforcement learning approach for heterogeneous environments (Hwang et al., 19 Jul 2025); and (2) a feature-indexed federated recommendation algorithm based on residual-quantized codebooks (Han et al., 26 Jan 2026). Both approaches aim to improve either the robustness of collaborative learning or the efficiency and noise resilience of model communication in privacy-preserving settings.

1. FedRQ in Federated Reinforcement Learning with Environmental Heterogeneity

FedRQ, as introduced in "Federated Reinforcement Learning in Heterogeneous Environments" (Hwang et al., 19 Jul 2025), is designed for distributed reinforcement learning where each agent operates in a Markov Decision Process (MDP) with distinct transition dynamics, i.e., $P_k(s'|s,a)$. The formalism addresses statistical heterogeneity by optimizing a robust global policy using a min-max (worst-case) criterion over a set of plausible transition models,

$$J^{\rm FRL}(\pi) = \inf_{P \in \mathcal{P}_\omega} J(\pi \mid P),$$

where $\mathcal{P}_\omega$ defines convex combinations of empirical kernels and neighborhood-supported perturbations. Unlike standard federated RL, which simply averages local returns, FedRQ explicitly accounts for environment uncertainty and ensures policy robustness under plausible deviations from observed dynamics.

The algorithm alternates rounds of local, robust Q-function updates and federated aggregation:

• Each agent $k$ updates its Q-table $Q^k_t$ via a robust Bellman-like step, mixing average and worst-case next-state evaluations.
• Every $E$ steps, all Q-tables are averaged to obtain $\bar{Q}_{t+1}$, which is then broadcast for synchronization. Policy improvement is performed as $\pi(s) = \arg\max_a \bar{Q}(s,a)$.

For continuous space, expectile regression is adopted to estimate lower quantiles, i.e., the worst-case value, using a dual-network architecture (Q-network and expectile network), enabling the procedure to scale to deep RL settings.

Convergence of FedRQ is established under mild conditions, providing explicit error bounds on $\bar{Q}_t$ with respect to the robust fixed point, decreasing as $O(1/t)$.

2. FedRQ in Feature-Indexed Federated Recommendation

Separately, "Feature-Indexed Federated Recommendation with Residual-Quantized Codebooks" (Han et al., 26 Jan 2026) presents RQFedRec, referred to as FedRQ, which is a communication-efficient federated recommendation algorithm. In contrast to classical ID-indexed federated approaches (which transmit all item embeddings corresponding to items observed by each client), FedRQ adopts a novel feature-indexed paradigm grounded in residual quantization (RQ).

Core mechanisms include:

• Each item $$J^{\rm FRL}(\pi) = \inf_{P \in \mathcal{P}_\omega} J(\pi \mid P),$$0 is assigned an $$J^{\rm FRL}(\pi) = \inf_{P \in \mathcal{P}_\omega} J(\pi \mid P),$$1-tuple of discrete code IDs $$J^{\rm FRL}(\pi) = \inf_{P \in \mathcal{P}_\omega} J(\pi \mid P),$$2, using RQ-Kmeans over item embeddings.
• Clients store $$J^{\rm FRL}(\pi) = \inf_{P \in \mathcal{P}_\omega} J(\pi \mid P),$$3 codebooks of size $$J^{\rm FRL}(\pi) = \inf_{P \in \mathcal{P}_\omega} J(\pi \mid P),$$4 and reconstruct item embeddings as $$J^{\rm FRL}(\pi) = \inf_{P \in \mathcal{P}_\omega} J(\pi \mid P),$$5.
• Instead of transmitting per-item embeddings, clients transmit the small set of codebook embeddings, resulting in per-round communication cost $$J^{\rm FRL}(\pi) = \inf_{P \in \mathcal{P}_\omega} J(\pi \mid P),$$6, which is controllable and independent of the number of interacted items.
• Dual-channel aggregation maintains both semantic (LLM-derived) and collaborative (user-data-learned) codebooks, combined via a curriculum-driven interpolation factor $$J^{\rm FRL}(\pi) = \inf_{P \in \mathcal{P}_\omega} J(\pi \mid P),$$7 scheduled over the training period.

This design ensures cross-item generalization (since embeddings are shared among items with the same codes) and increases robustness to noise, as demonstrated theoretically and empirically in the paper.

3. Algorithmic Frameworks and Mathematical Formalization

FedRQ in Reinforcement Learning

Formalization involves a robust objective:

$$J^{\rm FRL}(\pi) = \inf_{P \in \mathcal{P}_\omega} J(\pi \mid P),$$8

with local robust Bellman updates:

$$J^{\rm FRL}(\pi) = \inf_{P \in \mathcal{P}_\omega} J(\pi \mid P),$$9

For continuous state-action, the Q- and expectile-networks are trained with target losses incorporating both average and expectile terms.

FedRQ in Recommendation

Residual quantization assigns codes recursively: $\mathcal{P}_\omega$0 with optimization objective: $\mathcal{P}_\omega$1 The reconstructed embedding for user $\mathcal{P}_\omega$2, item $\mathcal{P}_\omega$3 is

$\mathcal{P}_\omega$4

Global aggregation is averaging of codebooks weighted by client data volume.

4. Empirical Results and Comparative Analysis

FedRQ in Reinforcement Learning was evaluated on both discrete (CartPole, MountainCar, Acrobot, LunarLander) and continuous (Ant, Hopper, Walker2d, HalfCheetah) control benchmarks with $\mathcal{P}_\omega$5 agents, each subjected to up to $\mathcal{P}_\omega$6 perturbations in environment parameters. The following findings were reported:

• FedRQ variants surpassed DQNAvg and DDPGAvg baselines in both average and worst-case reward.
• Under test-time perturbations, FedRQ demonstrated significantly improved robustness, with performance degrading gracefully relative to non-robust federated averaging approaches.

FedRQ in Recommendation was assessed on five public datasets:

• Communication cost was reduced to 30–60% of FedMF in sparser settings.
• On MovieLens-100k, Recall@10 improved from 0.1749 (FedCIA) to 0.1914, NDCG@10 from 0.6228 to 0.6695.
• Robustness to local differential privacy perturbations was documented, with NDCG@10 remaining above 0.61 even under substantial Laplace noise addition.
• Ablation confirmed that both dual-channel codebooks and curriculum schedule are necessary for optimal accuracy.

Comparison Table:

Application Domain	FedRQ Innovation	Main Empirical Benefit
Reinforcement learning (Hwang et al., 19 Jul 2025)	Robust min-max federated Q-update, expectile extension	Improved worst-case RL performance, graceful degradation to distribution shift
Recommendation (Han et al., 26 Jan 2026)	Feature-indexed codebooks via RQ, dual-channel curriculum	Reduced per-round comm., state-of-the-art accuracy, strong noise robustness

5. Theoretical Insights and Limitations

The robust RL FedRQ method provides a convergence rate of $\mathcal{P}_\omega$7 for the global robust Q-estimate, conditional on consistent neighborhood support among agents. Selection of the robustness hyperparameter $\mathcal{P}_\omega$8 affects the tradeoff between optimism and pessimism, with oversetting leading to overly conservative solutions.

In recommendation, the aggregation theorem ensures that feature-indexed codebooks lead to equal or lower expected noise error relative to ID-indexed updates, as more user data are pooled per code embedding. Communication complexity is fixed, independent of local data sparsity.

Limitations include:

• For RL FedRQ, coverage set design and selection of $\mathcal{P}_\omega$9 are dataset and heterogeneity dependent.
• The feature-indexed recommendation approach relies on the structure of the item embedding space and may underperform in cases with minimal embedding redundancy.
• Both methods assume honest majority in communication aggregation; neither is tailored for strong adversarial environments or malicious server behavior.

6. Connections and Distinctions Among FedRQ Variants

Although both algorithms are labeled "FedRQ," they belong to distinctly different domains and share no methodological overlap apart from the communication- and robustness-oriented philosophy of federated learning.

• In RL, "RQ" reflects robust Q-learning and expectile quantile estimation for worst-case value propagation (Hwang et al., 19 Jul 2025).
• In recommendation, "RQ" refers to Residual Quantization for compression and feature-indexed model sharing (Han et al., 26 Jan 2026).

The overlap in acronym is coincidental; each system is referenced uniquely in its domain and neither paper cites or builds on the other.

7. Practical Implications and Prospects

Both FedRQ variants address core operational obstacles in federated learning: environmental heterogeneity and communication cost. The RL variant is particularly suited for distributed robotics and safety-critical applications demanding worst-case policy guarantees; the recommendation system variant is appropriate for large-scale, privacy-preserving item recommendation with reduced bandwidth and improved noise handling. A plausible implication is that the "RQ" paradigm—whether robustification or quantization—provides a fertile axis for future federated algorithm research, particularly for environments or loss models with adverse statistical properties or demanding stringent privacy and scalability controls.

No evidence in the literature suggests a unified "FedRQ" framework that merges the RL and recommendation-feature-indexed lines; the notion persists as a naming collision in distinct research streams (Hwang et al., 19 Jul 2025, Han et al., 26 Jan 2026).