Dimensionality Reduction for Robust Federated Learning: A Theoretical Analysis and Convergence Guarantee

Abstract: Federated Learning (FL) enables multiple clients to collaboratively train models without sharing raw data, but it is highly vulnerable to Byzantine attacks. Existing robust approaches can neutralize these threats but incur substantial computational overhead during high-dimensional gradient aggregation, an overhead that scales poorly with model size and increasingly dominates the training cost as modern models grow larger. To address this computational bottleneck, we propose Projected Dimensionality Reduction (PDR), a universal acceleration framework for vector-level distance-based robust aggregators, which performs robust aggregation by compressing gradients into a drastically smaller subspace via sparse random projection to efficiently compute reliability weights. This approach reduces the server computational complexity to an optimal $ \mathcal{O}(Mp) $, where $ M $ is the number of clients and $ p $ is the model dimension, matching the theoretical lower bound required merely to read the gradients. We establish convergence guarantees under standard FL assumptions in prior Byzantine-robust FL analyses. By leveraging the Subspace Embedding Theorem, we show that PDR achieves optimal convergence rates of $ \mathcal{O}(1/\sqrt{T}) $ for non-convex functions and $ \mathcal{O}(1/T) $ for strongly convex functions, where $ T $ denotes the number of iterations. Crucially, we mathematically demonstrate that this massive acceleration comes almost for free, merely inflating the inherent Byzantine error floor by a bounded, tunable factor of $ \frac{1+ε}{1-ε} $. Experimental results on benchmark datasets confirm that integrating PDR with existing aggregators yields orders of magnitude speedups in time efficiency while maintaining highly competitive convergence performance.

• The paper presents Projected Dimensionality Reduction (PDR) that compresses high-dimensional client gradients into a lower-dimensional space, drastically reducing aggregation complexity.
• It offers theoretical guarantees that preserve distance measures and ensure near-optimal convergence rates even under Byzantine adversarial conditions.
• Empirical evaluations indicate that PDR significantly accelerates federated learning performance while maintaining high classification accuracy in adversarial environments.

Projected Dimensionality Reduction for Robust Byzantine-Robust Federated Learning

Context and Motivation

Byzantine-robust federated learning (FL) aims to aggregate model updates from clients, some of whom may submit arbitrary or malicious gradients. Vector-level, distance-based aggregators (e.g., Krum, Bulyan, Geometric Median, MCA) are canonical defenses, relying on the geometry of client gradients to neutralize adversarial updates. However, growth in DNN parameter counts ($p$ from millions to billions) exposes a severe computational bottleneck: pairwise $\ell_2$ distance calculations and robust aggregation scale at least linearly in $p$ and, for most methods, quadratically in the number of clients $M$. This overhead effectively dominates server-side iteration cost, impeding the deployment of robust FL at contemporary scales.

Prior heuristics for dimensionality reduction in FL—coordinate sampling, random subspace methods—lack rigorous guarantees, are not robust to targeted attacks that concentrate adversarial noise outside the sampled subspace, and often degrade statistical efficiency or robustness. Existing uses of the Johnson–Lindenstrauss (JL) lemma in FL focus primarily on privacy, not algorithmic acceleration of robust aggregation.

Methodological Framework

The paper introduces the Projected Dimensionality Reduction (PDR) framework, a principled and general wrapper for any vector-level, distance-based robust aggregator in FL (2605.28335). PDR leverages the Subspace Embedding Theorem and sparse random projection to compress client gradients $g_m\in\mathbb{R}^p$ into lower-dimensional representations $\tilde{g}_m\in\mathbb{R}^k$, where $k\ll p$. The robust aggregator operates exclusively in this subspace, assigning reliability weights $\tilde{\alpha}_m$ to each client. The final high-dimensional aggregate is then reconstructed as the reliability-weighted sum of the original $g_m$ vectors.

Explicitly, given $M$ client gradients in $\ell_2$0, PDR performs:

1. Sparse random projection: A matrix $\ell_2$1 is applied such that $\ell_2$2 satisfies $\ell_2$3. Matrix entries may be Rademacher or Gaussian, or follow the Achlioptas construction for computational efficiency.
2. Low-dimensional robust aggregation: Compute reliability coefficients $\ell_2$4 via any robust aggregator over $\ell_2$5.
3. High-dimensional reconstruction: Aggregate as $\ell_2$6; update $\ell_2$7.

This design is mathematically supported by the guarantee that, with high probability, all pairwise $\ell_2$8 distances (and therefore geometric outlier structure) among client gradients are preserved up to a multiplicative $\ell_2$9 distortion under $p$0. Convergence analyses are conducted under standard assumptions of Lipschitz continuity, strong convexity, unbiased local gradients, and bounded gradient variance and heterogeneity.

Theoretical Contributions

PDR achieves the following formal guarantees:

• Computational complexity is reduced from $p$1 (or worse for sequential aggregators) to $p$2 for each communication round, where $p$3 does not grow with $p$4—it depends only logarithmically on the inverse failure probability and desired embedding error.
• Robustness bound: The error induced by the projection inflates the inherent aggregator's Byzantine error floor by at most a factor of $p$5, where $p$6 is a user-chosen parameter controlling the target dimension $p$7 (see Theorem 1, Lemma 1).
• Convergence rate: The federation converges at the optimal rates for distributed stochastic optimization: $p$8 for non-convex loss and $p$9 for strongly convex loss (with $M$0 the number of rounds), up to the (inflated) optimality gap set by the aggregator and the variance/heterogeneity inherent to the non-IID setting.
• Error neighborhood: The final centering neighborhood is strictly determined by the statistical variance, data heterogeneity, the aggregator's robustness constant, and the inflation factor from the subspace embedding, with no further degradation due to projection.

PDR strictly inherits the breakdown point and threat models of the base aggregator—it is not more vulnerable to sophisticated Byzantine attacks than the underlying robust mechanism.

Empirical Evaluation

Experimental results span TinyImageNet, CIFAR10, and CIFAR100 on standard DL models (MobileNetV3, VGG16, ResNet18) and Dirichlet-based non-IID data partitioning. Robustness is evaluated against Gaussian, Sign-flip, LIE, and FoE attacks at Byzantine client ratios $M$1.

Key outcomes:

• PDR-augmented robust rules achieve wall time reductions of up to two orders of magnitude compared to uncompressed robust aggregation, regardless of dataset/model scale.
• Classification accuracy under attack is nearly preserved (often within $M$2–$M$3 of the baseline; in some settings, PDR acts as a regularizer and even improves generalization).
• The projection dimension $M$4 and matrix sparsity $M$5 hyperparameters show weak sensitivity, confirming algorithmic robustness to these settings.
• The PDR framework does not degrade the breakdown point compared to the original aggregation rule: performance degrades or diverges only when the baseline aggregator fails due to overwhelming Byzantine fraction.

Limitations, Implications, and Future Directions

By construction, PDR is only suitable for vector-level distance-based aggregation rules, not coordinate-wise (e.g., median, trimmed mean) methods—random projection would corrupt the independent per-coordinate anomaly signal. The inflation of the optimality gap by the factor $M$6 is inherent and must be managed via a judicious choice of $M$7 and $M$8. PDR assumes honest client gradients are sufficiently numerous ($M$9), consistent with all provable defenses.

Practically, PDR makes large-scale, robust FL tractable by pushing robust aggregation to the theoretical lower bound of server-side cost (the cost of reading the gradients). This enables deployment in settings with massive models and/or large numbers of clients, including potential extensions to LLM-scale FL or safety-critical distributed learning. The framework suggests new possibilities for subspace-based robust estimation and compressed geometric aggregation beyond FL, potentially stimulating work in both adversarial optimization and large-scale secure aggregation.

Conclusion

Projected Dimensionality Reduction offers a universal, theoretically justified acceleration for vector-level robust aggregation in Byzantine-robust federated learning. Its tight theoretical guarantees, computational optimality, and empirical efficacy position it as a foundational method for scalable, secure collaborative learning under adversarial threat models (2605.28335).