Sparse Self-Federated Learning

Updated 10 June 2026

Sparse self-federated learning is a method that integrates selective parameter updates with decentralized training to reduce computation and bandwidth while personalizing models for non-IID data.
Dynamic approaches like FedSpa and FedDST use adaptive prune-and-regrow cycles to maintain high sparsity levels, achieving up to 10× communication savings without compromising convergence.
Decentralized and resource-aware variants tailor sparse updates to client-specific data and hardware constraints, enabling efficient deployments in IoT and edge computing environments.

Sparse self-federated learning encompasses a family of techniques that integrate sparsity—meaning selective parameter utilization or update—with federated and decentralized training paradigms. By ensuring that each client communicates and trains with only a small, potentially personalized subset of the global model parameters, sparse self-federated learning achieves substantial reductions in computation, bandwidth, and storage load, while also offering mechanisms for personalization under heterogeneous and non-IID data regimes. Approaches range from static, global masked subnetworks derived from saliency, to fully personalized dynamic sparse masks with adaptive regrowth, to decentralized protocols for edge computing and green AI in IoT-scale deployments.

1. Problem Formulation and Motivations

Standard federated learning (FL) seeks to learn a model $w \in \mathbb{R}^d$ by minimizing an average objective over $K$ clients,

$\min_{w}~\frac{1}{K}\sum_{k=1}^K F_k(w)$

where $F_k$ reflects the local empirical risk on the private dataset of client $k$ . Two orthogonal bottlenecks motivate sparse self-federated approaches:

Resource constraints: Edge and IoT clients often lack the computation and communication capacity for dense, large neural networks (Domini et al., 10 Jul 2025).
Statistical and system heterogeneity: Clients have non-IID data, different capacities, and quality of data, calling for personalized, possibly heterogeneous models as opposed to a single global model (Huang et al., 2022, Guastella et al., 7 Apr 2025, Domini et al., 10 Jul 2025).

Sparse self-federated learning frameworks address these issues by associating each client with a personalized binary mask $m_k \in \{0,1\}^d$ , resulting in effective models $w_k = m_k \odot w$ , where only selected parameters are active. Communication is restricted to the support of $m_k$ , and computation is similarly reduced (Huang et al., 2022, Dai et al., 2022). Decentralized and self-organizing protocols can further eliminate single points of failure and adapt federation structure dynamically (Dai et al., 2022, Domini et al., 10 Jul 2025).

2. Algorithmic Foundations and Variants

2.1 Static and Saliency-Derived Masks

Several methods use a global, static sparse mask, generated by aggregating parameter saliency (e.g., SNIP criterion) across clients' local data at initialization (Ohib et al., 2023, Ohib et al., 2024). Clients only train and communicate the parameters in this mask:

SalientGrads/SSFL: Each client computes a local saliency score $s_k$ for each parameter pre-training; the scores are aggregated to form a global mask $m_g$ (Ohib et al., 2023, Ohib et al., 2024). Training proceeds on the subnetwork induced by $K$ 0, with all updates and communications remaining sparse.
Communication analysis: Per-round uplink and downlink are reduced by a factor of the sparsity (e.g., 90% sparsity yields $K$ 1 decrease) (Ohib et al., 2023, Ohib et al., 2024).

2.2 Dynamic Personalized Sparse Masks

Other frameworks dynamically adapt masks per client during training, sometimes including prune/grow cycles:

FedSpa: Each client maintains a binary mask, initialized layerwise by Erdős–Rényi Kernel (ERK). After each round, a fraction $K$ 2 of currently active weights (e.g., smallest-magnitude) are pruned and regrown (e.g., via largest-magnitude gradients) (Huang et al., 2022). The pruning rate decays over rounds to stabilize the mask. Variants include static mask (RSM) and dynamic sparse training (DST), with the latter yielding best empirical results.
FedDST: Similar in spirit, but the prune/regrow cycle is scheduled periodically, and regrowth is performed randomly. Each client thus traverses a trajectory within the space of sparse subnetworks, resulting in an "in-time self-ensembling effect" that empirically boosts generalization in non-IID FL (Bibikar et al., 2021).

2.3 Adaptive and Resource-Aware Sparsification

Newer variants introduce sophisticated adaptation based on hardware and statistical heterogeneity:

FedLPS: Learns importance-associated sparse patterns per device, plus adaptive sparse ratios via a Prompt Upper Confidence Bound Variance (P-UCBV) controller for simultaneous personalization and resource self-adaptation (Xue et al., 2024). This suggests that personalized adaptive sparsity ratios outperform fixed settings in the presence of system heterogeneity.
pFedGate: Employs a trainable gating layer to determine block-wise sparse masks per batch and per client, optimizing mask structure for individual data distributions and resource profiles (Chen et al., 2023).

2.4 Decentralized and Proximity-Based Approaches

Sparse self-federated learning can be fully decentralized:

DisPFL: No central server; each client holds its own mask and parameters, communicates with neighbors, and only exchanges/update overlaps among masked supports ("sparse gossip averaging"). Per-client masks can vary in density to accommodate resource heterogeneity (Dai et al., 2022).
SParSeFuL: Devices self-organize into local federations based on data distribution similarity, then train sparse models via per-layer magnitude masks, achieving order-of-magnitude savings in communication and energy on large-scale IoT deployments (Domini et al., 10 Jul 2025).

3. Model Aggregation and Communication Complexity

Sparse self-federated learning radically changes the communication protocol as only parameters corresponding to nonzero mask entries are ever uploaded or received by clients (Huang et al., 2022, Dai et al., 2022, Ohib et al., 2023, Ohib et al., 2024). Aggregation strategies include:

Standard FedAvg with Masked Updates: Each update is sparse; server-side aggregation is restricted to the union or intersection of supports as determined by masks (Huang et al., 2022, Dai et al., 2022).
Majority-Vote Mask Merge: When clients propose separate sparse supports, the updated global mask may be set by majority vote over active entries (FedSparsify-Local), balancing the stability-accuracy trade-off (Stripelis et al., 2022).
Mask Alignment and Proximity: In dynamic or decentralized settings, aggregation is over intersections of sparsity masks; "gossip" averages only the weights shared among two or more clients (Dai et al., 2022).

Communication and computation savings are consistently proportional to the achieved sparsity. In settings with 90–99% sparsity, uplink and downlink costs are reduced by $K$ 3 or more, and per-round compute/FLOPs are similarly decreased (Huang et al., 2022, Bibikar et al., 2021, Ohib et al., 2023, Ohib et al., 2024, Domini et al., 10 Jul 2025). Table 1 below summarizes the efficiency gains reported in several works:

Method	Communication Savings	Compute Savings	Typical Sparsity Levels
FedSpa (DST)	$K$ 450%	$K$ 540%	0.5 (50%)
SSFL/SalientGrads	$K$ 6– $K$ 7	$K$ 8– $K$ 9	0.9–0.95 (90–95%)
FedDST	$\min_{w}~\frac{1}{K}\sum_{k=1}^K F_k(w)$ 0	$\min_{w}~\frac{1}{K}\sum_{k=1}^K F_k(w)$ 1	0.8–0.9 (80–90%)
SParSeFuL	$\min_{w}~\frac{1}{K}\sum_{k=1}^K F_k(w)$ 2	$\min_{w}~\frac{1}{K}\sum_{k=1}^K F_k(w)$ 3	0.95 (95%)

4. Personalization, Adaptivity, and Heterogeneity Support

Personalized sparsification is addressed through multiple mechanisms:

Per-client Mask Evolution: Each client prunes/grows its mask based on local gradients, data distributions, or dynamic hardware/resource constraints (Huang et al., 2022, Guastella et al., 7 Apr 2025, Domini et al., 10 Jul 2025).
Mask Consensus and Diversity: High mask overlap (IoU > 0.9) across clients and rounds signals stable generalization, even under non-IID data (Guastella et al., 7 Apr 2025). Decentralized protocols exploit task similarity discovery via emerging mask clusters (Dai et al., 2022).
Resource scaling: Sparsity levels are automatically adapted to client capacity in frameworks with trainable or resource-aware mask allocation (e.g., via gating networks or bandit-style adaptivity) (Xue et al., 2024, Chen et al., 2023).
Hybrid Decompositions: Some works, e.g., FedSLR, combine a low-rank global base with a client-specific sparse correction, balancing global knowledge with lightweight personalization (Huang et al., 2023).

A plausible implication is that adaptive, client-aware sparsification enables robust personalization and improved accuracy under practical deployment regimes characterized by high data and system heterogeneity.

5. Theoretical Guarantees and Empirical Outcomes

Sparse self-federated learning methods attain convergence guarantees on par with classical FL under mild smoothness and bounded heterogeneity assumptions:

Convergence Rates: Typical gradient norm convergence rates are $\min_{w}~\frac{1}{K}\sum_{k=1}^K F_k(w)$ 4 (nonconvex) or $\min_{w}~\frac{1}{K}\sum_{k=1}^K F_k(w)$ 5 (strongly convex or PŁ conditions) (Huang et al., 2022, Meinhardt et al., 2024, Bibikar et al., 2021, Deng et al., 2024).
Staleness and Error-Correction: Techniques such as error-feedback and accumulated regularized embeddings (FLARE) allow arbitrarily high sparsity (e.g., 99.999%) without sacrificing convergence, provided error accumulation and regularization are properly managed (Greidi et al., 2023).
Performance vs. Sparsity: Empirical results show only modest accuracy degradation up to 95% sparsity. For instance, on ResNet-18/CIFAR-10, dense accuracy 83.7%, 95% sparse models (SSFL/SparsyFed) achieve 82.6%/77.7% (Guastella et al., 7 Apr 2025, Ohib et al., 2024).

Key empirical findings include superior efficiency for dynamic sparse FL (FedDST, FedSpa) over static or random mask baselines, and improved fairness/bottom-decile accuracy through personalization mechanisms (pFedGate) (Huang et al., 2022, Chen et al., 2023). In decentralized settings, DisPFL and SParSeFuL demonstrate high personalized accuracy, rapid convergence, and energy savings in realistic IoT deployments (Dai et al., 2022, Domini et al., 10 Jul 2025).

6. Advanced Topics: Acceleration, Serverless FL, and Future Directions

Recent work applies advanced optimizers and accelerates federated sparse training:

Momentum and Adaptive Optimizers: Sparse FedAdam (FedAdam-SSM) introduces a shared sparse mask across local model, first, and second moment estimates, achieving 1.1–5.5 $\min_{w}~\frac{1}{K}\sum_{k=1}^K F_k(w)$ 6 faster convergence versus alternative Adam sparsifications (Deng et al., 2024).
Communication Acceleration with Sparsity: Sparse-ProxSkip leverages ProxSkip acceleration with client-side sparse masking. Theoretical analysis shows that client masking before communication preserves the accelerated rate and correct control variate dynamics (Meinhardt et al., 2024).
Self-federated, Serverless Protocols: Approaches such as SParSeFuL and proposed self-federated extensions of SSFL advocate completely decentralized learning: mask negotiation is driven by peer-to-peer saliency vector gossip or via locally adaptive mask blending, supporting high scalability and resilience (Domini et al., 10 Jul 2025, Ohib et al., 2024).

Open problems and limits include rigorous characterization of convergence under full decentralization and partial participation, development of hardware-friendly structured sparsity regimes, and combining sparsification with quantization or other compression techniques for further gains (Bibikar et al., 2021, Meinhardt et al., 2024, Domini et al., 10 Jul 2025).

7. Summary Table of Representative Sparse Self-Federated Methods

Method	Mask Type / Update	Personalization	Key Features	Reference
FedSpa	Dynamic per-client, DST	Yes	Mask prune/grow; per-client	(Huang et al., 2022)
SparsyFed	Dynamic global, Powerprop	No	Mask via Powerprop, Top-K	(Guastella et al., 7 Apr 2025)
SalientGrads	Static global (saliency)	No	SNIP-style mask init	(Ohib et al., 2023)
DisPFL	Dynamic per-client	Yes	Sparse gossip/peer averaging	(Dai et al., 2022)
SSFL	Static global (saliency)	No, extendable	Serverless mask consensus	(Ohib et al., 2024)
SParSeFuL	Dynamic per-device	Yes	Proximity-based federation	(Domini et al., 10 Jul 2025)
pFedGate	Trainable gating per client	Yes	Resource & data adaptivity	(Chen et al., 2023)
FLARE	Top-R with error-feedback	No	Staleness-correcting pull	(Greidi et al., 2023)
FedAdam-SSM	Shared Top-K mask	No	Adam, moment-aligned sparsify	(Deng et al., 2024)
Sparse-ProxSkip	Top-K, STE	No	Acceleration, local masking	(Meinhardt et al., 2024)

Sparse self-federated learning thus synthesizes advances in efficient distributed optimization, personalized modeling, and decentralized system design. It provides effective mechanisms for achieving the accuracy–efficiency trade-offs necessary for modern privacy-preserving, large-scale, and environmentally sustainable AI applications.