FedPruner: Federated Pruning Methods

• FedPruner is a research category encompassing federated learning frameworks that integrate pruning during optimization to reduce communication, memory, and computational costs.
• It includes variants such as personalized masked pruning, structured filter pruning, and layer-pruning for LLM fine-tuning, each addressing system heterogeneity and client constraints.
• Empirical studies demonstrate that moderate pruning maintains accuracy while achieving significant savings in communication, memory footprint, and inference speed.

FedPruner denotes a family of federated learning methods that incorporate pruning into the training loop in order to reduce communication, memory, and computation, while in several formulations also addressing system heterogeneity, personalization, or privacy leakage. In the literature, the name has been used for multiple, non-identical frameworks: resource-aware submodel serving for straggling clients, masked parameter pruning under model heterogeneity, server-side structured filter pruning, and layer-pruned federated fine-tuning of LLMs. Closely related work such as FedSparsify and FedPrune is often presented as a concrete realization of the same broader federated pruning archetype (Munir et al., 2021, Lin et al., 2022, Yi et al., 2024, Nguyen et al., 2024, Wu et al., 24 Aug 2025).

1. Terminological scope and research lineage

The term does not refer to a single canonical algorithm. Instead, it spans several pruning-centric federated frameworks that share a common systems objective: train useful models when dense federation is constrained by uplink/downlink cost, on-device memory, local compute, or heterogeneous client capabilities. This usage appears across at least five distinct lines of work.

One line, FedPrune, addresses inclusive participation by pruning the global model for slow clients and combining updates with a Central Limit Theorem-inspired aggregation rule under non-IID data (Munir et al., 2021). A second line, “Federated Pruning,” formulates sparse subnetwork training with explicit masks, shrink/expand operators, structured pruning patterns, mask refinement, and adaptive per-layer sparsity for Automatic Speech Recognition (Lin et al., 2022). A third line, FedSparsify, performs progressive magnitude pruning during federated training and is explicitly described as instantiating what one might call a “FedPruner” framework (Stripelis et al., 2022, Stripelis et al., 2022). A fourth line, FedPruner formerly “FedP3,” targets model heterogeneity through client-specific sparsity budgets and masked personalized subnetworks (Yi et al., 2024). A fifth line, “Automatic Structured Pruning for Efficient Architecture in Federated Learning,” uses the name FedPruner for server-side structured filter pruning that preserves dense convolutional kernels for mobile deployment (Nguyen et al., 2024). The name is further reused for federated fine-tuning of LLMs via layer pruning and a fine-grained MHA/FFN variant, FedPruner$^{+}$ (Wu et al., 24 Aug 2025).

This multiplicity of usages suggests that “FedPruner” is best understood as a research category rather than a standardized protocol. The unifying idea is to move pruning from an offline model-compression step into the federated optimization process itself, either by maintaining masks, pruning filters or layers after aggregation, or assigning different subnetworks to different clients.

2. Optimization formulations and federated update mechanics

A standard formalization introduces a global parameter vector and client-specific binary masks. In the personalized masked setting, client $i$ has loss

$$L_i(w) \coloneqq \mathbb E_{(x,y)\sim X_i}[\ell(w;x,y)],$$

and the federated objective becomes

$$\min_{w,\{m_i\}} \sum_i p_i\,L_i(m_i\odot w) \quad \text{subject to} \quad \|m_i\|_0 \le (1-s_i)d,\ \forall i,$$

where $m_i\in\{0,1\}^d$ and $s_i$ is the target sparsity for client $i$ (Yi et al., 2024). In the global-mask setting used in federated pruning for ASR, the objective is

$$\min_{\theta,m} \sum_{i=1}^K \mathbb E_{(x,y)\sim D_i}[\ell((\theta\odot m);x,y)] \quad \text{subject to} \quad \|m\|_0 \le S,\ m_j\in\{0,1\},$$

with $m$ defining a sparse subnetwork shared across clients (Lin et al., 2022).

The most common algorithmic pattern is server-side broadcast of either a masked model or a pruned dense architecture, local training restricted to the retained parameters, and sparse or masked aggregation. In FedSparsify, after aggregation the server computes a target sparsity

$$s_t = S_T + (S_0 - S_T)\left(1 - \frac{F\lfloor t/F\rfloor - t_0}{T - t_0}\right)^n,$$

then derives a binary mask by magnitude thresholding,

$i$0

and enforces the mask during local SGD so that pruned weights are not updated or resurrected (Stripelis et al., 2022). In FedPruner formerly “FedP3,” the server computes a mask $i$1 per selected client, sends the trainable subnetwork $i$2, the client performs $i$3 local steps with zero gradient on masked coordinates, and returns

$i$4

which is aggregated as

$i$5

(Yi et al., 2024).

A structurally different design appears in automatic structured FedPruner. There, clients never receive masks and never execute sparse kernels. Instead, each round clients train a dense architecture; the server aggregates via FedAvg, prunes complete convolutional filters using server-side statistics, produces a smaller dense model, and repeats until no further filters can be removed for $i$6 consecutive rounds (Nguyen et al., 2024). In LLM fine-tuning, the pruning object is not an individual scalar weight or filter but a transformer layer or a component such as MHA or FFN, and the communication object is the LoRA update rather than the full pretrained parameter tensor (Wu et al., 24 Aug 2025).

3. Principal design families

The literature partitions naturally by pruning granularity and by where the pruning logic resides.

Variant	Pruning unit	Salient mechanism
Progressive masked pruning	Individual weights	Magnitude-based mask, progressive sparsity schedule, masked local SGD
Personalized masked pruning	Individual weights per client	Client-specific sparsity budgets and client-specific masks
Structured architecture pruning	Convolutional filters	Server-side layer-wise filter removal with dense execution on clients
Layer-pruned federated fine-tuning	Transformer layers or MHA/FFN units	FDLO + IALS to build device-specific submodels
Differential model serving	Neurons or filters for slow clients	Activation-guided mask updates and pruned submodels for stragglers

Progressive weight pruning is exemplified by FedSparsify. Its core assumptions are that pruning can occur after each aggregation, that the mask should be shared globally, and that no regrowth of pruned parameters is needed for stable training. The 2022 neuroimaging formulation sends the sparse model and mask every round, and clients multiply both gradients and weights by the mask to prevent updates on zeroed coordinates (Stripelis et al., 2022). The closely related “Purge–Merge–Tune” account organizes the method into Purge, Merge, and Tune phases and distinguishes a global-prune variant from a local-prune variant with majority-vote mask aggregation (Stripelis et al., 2022).

Personalized masked pruning is represented by FedPruner formerly “FedP3.” Its defining feature is system heterogeneity: each client is assigned its own pruning constraint according to compute, memory, and bandwidth, so the objective becomes a joint optimization over shared global parameters and individualized masks (Yi et al., 2024). This differs from a single global sparsity schedule because the subnetwork allocated to each client is resource-dependent by construction.

Structured architecture pruning is represented by the automatic structured FedPruner. For each convolutional layer with $i$7 filters, the server computes a filter score

$i$8

then obtains $i$9 and $$L_i(w) \coloneqq \mathbb E_{(x,y)\sim X_i}[\ell(w;x,y)],$$0 across filters and removes filters whose scores lie outside the interval $$L_i(w) \coloneqq \mathbb E_{(x,y)\sim X_i}[\ell(w;x,y)],$$1 with

$$L_i(w) \coloneqq \mathbb E_{(x,y)\sim X_i}[\ell(w;x,y)],$$2

Because entire filters are removed, the resulting model remains dense and requires no sparse-matrix support on mobile devices (Nguyen et al., 2024).

Layer-pruned federated fine-tuning generalizes the idea to transformer depth. FedPruner for LLMs computes inter-layer similarity by Centered Kernel Alignment, partitions layers into groups by spectral clustering, then samples one representative layer per group using an importance-aware probability derived from feature-transformation significance. FedPruner$$L_i(w) \coloneqq \mathbb E_{(x,y)\sim X_i}[\ell(w;x,y)],$$3 further decomposes each transformer block into MHA and FFN units and performs the same macro-micro procedure on $$L_i(w) \coloneqq \mathbb E_{(x,y)\sim X_i}[\ell(w;x,y)],$$4 units rather than $$L_i(w) \coloneqq \mathbb E_{(x,y)\sim X_i}[\ell(w;x,y)],$$5 whole layers (Wu et al., 24 Aug 2025).

A distinct but related design is differential model serving for slow clients. FedPrune first uses random pruning masks during bootstrap rounds, then replaces them with activation-guided masks based on aggregated post-activation averages, while fast clients continue to receive the full model (Munir et al., 2021).

4. Empirical behavior across tasks and architectures

Across the reported studies, pruning is repeatedly shown to reduce communication and local resource use while preserving accuracy within a substantial sparsity range, although the stable range is strongly task- and architecture-dependent.

In federated neuroimaging for brain age prediction, FedSparsify reports that models can be pruned up to 95% sparsity without affecting performance even in highly heterogeneous data distributions. In the skewed-IID environment, the dense baseline has 2.95 M parameters, size 10.85 MB, communication 1,888 $$L_i(w) \coloneqq \mathbb E_{(x,y)\sim X_i}[\ell(w;x,y)],$$6 scalars, test MAE 2.879, MIA accuracy 0.660, and throughput 64.3 img/s. At 95% sparsity, the model has 0.15 M parameters, size 0.73 MB, communication 576 $$L_i(w) \coloneqq \mathbb E_{(x,y)\sim X_i}[\ell(w;x,y)],$$7 scalars, test MAE 2.861, MIA accuracy 0.659, and throughput 78.3 img/s. The paper summarizes this point as communication $$L_i(w) \coloneqq \mathbb E_{(x,y)\sim X_i}[\ell(w;x,y)],$$8M versus 1888M, a 3.3$$L_i(w) \coloneqq \mathbb E_{(x,y)\sim X_i}[\ell(w;x,y)],$$9 reduction, a model 67$$\min_{w,\{m_i\}} \sum_i p_i\,L_i(m_i\odot w) \quad \text{subject to} \quad \|m_i\|_0 \le (1-s_i)d,\ \forall i,$$0 smaller, and inference speed up $$\min_{w,\{m_i\}} \sum_i p_i\,L_i(m_i\odot w) \quad \text{subject to} \quad \|m_i\|_0 \le (1-s_i)d,\ \forall i,$$1. At 99% sparsity, throughput reaches 128.6 img/s, but test MAE degrades to 3.024 (Stripelis et al., 2022).

In automatic structured FedPruner, the strongest reported compression occurs for FEMNIST with an Inception model: parameters drop from 246 236 to 26 744, a reduction of 89%, and FLOPS fall from 31 M to 3 M, a reduction of 90%, while accuracy changes from 80.19% to 78.81%. The same study reports communication savings of up to 5$$\min_{w,\{m_i\}} \sum_i p_i\,L_i(m_i\odot w) \quad \text{subject to} \quad \|m_i\|_0 \le (1-s_i)d,\ \forall i,$$2 less total bytes over 100 global rounds and inference time up to 50% faster on Google Pixel 3, Samsung A9, Samsung A53, Dell Inspiron i7-7700, and Dell Desktop i7-4790 (Nguyen et al., 2024).

In federated ASR, “Federated Pruning” reports that up to approximately 30% sparsity, WER rises only by about 0.1–0.2% absolute on LibriSpeech, and that on multi-domain voice search the WER moves from 6.4 at sparsity 0.0 to 6.7 at 0.10 and 7.0 at 0.20. The same work states that FedPruner reduces on-device training memory and communication by up to 50% while preserving nearly full accuracy, specified as $$\min_{w,\{m_i\}} \sum_i p_i\,L_i(m_i\odot w) \quad \text{subject to} \quad \|m_i\|_0 \le (1-s_i)d,\ \forall i,$$3 WER loss for $$\min_{w,\{m_i\}} \sum_i p_i\,L_i(m_i\odot w) \quad \text{subject to} \quad \|m_i\|_0 \le (1-s_i)d,\ \forall i,$$4 (Lin et al., 2022).

In LLM fine-tuning, FedPruner on LLaMA2-7B (INT4) reports Close-Ended Avg 41.99 and Open-Ended Avg 5.59, compared with 40.27 and 4.98 for LoRA, while memory usage is 25%, corresponding to a 75% reduction. FedPruner$$\min_{w,\{m_i\}} \sum_i p_i\,L_i(m_i\odot w) \quad \text{subject to} \quad \|m_i\|_0 \le (1-s_i)d,\ \forall i,$$5 reports 42.46 and 5.74 at the same 25% memory usage. The abstract characterizes the overall result as up to a 1.98% improvement in average model accuracy while reducing peak memory usage by 75% (Wu et al., 24 Aug 2025).

These empirical envelopes are reported on different tasks—brain MRI regression, image classification, ASR, and instruction tuning—so direct numeric comparison is not straightforward. The consistent pattern is that moderate or progressive pruning is usually compatible with dense-model performance, whereas very aggressive pruning produces a visible accuracy or quality penalty.

5. Personalization, heterogeneity, and privacy

FedPruner has been repeatedly motivated by system heterogeneity. In the personalized FedPruner framework, each client is assigned a pruning constraint according to available memory, compute speed, and bandwidth; the server may re-prune globally and then customize the mask $$\min_{w,\{m_i\}} \sum_i p_i\,L_i(m_i\odot w) \quad \text{subject to} \quad \|m_i\|_0 \le (1-s_i)d,\ \forall i,$$6 to meet each client’s budget, so that over time each client receives the “best” subnetwork under its constraint (Yi et al., 2024). In the LLM setting, the analogous constraint is a device memory budget of 3–9 GB, and the goal is to build device-specific submodels by pruning layers while still aggregating LoRA updates at the server (Wu et al., 24 Aug 2025).

FedPrune addresses heterogeneity by differential model serving. Fast clients receive the full model, while slow clients receive a pruned submodel chosen to match device characteristics. The reported real-device result is that, at 50% prune, training time on a Nokia 1 shrinks from 33.1 min to 11.6 min, bringing it within 1.1$$\min_{w,\{m_i\}} \sum_i p_i\,L_i(m_i\odot w) \quad \text{subject to} \quad \|m_i\|_0 \le (1-s_i)d,\ \forall i,$$7 of the Nexus 6P, and the corresponding FLOPs drop by 3.8$$\min_{w,\{m_i\}} \sum_i p_i\,L_i(m_i\odot w) \quad \text{subject to} \quad \|m_i\|_0 \le (1-s_i)d,\ \forall i,$$8. In simulation on FEMNIST, under 50% slow clients FedPrune reaches 80% test-accuracy in 75 rounds versus 150 for FedAvg; under 90% slow clients it outperforms by 22.7% absolute accuracy at round 200; and it reduces the standard deviation of per-client test-accuracy by up to 30% relative to FedAvg (Munir et al., 2021).

Privacy enters the FedPruner literature in two different senses. First, sparsity itself can reduce privacy leakage. In the neuroimaging study, centralized membership inference attack accuracy drops from approximately 0.72 at 0% sparsity to approximately 0.55 at at least 95% sparsity; in the federated setting the baseline MIA is approximately 0.66 and drops to approximately 0.60 at 99% sparsity, described as a 5–20% reduction depending on split (Stripelis et al., 2022). Second, some formulations explicitly discuss differential privacy. FedPruner formerly “FedP3” includes a locally differential-private variant, DP-FedP3, and sketches a clipped-and-noised Gaussian mechanism, but the draft does not fix $$\min_{w,\{m_i\}} \sum_i p_i\,L_i(m_i\odot w) \quad \text{subject to} \quad \|m_i\|_0 \le (1-s_i)d,\ \forall i,$$9 nor give a full proof. It states only that the privacy-friendly approach can be augmented by clipping the masked update, adding Gaussian noise, and tracking privacy loss by a moments accountant (Yi et al., 2024).

A recurrent misconception is that pruning is itself a formal privacy guarantee. The cited works do not support that conclusion. Lower MIA accuracy is an empirical privacy effect, whereas $m_i\in\{0,1\}^d$0-DP requires an explicitly specified mechanism and accounting procedure.

6. Theory, limitations, and adjacent extensions

The theoretical picture is partial and method-specific. FedSparsify reports no formal convergence proof in the neuroimaging study, but empirical stability shows that it converges within 1–2 rounds of FedAvg up to 95% sparsity, while 99% pruning can slow convergence and slightly degrade final accuracy (Stripelis et al., 2022). The “Purge–Merge–Tune” analysis is more explicit: under standard smoothness, bounded-variance, and unbiased-gradient assumptions, Theorem 1 is summarized as an $m_i\in\{0,1\}^d$1 convergence rate similar to FedAvg, with an additional term

$m_i\in\{0,1\}^d$2

that measures the magnitude of the pruned weights; pruning the smallest-magnitude parameters minimizes this extra cost (Stripelis et al., 2022). The personalized FedPruner draft offers a theorem sketch under L-smoothness and bounded variance, stating

$m_i\in\{0,1\}^d$3

with the additive term attributed to subnetwork error, but also states that the current draft does not contain a full convergence proof (Yi et al., 2024).

The practical limits are likewise consistent across papers. Extreme pruning is usually the failure mode. The neuroimaging results identify 95% sparsity as a practical upper regime and 99% as the point where MAE begins to degrade (Stripelis et al., 2022). In FedPhD, which is explicitly described as something that can be seen as a “FedPruner” for diffusion models, very aggressive sparsity above 60% is said to hurt image quality, with the best reported trade-off occurring at a 44% pruning ratio. FedPhD combines hierarchical FL, homogeneity-aware aggregation, and dependency-aware structured pruning, reporting up to 88% reduction in cumulative communication cost, at least a 34% improvement in FID, and only 56% of the total computation and communication resources of competitive baselines (Long et al., 8 Jul 2025).

Viewed across these works, the central research problem is not whether pruning can be inserted into federated learning, but how to choose the pruning unit, the schedule, and the aggregation rule so that resource savings do not destabilize optimization or amplify heterogeneity. The literature therefore treats parameters such as final sparsity $m_i\in\{0,1\}^d$4, schedule parameters $m_i\in\{0,1\}^d$5, layer or client budgets, boundary hyperparameter $m_i\in\{0,1\}^d$6, and noise or regularization terms as first-class design variables rather than secondary implementation details.