FedPuReL: Federated Learning via Gradient Purification

• FedPuReL is a federated learning method that uses gradient purification to preserve the inherent class-balanced behavior of zero-shot foundation models under long-tailed data distributions.
• It employs residual-based personalization to adapt to client-specific imbalances without degrading the global model's balanced performance.
• The method consistently outperforms baseline approaches, achieving significant accuracy gains and stability across various long-tailed non-IID datasets.

FedPuReL, short for Federated Learning via Gradient Purification and Residual Learning, is a method for long-tailed personalized federated learning with foundation models that is designed to preserve the class-balanced behavior of zero-shot models while still enabling client-specific adaptation under non-IID data heterogeneity (Hou et al., 4 May 2026). It is motivated by the observation that, in realistic personalized federated learning settings, long-tailed class distributions and client-level distribution shift co-occur, and that standard fine-tuning and conventional personalization can degrade performance by eroding the inherent balance of foundation models and propagating that bias into local models. FedPuReL addresses this with a two-stage design: balanced global training through gradient purification, followed by residual-based personalization on top of a frozen global model.

1. Problem formulation and learning objective

FedPuReL is formulated for a federated system with $K$ clients indexed by $k=1,\dots,K$, where client $k$ holds a local dataset

$D_k = \{(x_i,y_i)\}_{i=1}^{n_k},$

with $\sum_{k=1}^K n_k = N$ (Hou et al., 4 May 2026). The label space contains $C$ classes, whose global frequencies are

$n_c = \sum_{k=1}^K n_c^k,$

sorted as $n_1 \ge n_2 \ge \dots \ge n_C$. The degree of class imbalance is quantified by the imbalance factor

$IF = n_1 / n_C.$

Client-specific class proportions are

$p_k(c)=n_c^k/n_k,$

and may differ across clients; the data heterogeneity is often modeled by a Dirichlet$k=1,\dots,K$0 distribution over class counts.

The method builds on a frozen vision-language foundation model with parameters $k=1,\dots,K$1 and introduces small trainable “PEFT” parameters $k=1,\dots,K$2. Its explicit goal in long-tailed personalized federated learning is to learn two kinds of parameters. The first is a single global adapter $k=1,\dots,K$3 that, when combined with $k=1,\dots,K$4, remains balanced across all $k=1,\dots,K$5 classes under the global long-tailed distribution. The second is a set of local corrections $k=1,\dots,K$6, one per client, that adapt to each client’s idiosyncratic class imbalance without re-introducing global bias. This decomposition is central: the global component is intended to retain balanced knowledge, while the local component specializes.

2. Balancedness and the failure of naïve fine-tuning

A central empirical observation is that zero-shot CLIP already exhibits near-uniform per-class accuracy, referred to as balancedness, whereas fine-tuning on long-tailed data shifts predictions toward head classes (Hou et al., 4 May 2026). The paper characterizes this shift through temperature-aligned KL (TKL), which compares zero-shot logits $k=1,\dots,K$7 and fine-tuned logits $k=1,\dots,K$8 after entropy matching:

$k=1,\dots,K$9

$k$0

$k$1

$k$2

$k$3

Balancedness itself is measured by $k$4, following the formulation attributed in the paper to kang2020exploring:

$k$5

where $k$6 is the accuracy on class $k$7.

The reported empirical relation is direct: as TKL increases, $k$8 decreases; head-class accuracy increases while tail-class accuracy decreases, and overall performance falls below the zero-shot baseline. The paper therefore argues that the issue is not merely overfitting in the conventional sense, but the erosion of the balanced prior already embedded in the foundation model. It further identifies a second limitation: conventional personalization techniques based on parameter-level or feature-level fusion can transfer this bias from the global model into local personalized models. This suggests that, in long-tailed personalized federated learning, preserving zero-shot balancedness is not an auxiliary concern but a primary optimization constraint.

3. Gradient purification and balanced global updates

FedPuReL’s first stage is global balanced training through gradient purification (Hou et al., 4 May 2026). At each client and each local step, the method computes zero-shot logits $k$9 and fine-tuned logits $D_k = \{(x_i,y_i)\}_{i=1}^{n_k},$0, then forms two gradients with respect to $D_k = \{(x_i,y_i)\}_{i=1}^{n_k},$1: a task gradient from the cross-entropy loss and an alignment gradient from the TKL-based alignment loss.

The task loss is

$D_k = \{(x_i,y_i)\}_{i=1}^{n_k},$2

with gradient

$D_k = \{(x_i,y_i)\}_{i=1}^{n_k},$3

The alignment loss is derived from TKL between the zero-shot and fine-tuned predictions, with gradient

$D_k = \{(x_i,y_i)\}_{i=1}^{n_k},$4

The purification rule removes from $D_k = \{(x_i,y_i)\}_{i=1}^{n_k},$5 any component that opposes $D_k = \{(x_i,y_i)\}_{i=1}^{n_k},$6:

$D_k = \{(x_i,y_i)\}_{i=1}^{n_k},$7

and otherwise

$D_k = \{(x_i,y_i)\}_{i=1}^{n_k},$8

The local update is then

$D_k = \{(x_i,y_i)\}_{i=1}^{n_k},$9

After $\sum_{k=1}^K n_k = N$0 local steps, each client uploads $\sum_{k=1}^K n_k = N$1, and the server aggregates by weighted averaging:

$\sum_{k=1}^K n_k = N$2

The paper states that this ensures every aggregated update remains in the subspace that preserves zero-shot balance. Operationally, FedPuReL does not estimate class priors and does not rebalance by explicit frequency correction. Instead, it constrains optimization using the zero-shot model itself as a reference for balancedness.

4. Residual learning for unbiased personalization

After $\sum_{k=1}^K n_k = N$3 rounds of balanced global training, FedPuReL freezes $\sum_{k=1}^K n_k = N$4 and enters a second stage: personalized residual learning (Hou et al., 4 May 2026). For each client $\sum_{k=1}^K n_k = N$5, a private residual adapter $\sum_{k=1}^K n_k = N$6 is introduced and only $\sum_{k=1}^K n_k = N$7 is optimized.

The frozen global branch produces

$\sum_{k=1}^K n_k = N$8

while the personalized branch produces

$\sum_{k=1}^K n_k = N$9

The final client-specific prediction is additive:

$C$0

Training uses two losses. The first is a fusion loss,

$C$1

and the second is a personalization loss,

$C$2

The total objective is

$C$3

Gradients flow only into $C$4; both $C$5 and $C$6 remain fixed. The paper characterizes this as an “additive” residual that lets the client correct specific tail-class biases without disturbing the globally balanced backbone. Conceptually, the global model acts as a stable balanced prior, while the local residual expresses client-specific departures from that prior. A plausible implication is that FedPuReL treats personalization not as replacement of the global model but as a controlled correction layered on top of it.

5. Experimental setting and quantitative results

The empirical study covers six small long-tailed datasets—Food101-LT, DTD-LT, Aircraft-LT, Dogs-LT, Pets-LT, and CIFAR-100-LT—with $C$7, and two large benchmarks, ImageNet-LT and Places-LT, with $C$8; the default non-IID heterogeneity is $C$9 (Hou et al., 4 May 2026). Evaluation uses three metrics: global model test accuracy on a balanced global test set, reported as All/Many/Med/Few; personalized model local test accuracy averaged across clients; and balancedness $n_c = \sum_{k=1}^K n_c^k,$0 tracked through training rounds.

The paper reports that FedPuReL consistently outperforms state-of-the-art methods on both the global model and the personalized model. Representative headline results are summarized below.

Benchmark	Global model All	Personalized model All
ImageNet-LT	72.96% vs best prior 69.64%	70.12% vs 68.14%
Places-LT	43.88% vs 41.99%	39.68% vs 39.14%
CIFAR-100-LT IF=100	69.77% vs 65.14%	73.37% vs 71.79%

Beyond these benchmark numbers, the paper states that the method yields consistent gains across all imbalance-factor levels and all six smaller datasets, often by 3–10% absolute, and that it outperforms methods enhanced with explicit re-balancing, specifically Fed-GraB, despite using no class-prior estimates (Hou et al., 4 May 2026).

The analytical results are aligned with the method’s design. Balancedness $n_c = \sum_{k=1}^K n_c^k,$1 remains close to its zero-shot level across rounds, unlike standard fine-tuning where $n_c = \sum_{k=1}^K n_c^k,$2. Tail-class accuracy rises dramatically in the few-shot buckets while head-class accuracy remains stable. The norm of client drift, $n_c = \sum_{k=1}^K n_c^k,$3, is lower and more stable than in baseline methods, which the paper interprets as better cross-client consistency. In the branch-contribution analysis, the global branch dominates head classes, whereas the personalized branch steadily increases its share on tail classes; even on the fewest-shot classes, however, the global branch still contributes 20–25%.

6. Interpretation, misconceptions, and scope

FedPuReL is organized around two claims that run counter to common practice in federated adaptation of foundation models (Hou et al., 4 May 2026). First, fine-tuning need not improve over zero-shot performance in long-tailed personalized federated learning; the paper explicitly reports that fine-tuning can degrade performance below zero-shot baselines because it destroys the foundation model’s inherent class balance. Second, personalization is not automatically debiasing; conventional parameter-level or feature-level fusion can propagate the global model’s head-class bias into client-specific models.

The method’s response is correspondingly structured. Gradient purification “anchors” federated updates to the zero-shot balanced prior, and residual learning “unlocks” client-specific corrections only after the global model has been stabilized. The paper’s summary states that this two-stage design yields both a class-balanced global model and unbiased, effective personalization under long-tailed non-IID federated settings.

Several broader implications follow, although they should be read as interpretation rather than direct experimental claims. The results suggest that zero-shot predictions can function as a balance-preserving supervisory signal during federated adaptation, not merely as an initialization. They also suggest that, in long-tailed settings, the architectural separation between a frozen global branch and a trainable local residual may be more reliable than joint fine-tuning or direct fusion. The released implementation is available at the repository specified by the paper: https://github.com/shihaohou/FedPuReL.