---
title: 'FedPuReL: Federated Learning via Gradient Purification'
url: https://www.emergentmind.com/topics/fedpurel
type: topic
---

# FedPuReL: Federated Learning via Gradient Purification

FedPuReL, short for **Federated Learning via Gradient Purification and Residual Learning**, is a method for **long-tailed personalized federated learning** with foundation models that is designed to preserve the class-balanced behavior of zero-shot models while still enabling client-specific adaptation under non-IID data heterogeneity [2605.02247]. It is motivated by the observation that, in realistic personalized federated learning settings, long-tailed class distributions and client-level distribution shift co-occur, and that standard fine-tuning and conventional personalization can degrade performance by eroding the inherent balance of foundation models and propagating that bias into local models. FedPuReL addresses this with a two-stage design: balanced global training through **gradient purification**, followed by **residual-based personalization** on top of a frozen global model.

## 1. Problem formulation and learning objective

FedPuReL is formulated for a federated system with $K$ clients indexed by $k=1,\dots,K$, where client $k$ holds a local dataset

$$
D_k = \{(x_i,y_i)\}_{i=1}^{n_k},
$$

with $\sum_{k=1}^K n_k = N$ [2605.02247]. The label space contains $C$ classes, whose global frequencies are

$$
n_c = \sum_{k=1}^K n_c^k,
$$

sorted as $n_1 \ge n_2 \ge \dots \ge n_C$. The degree of class imbalance is quantified by the imbalance factor

$$
IF = n_1 / n_C.
$$

Client-specific class proportions are

$$
p_k(c)=n_c^k/n_k,
$$

and may differ across clients; the data heterogeneity is often modeled by a Dirichlet$(\alpha_{\text{dir}})$ distribution over class counts.

The method builds on a **frozen vision-language foundation model** with parameters $W$ and introduces small trainable “PEFT” parameters $\phi$. Its explicit goal in long-tailed personalized federated learning is to learn two kinds of parameters. The first is a **single global adapter** $\phi_g$ that, when combined with $W$, remains balanced across all $C$ classes under the global long-tailed distribution. The second is a set of **local corrections** $\phi_k$, one per client, that adapt to each client’s idiosyncratic class imbalance without re-introducing global bias. This decomposition is central: the global component is intended to retain balanced knowledge, while the local component specializes.

## 2. Balancedness and the failure of naïve fine-tuning

A central empirical observation is that **zero-shot CLIP already exhibits near-uniform per-class accuracy**, referred to as balancedness, whereas fine-tuning on long-tailed data shifts predictions toward head classes [2605.02247]. The paper characterizes this shift through **temperature-aligned KL** (TKL), which compares zero-shot logits $z(x)$ and fine-tuned logits $f(x)$ after entropy matching:

$$
\sigma_\tau(a)_c = \exp(a_c/\tau) \Big/ \sum_{j=1}^C \exp(a_j/\tau).
$$

$$
H(p) = -\sum_c p_c \log p_c.
$$

$$
H^* = \tfrac{1}{2}[H(\sigma_1(f)) + H(\sigma_1(z))].
$$

$$
\tau_f = H_f^{-1}(H^*), \qquad \tau_z = H_z^{-1}(H^*).
$$

$$
TKL(x) = D_{KL}(\sigma_{\tau_f}(f(x)) \,\|\, \sigma_{\tau_z}(z(x))).
$$

Balancedness itself is measured by $\beta(V)$, following the formulation attributed in the paper to *kang2020exploring*:

$$
\beta(V) = \frac{1}{C^2} \sum_{i,j} \exp(-|a_i-a_j|^2/\sigma),
$$

where $a_i$ is the accuracy on class $i$.

The reported empirical relation is direct: as TKL increases, $\beta(V)$ decreases; head-class accuracy increases while tail-class accuracy decreases, and overall performance falls below the zero-shot baseline. The paper therefore argues that the issue is not merely overfitting in the conventional sense, but the **erosion of the balanced prior** already embedded in the foundation model. It further identifies a second limitation: conventional personalization techniques based on parameter-level or feature-level fusion can transfer this bias from the global model into local personalized models. This suggests that, in long-tailed personalized federated learning, preserving zero-shot balancedness is not an auxiliary concern but a primary optimization constraint.

## 3. Gradient purification and balanced global updates

FedPuReL’s first stage is **global balanced training** through gradient purification [2605.02247]. At each client and each local step, the method computes zero-shot logits $z(x)$ and fine-tuned logits $f(x;W,\phi_g)$, then forms two gradients with respect to $\phi_g$: a task gradient from the cross-entropy loss and an alignment gradient from the TKL-based alignment loss.

The task loss is

$$
\mathcal{L}_{\text{task}} = CE(y,\sigma(f(\cdot;\phi_g))),
$$

with gradient

$$
g_{\text{task}} = \nabla_{\phi_g}\mathcal{L}_{\text{task}}.
$$

The alignment loss is derived from TKL between the zero-shot and fine-tuned predictions, with gradient

$$
g_{\text{align}} = \nabla_{\phi_g}\mathcal{L}_{\text{align}}.
$$

The purification rule removes from $g_{\text{task}}$ any component that opposes $g_{\text{align}}$:

$$
\text{if } \langle g_{\text{task}}, g_{\text{align}} \rangle < 0:
\qquad
\tilde g_{\text{task}} = g_{\text{task}} -
\frac{\langle g_{\text{task}}, g_{\text{align}} \rangle}{\|g_{\text{align}}\|^2} \cdot g_{\text{align}}
$$

and otherwise

$$
\tilde g_{\text{task}} = g_{\text{task}}.
$$

The local update is then

$$
\phi_g \leftarrow \phi_g - \eta \cdot \tilde g_{\text{task}}.
$$

After $E$ local steps, each client uploads $\phi_g^k$, and the server aggregates by weighted averaging:

$$
\phi_g \leftarrow \sum_{k\in S_t} \left( \frac{n_k}{\sum_j n_j} \right)\phi_g^k.
$$

The paper states that this ensures every aggregated update remains in the subspace that preserves zero-shot balance. Operationally, FedPuReL does not estimate class priors and does not rebalance by explicit frequency correction. Instead, it constrains optimization using the zero-shot model itself as a reference for balancedness.

## 4. Residual learning for unbiased personalization

After $T$ rounds of balanced global training, FedPuReL freezes $\phi_g$ and enters a second stage: **personalized residual learning** [2605.02247]. For each client $k$, a private residual adapter $\phi_k$ is introduced and only $\phi_k$ is optimized.

The frozen global branch produces

$$
\ell_G(x) = f(x;W,\phi_g),
$$

while the personalized branch produces

$$
\ell_P^k(x) = f(x;W,\phi_k).
$$

The final client-specific prediction is additive:

$$
\ell_{\text{final}}^k(x) = \ell_G(x) + \ell_P^k(x).
$$

Training uses two losses. The first is a fusion loss,

$$
\mathcal{L}_{\text{fusion}}^k = CE(y,\sigma(\ell_G+\ell_P^k)),
$$

and the second is a personalization loss,

$$
\mathcal{L}_{\text{personal}}^k = CE(y,\sigma(\ell_P^k)).
$$

The total objective is

$$
\mathcal{L}^k = (1-\lambda)\mathcal{L}_{\text{fusion}}^k + \lambda \mathcal{L}_{\text{personal}}^k.
$$

Gradients flow only into $\phi_k$; both $\phi_g$ and $W$ remain fixed. The paper characterizes this as an “additive” residual that lets the client correct specific tail-class biases without disturbing the globally balanced backbone. Conceptually, the global model acts as a stable balanced prior, while the local residual expresses client-specific departures from that prior. A plausible implication is that FedPuReL treats personalization not as replacement of the global model but as a controlled correction layered on top of it.

## 5. Experimental setting and quantitative results

The empirical study covers **six small long-tailed datasets**—Food101-LT, DTD-LT, Aircraft-LT, Dogs-LT, Pets-LT, and CIFAR-100-LT—with $IF \in \{50,100,200,500\}$, and **two large benchmarks**, ImageNet-LT and Places-LT, with $IF \approx 100$; the default non-IID heterogeneity is $\alpha_{\text{dir}}=1$ [2605.02247]. Evaluation uses three metrics: global model test accuracy on a balanced global test set, reported as All/Many/Med/Few; personalized model local test accuracy averaged across clients; and balancedness $\beta(V)$ tracked through training rounds.

The paper reports that FedPuReL consistently outperforms state-of-the-art methods on both the global model and the personalized model. Representative headline results are summarized below.

| Benchmark | Global model All | Personalized model All |
|---|---:|---:|
| ImageNet-LT | 72.96% vs best prior 69.64% | 70.12% vs 68.14% |
| Places-LT | 43.88% vs 41.99% | 39.68% vs 39.14% |
| CIFAR-100-LT IF=100 | 69.77% vs 65.14% | 73.37% vs 71.79% |

Beyond these benchmark numbers, the paper states that the method yields **consistent gains across all imbalance-factor levels and all six smaller datasets**, often by **3–10% absolute**, and that it outperforms methods enhanced with explicit re-balancing, specifically **Fed-GraB**, despite using no class-prior estimates [2605.02247].

The analytical results are aligned with the method’s design. Balancedness $\beta(V)$ remains close to its zero-shot level across rounds, unlike standard fine-tuning where $\beta \to 0$. Tail-class accuracy rises dramatically in the few-shot buckets while head-class accuracy remains stable. The norm of client drift, $\|\phi_k-\phi_g\|_2$, is lower and more stable than in baseline methods, which the paper interprets as better cross-client consistency. In the branch-contribution analysis, the global branch dominates head classes, whereas the personalized branch steadily increases its share on tail classes; even on the fewest-shot classes, however, the global branch still contributes **20–25%**.

## 6. Interpretation, misconceptions, and scope

FedPuReL is organized around two claims that run counter to common practice in federated adaptation of foundation models [2605.02247]. First, **fine-tuning need not improve over zero-shot performance** in long-tailed personalized federated learning; the paper explicitly reports that fine-tuning can degrade performance below zero-shot baselines because it destroys the foundation model’s inherent class balance. Second, **personalization is not automatically debiasing**; conventional parameter-level or feature-level fusion can propagate the global model’s head-class bias into client-specific models.

The method’s response is correspondingly structured. Gradient purification “anchors” federated updates to the zero-shot balanced prior, and residual learning “unlocks” client-specific corrections only after the global model has been stabilized. The paper’s summary states that this two-stage design yields both a class-balanced global model and unbiased, effective personalization under long-tailed non-IID federated settings.

Several broader implications follow, although they should be read as interpretation rather than direct experimental claims. The results suggest that zero-shot predictions can function as a **balance-preserving supervisory signal** during federated adaptation, not merely as an initialization. They also suggest that, in long-tailed settings, the architectural separation between a frozen global branch and a trainable local residual may be more reliable than joint fine-tuning or direct fusion. The released implementation is available at the repository specified by the paper: `https://github.com/shihaohou/FedPuReL`.

Source: https://www.emergentmind.com/topics/fedpurel