Fine-Tuning Impairs the Balancedness of Foundation Models in Long-tailed Personalized Federated Learning

Abstract: Personalized federated learning (PFL) with foundation models has emerged as a promising paradigm enabling clients to adapt to heterogeneous data distributions. However, real-world scenarios often face the co-occurrence of non-IID data and long-tailed class distributions, presenting unique challenges that remain underexplored in PFL. In this paper, we investigate this long-tailed personalized federated learning and observe that current methods suffer from two limitations: (i) fine-tuning degrades performance below zero-shot baselines due to the erosion of inherent class balance in foundation models; (ii) conventional personalization techniques further transfer this bias to local models through parameter or feature-level fusion. To address these challenges, we propose Federated Learning via Gradient Purification and Residual Learning (FedPuReL), which preserves balanced knowledge in the global model while enabling unbiased personalization. Specifically, we purify local gradients using zero-shot predictions to maintain a class-balanced global model, and model personalization as residual correction atop the frozen global model. Extensive experiments demonstrate that FedPuReL consistently outperforms state-of-the-art methods, achieving superior performance on both global and personalized models across diverse long-tailed scenarios. The code is available at https://github.com/shihaohou/FedPuReL.

• The paper shows that fine-tuning erodes the inherent balanced predictions of pre-trained foundation models on non-IID, long-tailed data.
• It introduces FedPuReL, a framework using gradient purification and residual logit corrections to maintain unbiased global aggregation.
• Experimental evaluations on CIFAR-100-LT and ImageNet-LT demonstrate significantly improved tail-class accuracy and overall balance.

Fine-Tuning Impairs Balancedness in Long-Tailed Personalized Federated Learning: An Analysis and FedPuReL

Problem Definition and Empirical Findings

Personalized federated learning (PFL) with foundation models allows clients to adapt large-scale pre-trained models (e.g., CLIP) to heterogeneous, locally collected data while maintaining privacy constraints. However, a salient and underexplored challenge arises when local datasets are non-IID and exhibit long-tailed class distributions: class-imbalance phenomena amplify during federated adaptation.

The paper identifies two primary issues in existing PEFT-based PFL methodologies:

• Erosion of Pre-trained Balancedness: Fine-tuning, either globally or locally, diminishes the inherent cross-class balance of foundation models; global models become biased toward head classes, and even established correction techniques (e.g., logit adjustment) fail to restore zero-shot balancedness.
• Bias Propagation via Personalization: Common parameter or feature fusion personalizations transfer the global model's amplified bias to local models, resulting in persistent head-class overfitting and further degradation on tail classes.

The authors substantiate these phenomena quantitatively using temperature-aligned KL divergence (TKL) and a balancedness metric, demonstrating a robust negative correlation: as the global model shifts from its zero-shot predictions, balancedness decays precipitously (Figure 1).

Figure 1: (a) TKL divergence and balancedness are strongly negatively correlated as federated fine-tuning progresses; (b) accuracy improvement on head classes comes at the cost of tail-class degradation.

Notably, this balance erosion occurs even when leveraging prompt- and LoRA-based parameter-efficient architectures (Figure 2).

Figure 2: Aggregation of global parameters and feature fusion mechanisms induce significant balancedness loss during federated adaptations.

Methodological Contributions: The FedPuReL Framework

FedPuReL (Federated Learning via Gradient Purification and Residual Learning) addresses both challenges by decoupling the adaptation and balancing processes:

• Gradient Purification: During global model updates, each client constrains its gradient direction so that it does not deviate from the (balanced) zero-shot model's output distribution. Specifically, the gradient component conflicting with balance preservation (as measured by TKL divergence from zero-shot predictions) is projected out, ensuring that global aggregation maintains class-equitable representations.

  Figure 3: FedPuReL: Clients update shared PEFT parameters using purified gradients guided by zero-shot prediction alignment (left); the server aggregates only the balanced global parameters (center); personalization proceeds as separate residuals (right).

• Personalization via Output-Space Residuals: Instead of fusing global and local parameters directly, client-specific adaptation is formulated as a “residual” logit correction over the frozen global model. At inference, each client sums outputs from the balanced global branch and its local residual branch, effectively absorbing local data distribution peculiarities without inheriting global head-class bias.

This dual-branch personalization ensures that even under severe imbalance, the global model remains stable and unbiased while the personalized model efficiently absorbs heterogeneity.

Experimental Evidence and Analysis

The framework is benchmarked on CIFAR-100-LT, ImageNet-LT, Places-LT, and a suite of fine-grained and texture datasets with varying imbalance factors and client heterogeneity. Across all settings, FedPuReL demonstrates dominance in both global and personalized metrics, outperforming prompt, adapter, and LoRA-based federated SOTA—including recent gradient balancing methods—especially in tail-class accuracy and balancedness (see experimental tables in the paper).

For instance, on CIFAR-100-LT (IF=100), FedPuReL’s global branch reaches 69.77% while the best prompt-based baseline delivers 65.14% (Table 2 in the paper). The advantage widens as class imbalance intensifies. On ImageNet-LT, FedPuReL leads by several points on few-shot classes compared to both PromptFolio+Fed-Grab and CLIPLoRA+Fed-Grab.

The balancedness improvement is visualized in Figure 4, where FedPuReL maintains class-equitable performance in both global and personalized models; alternative methods display pronounced degradation, especially personalized ones.

Figure 4: FedPuReL maintains superior balancedness throughout training for both global and personalized paths on CIFAR-100-LT.

Further analysis reveals:

• Client Drift Reduction: By anchoring local updates to the zero-shot manifold, FedPuReL reduces the $L^2$ divergence between client and global parameters, further accelerating convergence and ensuring stability (Figure 5a).
• Complementary Branch Contributions: The global branch dominates head-class predictions, whereas the personalized branch’s contribution increases for tail classes—without ever fully supplanting the role of the global model, which consistently confers class balance (Figure 5b).

Figure 5: (a) FedPuReL suppresses client drift relative to baselines. (b) Global branch is essential for head classes; personalized branch is critical for tail classes.

• Gradient Alignment Dynamics: The angles between task and alignment gradients empirically validate that without purification, updates are typically orthogonal and thus do not preserve pre-trained balance (Figure 6).
• TKL over Standard KL: Ablation studies show that TKL, by normalizing confidence and isolating class-wise drift, is a more discriminative proxy for genuine bias than standard KL, which conflates bias and overall model confidence (Figure 7).

Figure 7: TKL cleanly separates balanced adaptation from true class bias, unlike standard KL.

Implications and Future Directions

Practical: FedPuReL enables deployment of large vision-LLMs in privacy-sensitive, non-IID federated environments with long-tailed distributions, mitigating risk of fairness violations and head-class collapse. Because gradient purification operates without using explicit class priors, it avoids challenges of privacy leakage and prior estimation inherent to other rebalancing strategies.

Theoretical: The empirical observation that fine-tuning erodes equilibrium learned from large-scale pretraining points to an underappreciated trade-off in FL. The success of gradient purification suggests that regularizing toward the structural “anchor” provided by zero-shot predictions is essential to retaining socially desirable generalization properties.

Outlook: The architectural decoupling in FedPuReL provides a template for federated learning with foundation models in other modalities and domains, potentially informing future advances in federated prompt learning and LLM deployment. Future research can investigate richer residual parameterizations and more adaptive forms of gradient purification, as well as the impact on privacy and security guarantees.

Conclusion

This paper systematically exposes the pathological effects of naive fine-tuning on class balance within foundation-model-based PFL under long-tailed distributions. The FedPuReL framework, through zero-shot-driven gradient purification and decoupled residual personalization, consistently maintains global class balance and achieves unbiased client adaptation. These findings set a new methodological standard for fairness-aware adaptation of foundation models in federated and heterogeneous data contexts.