Taming Noise-Induced Prototype Degradation for Privacy-Preserving Personalized Federated Fine-Tuning

Abstract: Prototype-based Personalized Federated Learning (ProtoPFL) enables efficient multi-domain adaptation by communicating compact class prototypes, but directly sharing them poses privacy risks. A common defense involves per-example $\ell_2$ clipping before prototype computation to bound sensitivity, followed by isotropic Gaussian noise to enforce Local Differential Privacy (LDP). However, Isotropic Gaussian Prototype Perturbation (IGPP) typically over-perturbs discriminative dimensions and struggles to balance the clipping threshold with representation fidelity. In this paper, we propose VPDR, a client-side privacy plug-in that seamlessly integrates into existing ProtoPFLs. Motivated by the observation that dimension-wise class variance reflects discriminability, we introduce Variance-adaptive Prototype Perturbation (VPP), which allocates less noise to discriminative subspaces, preserving semantic separability while ensuring privacy. We further develop Distillation-guided Clipping Regularization (DCR), which enables feature norms to adaptively concentrate near the predefined clipping threshold while maintaining prediction consistency. Theoretical analysis shows that our groupwise mechanism provides privacy guarantees no weaker than the isotropic baseline under the same privacy constraints. Extensive experiments on multi-domain benchmarks demonstrate that VPDR achieves a superior privacy-utility trade-off, outperforming IGPP in personalized federated fine-tuning without sacrificing robustness against realistic attacks.

• The paper introduces VPDR, a method that adaptively perturbs prototypes to preserve discriminative features while enforcing strong local differential privacy.
• It employs variance-adaptive prototype perturbation to reduce noise in key feature dimensions and distillation-guided clipping regularization to maintain feature norm integrity.
• Empirical evaluations across multiple benchmarks show improved accuracy, reduced performance variance, and robust privacy protection with modest computational overhead.

Taming Noise-Induced Prototype Degradation for Privacy-Preserving Personalized Federated Fine-Tuning

Introduction and Problem Statement

Prototype-based Personalized Federated Learning (ProtoPFL) enables multi-domain model adaptation by exchanging global or local class prototypes (e.g., class means) rather than full model gradients or parameters. While this paradigm yields substantial communication and statistical efficiency, prototype sharing exposes sensitive semantic structure, especially in low-sample regimes where prototypes may closely resemble individual instances. To mitigate these privacy threats, existing defenses typically employ per-example $\ell_2$ clipping followed by isotropic Gaussian noise addition, enforcing client-side Local Differential Privacy (LDP). However, as visualized in Figure 1, this Isotropic Gaussian Prototype Perturbation (IGPP) suffers from two fundamental drawbacks: (1) dimension-agnostic noise allocation that degrades discriminative feature space geometry, and (2) an inherent privacy–utility tradeoff in setting the clipping bound.

Figure 1: IGPP degrades class separability by moving prototypes along discriminative dimensions; aggressive or lax clipping induces either over-noising or semantics loss.

VPDR: Variance- and Clipping-Adaptive Privacy for ProtoPFL

The paper introduces VPDR, a plug-in for ProtoPFL frameworks, consisting of two key modules:

• Variance-adaptive Prototype Perturbation (VPP): Allocates less perturbation to discriminative subspaces by privately quantifying dimension-wise discriminability via normalized ANOVA scores. This allows massive noise to be shifted to redundant dimensions without weakening LDP guarantees, thus preserving task-relevant information.
• Distillation-guided Clipping Regularization (DCR): Applies a differentiable soft-clipping layer to the feature encoder, coupled with knowledge distillation from an EMA teacher that operates on unclipped features, ensuring feature norms concentrate near the chosen clipping threshold and preventing the classifier from circumventing clipping via parameter scaling.

The combination of VPP and DCR directly addresses the two limitations of IGPP by (i) respecting geometric structure in prototype privatization, and (ii) resolving the clipping/budget dilemma via norm regularization.

Figure 2: ProtoPFL with VPDR: private prototype calculation via VPP, upload with budgeted noise, server-side aggregation, and fine-tuning with DCR.

Methodological Details

Variance-Adaptive Prototype Perturbation

Each client computes per-dimension discriminativity scores $S_j$ by fusing intra- and inter-class variances using an ANOVA-inspired normalization, yielding a privacy-sensitive subspace selection problem. The dimension scores are privatized using the oneshot Laplace Top-$k$ mechanism under the allocated sub-budget $\epsilon_1$, with the top $\rho d$ dimensions forming the discriminative subspace. Groupwise $\ell_2$ clipping and anisotropic Gaussian noise are then applied to discriminative and residual groups, with the groupwise noise variances calibrated to ensure the total LDP guarantee is retained according to a harmonic sum constraint.

Figure 3: Empirical scatterplot of the ANOVA-normalized score $S_j$ vs. label mutual information $I(z_j; y)$ confirms $S_j$'s reliability as a proxy for discriminability.

Distillation-Guided Clipping Regularization

To alleviate sensitivity to rigid $\ell_2$ clipping, a differentiable soft-clipping layer expands small-norm features and contracts large-norm features smoothly. An EMA teacher head with hard features guides the fine-tuning of a student head on soft-clipped features via KL-divergence, breaking the degeneracy where coupled weight/norm rescaling would nullify the effect of clipping.

Figure 4: Pre-clipping feature norm distribution shows DCR concentrates norms near threshold $S_j$0, reducing information loss and mitigating drift.

Theoretical Guarantees

The authors provide formal privacy proofs based on sequential budget composition, showing that (1) the Laplace top-$S_j$1 selection achieves $S_j$2-LDP, and (2) the groupwise prototype release—using dimension-dependent noise and clipping—achieves $S_j$3-LDP provided a harmonic bound on the inverses of the groupwise variances, maintaining equivalence to an isotropic reference mechanism at the same total budget. DCR is a local-only operation and does not consume any privacy budget.

Figure 5: Privacy allocation and calibration: VPDR splits the client-side budget for private subspace selection and prototype release, calibrating groupwise variances to match overall LDP.

Empirical Evaluation

Privacy-Utility Tradeoffs

Experimental results across Digits, Office-Caltech, and PACS benchmarks show that VPDR consistently improves average accuracy and reduces performance variance relative to IGPP under matched $S_j$4. The numerical results indicate gains up to several percentage points in challenging, high-heterogeneity regimes (see main paper Table 1). On task/attack dual evaluation, the defense strength of VPDR matches that of IGPP: both drive MIA attack ROC-AUC to near 0.5 and severely limit prototype-based reconstruction.

The method scales robustly under label skew and joint model-data heterogeneity, and is effective for both vision (ViT backbones) and natural language tasks (RoBERTa on AG News).

Ablations and Hyperparameter Sensitivity

Ablations confirm both VPP and DCR contribute additively to accuracy, with neither module alone matching the full system. Hyperparameter grids reveal broad insensitivity to the partition ratio ($S_j$5) and discriminative subspace fraction ($S_j$6), validating that VPDR is not brittle.

Figure 6: Performance as a function of discriminative subspace fraction $S_j$7—optimal utility when $S_j$8 is within a moderate range, as predicted by the theoretical allocation bound.

Qualitative Visualization

t-SNE plots of the embedding geometry demonstrate that VPDR leads to tighter, more discriminative clusters, and less class overlap after multiple rounds compared to isotropic baselines.

Figure 7: T-SNE of features for FedProto with VPDR—late rounds show improved cluster separation and compactness.

Overhead

The added computational cost is modest (<10% per round) and communication costs remain unchanged, as only privatized prototypes are transmitted.

Implications and Future Directions

This work raises the bar for privacy-preserving federated adaptation in prototype-sharing settings. By reconciling task-geometry-aware noise allocation with formal LDP accounting, its modular plug-in design enables upgrade of existing ProtoPFL frameworks for both i.i.d. and non-i.i.d. scenarios. Its methods are especially impactful in applications where domain or label skew is extreme and where prototype leakage is a credible privacy risk.

Theoretically, the paper suggests fruitful directions in designing data-dependent LDP mechanisms that exploit local geometry while enforcing provable protections. In terms of systems, the approach is directly extensible to more complex adaptation modules or domain-specific architectures, serving as a model for tightly integrating privacy controls with representation learning.

Conclusion

The paper makes a significant contribution by demonstrating that privacy-preserving prototype communication need not be at odds with discriminative capacity in federated fine-tuning. Through variance-adaptive perturbation and norm-concentrating distillation regularization, VPDR achieves strong privacy, high utility, and practical deployability, backed by both theoretical analysis and empirical validation. The approach is readily adaptable to a range of federated and domain-adaptive settings involving class statistics sharing.