FedHPro: Federated Hyper-Prototype Learning via Gradient Matching

Abstract: Federated Learning (FL) enables collaborative training of distributed clients while protecting privacy. To enhance generalization capability in FL, prototype-based FL is in the spotlight, since shared global prototypes offer semantic anchors for aligning client-specific local prototypes. However, existing methods update global prototypes at the prototype-level via averaging local prototypes or refining global anchors, which often leads to semantic drift across clients and subsequently yields a misaligned global signal. To alleviate this issue, we introduce hyper-prototypes, defined by a set of learnable global class-wise prototypes to preserve underlying semantic knowledge across clients. The hyper-prototypes are optimized via gradient matching to align with class-relevant characteristics distilled directly from clients' real samples, rather than prototype-level descriptors. We further propose FedHPro, a Federated Hyper-Prototype Learning framework, to leverage hyper-prototypes to promote inter-class separability via mutual-contrastive learning with client-specific margin, while encouraging intra-class uniformity through a consistency penalty. Comprehensive experiments under diverse heterogeneous scenarios confirm that 1) hyper-prototypes produce a more semantically consistent global signal, and 2) FedHPro achieves state-of-the-art performance on several benchmark datasets. Code is available at \href{https://github.com/mala-lab/FedHPro}{https://github.com/mala-lab/FedHPro}.

• The paper introduces gradient-matched hyper-prototypes as robust semantic anchors that directly align with aggregated client gradients to overcome semantic drift in non-IID settings.
• It leverages two modules, HPCL and HPAL, to enhance inter-class separability and intra-class uniformity, resulting in superior accuracy on domain-skewed benchmarks.
• Empirical evaluations demonstrate state-of-the-art performance and faster convergence compared to baselines, validating the practical effectiveness of the FedHPro framework.

Authoritative Summary of "FedHPro: Federated Hyper-Prototype Learning via Gradient Matching" (2605.13475)

Motivation and Problem Formulation

The paper addresses the challenge of semantic misalignment in prototype-based Federated Learning (FL). Prototype-based FL methods (e.g., FedProto, FedSA) typically aggregate local class-wise prototypes from heterogeneous clients to form global prototypes used as alignment signals. However, this prototype-level averaging or refinement inherits client-induced representation bias, resulting in semantic drift and suboptimal global signals—particularly under non-IID or domain-skewed distribution scenarios. The authors present compelling visual evidence of this misalignment and its impact on feature distributions and inter-class separation.

Figure 1: Comparison between conventional FedProto aggregation and the proposed method in heterogeneous FL, illustrating improved intra-class compactness and inter-class separation.

Hyper-Prototype Construction via Gradient Matching

The paper proposes the concept of hyper-prototypes, a set of learnable, global, class-wise prototypes maintained at the server side. Rather than aggregating prototype descriptors from local clients, hyper-prototypes are optimized to simulate the gradients of all client data samples directly, effectively distilling class-relevant semantic characteristics. The optimization leverages gradient matching—the hyper-prototypes are updated to match the aggregated average gradients computed from real client samples. This approach enables semantic anchors that are robust to heterogeneous data and domain drift, closely approximating centralized prototypes.

Strong empirical evidence is provided: hyper-prototypes achieve a lower $L_2$ distance to centralized prototypes than FedAvg and FedProto, which correlates with improved accuracy on domain-skewed benchmarks (Digits dataset).

Figure 2: Quantitative comparison of prototype quality and classification accuracy; hyper-prototypes exhibit a significant reduction in semantic drift.

Figure 3: t-SNE visualization demonstrating superior inter-domain and intra-class consistency for hyper-prototypes compared to conventional global prototypes.

FedHPro Framework and Objective

Building on the hyper-prototype mechanism, the authors introduce the FedHPro framework, integrating two main modules:

• Hyper-Prototype Contrastive Learning (HPCL): Promotes inter-class separability by contrastively aligning sample embeddings with their respective hyper-prototypes and repelling them from others, utilizing client-specific adaptive margins.
• Hyper-Prototype Alignment Learning (HPAL): Enforces intra-class uniformity across clients by penalizing deviations between local embeddings and averaged hyper-prototypes, utilizing a smooth feature-level penalty.

This combined objective drives local models to optimize both decision boundary sharpness (via HPCL) and semantic consistency (via HPAL), effectively regularizing FL training against representation drift.

Figure 4: FedHPro architecture showing client gradient upload and hyper-prototype optimization with integrated HPCL and HPAL modules.

Empirical Results and Benchmarks

FedHPro is extensively evaluated across multiple FL scenarios—label skew, quantity skew, and domain skew—using natural and artificial datasets (e.g., CIFAR10, HAM10000, TinyImageNet, Digits, Office-Caltech). The authors report state-of-the-art accuracy across all scenarios, outperforming baselines including FedAvg, FedProx, MOON, FedProto, FedTGP, FedGMKD, FedRCL, and FedSA.

• On Digits (domain skew): FedHPro achieves an $84.80\%$ average accuracy, surpassing the closest baseline (FedSA: $82.54\%$).
• On Office-Caltech: FedHPro improves average accuracy by $9.10$ percentage points over FedAvg.
• On long-tailed scenarios (CIFAR10-LT, $\rho=100$): FedHPro maintains robustness, outperforming baselines by $2.27\%$ or more.

FedHPro also demonstrates faster convergence, requiring fewer communication rounds to reach high accuracy relative to baselines.

Figure 5: t-SNE showing superior inter-class separation and intra-class tightness for FedHPro on Digits and Office-Caltech.

Figure 6: Convergence comparison illustrating accelerated learning behavior and superior final accuracy for FedHPro.

Ablations and Module Analysis

The paper provides rigorous ablation studies validating the separate and combined contributions of HPCL and HPAL. Both modules individually yield significant performance gains; their combination produces optimal accuracy on all benchmarks. Hyper-parameter sweeps confirm FedHPro's stability and robustness to changes in temperature $\tau$, margin $d_k$, number of hyper-prototypes, and communication rounds. The superiority of hyper-prototypes over conventional global prototype aggregation is evidenced by their plug-and-play enhancement to other prototype-based methods.

Figure 7: Impact analysis of $\tau$ and margin $d_k$ hyper-parameters on accuracy across different heterogeneity scenarios.

Figure 8: Performance scaling with hyper-prototype set size and gradient-matching rounds; moderate values yield efficiency and stability.

Efficiency and Practicality

FedHPro maintains competitive computational and communication efficiency. Per-round training and communication overheads are comparable to FedSA, with only minimal additional gradient transmission costs. The method scales efficiently with increasing client or class count and remains robust under extreme non-IID, large-scale, or model heterogeneous settings.

Theoretical Analysis

The paper includes a theoretical convergence analysis for FedHPro under non-convex objectives, adapting standard FL assumptions regarding smoothness, gradient boundedness, and Lipschitz continuity. Results show the boundedness and monotonic decrease of the objective function under the proposed framework, providing explicit conditions on learning rate and round count for guaranteed convergence.

Practical and Theoretical Implications

FedHPro significantly advances prototype-based FL by overcoming client-induced semantic drift, enabling generalizable, robust, and privacy-preserving collaborative learning in heterogeneous environments. The hyper-prototype construction via gradient alignment directly addresses the gap between decentralized and centralized learning, making FL more applicable to domains such as medical imaging, financial risk monitoring, and multi-modal cross-domain tasks. The plug-and-play nature of hyper-prototypes further suggests broad applicability and enhancement potential for existing FL paradigms.

Speculative Directions for Future Research

• Integration of hyper-prototypes with personalized FL and multi-modal foundation models.
• Adaptive hyper-prototype selection for communication-complexity reduction.
• Extending gradient-matching semantic anchoring to other modalities (text, speech, graph).
• Automated search for hyper-prototype structures under extreme domain shift or client drift.
• Alignment of hyper-prototype learning with foundation models and interpretable FL frameworks.

Conclusion

FedHPro introduces gradient-matched hyper-prototypes as robust semantic anchors in federated learning, systematically combatting data and model heterogeneity. By combining HPCL and HPAL, FedHPro achieves superior accuracy, stable convergence, and efficient training across diverse FL scenarios. Extensive empirical and theoretical analyses validate the practical effectiveness and theoretical soundness of the approach. Hyper-prototypes offer a promising direction for future developments in decentralized and privacy-preserving AI architectures.