HtFLlib: A Comprehensive Heterogeneous Federated Learning Library and Benchmark

Abstract: As AI evolves, collaboration among heterogeneous models helps overcome data scarcity by enabling knowledge transfer across institutions and devices. Traditional Federated Learning (FL) only supports homogeneous models, limiting collaboration among clients with heterogeneous model architectures. To address this, Heterogeneous Federated Learning (HtFL) methods are developed to enable collaboration across diverse heterogeneous models while tackling the data heterogeneity issue at the same time. However, a comprehensive benchmark for standardized evaluation and analysis of the rapidly growing HtFL methods is lacking. Firstly, the highly varied datasets, model heterogeneity scenarios, and different method implementations become hurdles to making easy and fair comparisons among HtFL methods. Secondly, the effectiveness and robustness of HtFL methods are under-explored in various scenarios, such as the medical domain and sensor signal modality. To fill this gap, we introduce the first Heterogeneous Federated Learning Library (HtFLlib), an easy-to-use and extensible framework that integrates multiple datasets and model heterogeneity scenarios, offering a robust benchmark for research and practical applications. Specifically, HtFLlib integrates (1) 12 datasets spanning various domains, modalities, and data heterogeneity scenarios; (2) 40 model architectures, ranging from small to large, across three modalities; (3) a modularized and easy-to-extend HtFL codebase with implementations of 10 representative HtFL methods; and (4) systematic evaluations in terms of accuracy, convergence, computation costs, and communication costs. We emphasize the advantages and potential of state-of-the-art HtFL methods and hope that HtFLlib will catalyze advancing HtFL research and enable its broader applications. The code is released at https://github.com/TsingZ0/HtFLlib.

• The paper introduces HtFLlib, a standardized framework for evaluating heterogeneous federated learning across varied datasets and model architectures.
• The paper demonstrates that increasing model and data heterogeneity substantially affects performance and convergence, with methods like FedMRL showing resilience.
• The paper highlights trade-offs between communication, computation, and accuracy, providing actionable insights for advancing HtFL research and practical applications.

HtFLlib: A Comprehensive Heterogeneous Federated Learning Library and Benchmark

This paper presents HtFLlib, a comprehensive framework and benchmark specifically designed for Heterogeneous Federated Learning (HtFL). It addresses the challenges associated with collaborations among clients possessing heterogeneous model architectures, a scenario where traditional Federated Learning (FL) falls short due to its limitation of supporting only homogeneous model architectures.

Core Contributions

The authors introduce the first standardized benchmarking framework, HtFLlib, which facilitates the evaluation and comparison of emerging HtFL methods. The framework integrates a variety of datasets, model architectures, and methodological implementations to overcome current obstacles in the field. Key components of HtFLlib include:

1. Dataset and Model Variety: HtFLlib encompasses 12 datasets across different domains and modalities, and 40 diversified model architectures accommodating both small and large models.
2. Codebase and Implementations: It offers a modular and extensible codebase with implementations of ten representative HtFL methods, allowing researchers to conduct systematic evaluations in terms of accuracy, convergence, computation, and communication costs.
3. Evaluation Metrics: The framework provides a robust mechanism for evaluating HtFL methods based on a variety of criteria, including performance on tasks such as image recognition, text classification, and sensor signal processing.

Key Findings

The paper reports several important findings from the evaluation using HtFLlib:

• Model Heterogeneity Impact: As model heterogeneity increases, performance generally declines. However, methods like FedMRL show relative robustness across model variations.
• Data Heterogeneity: The degree of data heterogeneity significantly affects convergence rates, with FedGen, FedTGP, and FedKTL displaying more stable convergence across varied client data distributions.
• Communication and Computation Costs: Methods relying on mutual distillation incur significant communication overhead due to parameter transmission, while prototype-sharing methods show minimized communication costs.

Implications for HtFL Research

The introduction of HtFLlib has notable implications for both theoretical research and practical applications:

• Theoretical Development: HtFLlib enables researchers to more accurately simulate real-world federated learning scenarios involving heterogeneous models and data distributions, accelerating theoretical advancements.
• Practical Applications: By supporting heterogeneous model collaboration, HtFLlib opens possibilities for broader deployments of federated learning in sectors such as healthcare, finance, and IoT, where bespoke models are common and data sharing is a concern.
• Future Directions: The authors highlight potential areas for future research, like integrating solutions with pre-trained large models to further enhance the capabilities and deployment of HtFL in large-scale settings.

Overall, HtFLlib stands as a significant asset in federated learning research, providing a foundation for evaluating and developing robust solutions to the challenges of data and model heterogeneity. The release of HtFLlib not only catalyzes advancing HtFL research but also underscores the importance of collaboration and adaptability in privacy-preserving AI applications.