Algorithms for Collaborative Machine Learning under Statistical Heterogeneity (2408.00050v1)

Abstract: Learning from distributed data without accessing them is undoubtedly a challenging and non-trivial task. Nevertheless, the necessity for distributed training of a statistical model has been increasing, due to the privacy concerns of local data owners and the cost in centralizing the massively distributed data. Federated learning (FL) is currently the de facto standard of training a machine learning model across heterogeneous data owners, without leaving the raw data out of local silos. Nevertheless, several challenges must be addressed in order for FL to be more practical in reality. Among these challenges, the statistical heterogeneity problem is the most significant and requires immediate attention. From the main objective of FL, three major factors can be considered as starting points -- \textit{parameter}, textit{mixing coefficient}, and \textit{local data distributions}. In alignment with the components, this dissertation is organized into three parts. In Chapter II, a novel personalization method, \texttt{SuPerFed}, inspired by the mode-connectivity is introduced. In Chapter III, an adaptive decision-making algorithm, \texttt{AAggFF}, is introduced for inducing uniform performance distributions in participating clients, which is realized by online convex optimization framework. Finally, in Chapter IV, a collaborative synthetic data generation method, \texttt{FedEvg}, is introduced, leveraging the flexibility and compositionality of an energy-based modeling approach. Taken together, all of these approaches provide practical solutions to mitigate the statistical heterogeneity problem in data-decentralized settings, paving the way for distributed systems and applications using collaborative machine learning methods.

• The paper introduces SuPerFed, AAggFF, and FedEvg to tackle statistical heterogeneity by enhancing model personalization, adaptive aggregation, and synthetic data generation.
• It employs orthogonality regularization, online convex optimization, and energy-based models for improved calibration, client fairness, and efficient communication.
• Empirical results demonstrate enhanced accuracy, sublinear regret bounds, and superior synthetic data quality in non-IID federated learning environments.

Overview of "Algorithms for Collaborative Machine Learning under Statistical Heterogeneity"

Introduction

The paper "Algorithms for Collaborative Machine Learning under Statistical Heterogeneity" by Seok-Ju Hahn focuses on perspectives for improving performance in federated learning (FL) under the constraint of data heterogeneity. The primary objective of FL is to enable collaborative training of a machine learning model across multiple clients without sharing raw data, thus preserving privacy. Despite its advantages, FL encounters significant challenges due to the inherent statistical heterogeneity across clients. This paper investigates three perspectives—model parameters, mixing coefficients, and local data distributions—for addressing statistical heterogeneity.

Parameter Perspective: SuPerFed

Chapter~\ref{ch:superfed} introduces SuPerFed, which aims to mitigate statistical heterogeneity through model mixture-based personalization. By leveraging mode connectivity, SuPerFed establishes an explicit synergy between global and local models to enhance personalization performance while maintaining good model calibration and robustness to label noise.

Key Contributions:

• SuPerFed employs orthogonality regularization to diversify the knowledge captured by local and federated models.
• The method yields notable improvements in personalization performance across various datasets and non-IID settings.
• One significant aspect of SuPerFed is its robustness to label noise and enhanced calibration performance.

Results:

SuPerFed demonstrates superior accuracy in various statistical heterogeneity scenarios and ensures consistent performance regardless of the degree of heterogeneity.

Mixing Coefficient Perspective: AAggFF

Chapter~\ref{ch:aaggff} proposes AAggFF, an adaptive aggregation framework for FL designed to achieve client-level fairness by updating mixing coefficients dynamically. This framework unifies existing fair FL strategies under an online convex optimization (OCO) framework, addressing the problem of sample deficiency in the central server's decision-making process.

Key Contributions:

• AAggFF-S and AAggFF-D are tailored for cross-silo and cross-device FL settings, respectively.
• AAggFF-S uses the Online Newton Step algorithm, achieving optimal regret bounds with logarithmic dependence on the number of rounds.
• AAggFF-D employs a linear-runtime FTRL algorithm, which is computationally efficient for large-scale FL settings.
• The theoretical analysis guarantees sublinear regret bounds.

Results:

AAggFF consistently improves worst-case client performance and reduces performance disparity across clients, thus enhancing client-level fairness.

Local Data Distribution Perspective: FedEvg

Chapter~\ref{ch:fedevg} presents FedEvg, a method for federated synthetic data generation that leverages energy-based models (EBMs). FedEvg synthesizes data by aggregating signals from clients and refining synthetic data iteratively, improving training efficiency and reducing communication overhead.

Key Contributions:

• FedEvg initializes synthetic data on the server and refines it with client signals based on EBMs.
• The method avoids the need for explicit model parameter exchange, lowering communication costs.
• By utilizing server-side SGLD steps, FedEvg ensures that synthetic data approximates the underlying local data distributions.

Results:

FedEvg produces high-quality synthetic data that serve as a proxy for local data distributions, evidenced by improved FID scores and discriminative performance of classifiers trained on the synthetic data.

Discussion

The paper highlights several promising directions for future work:

• Extending SuPerFed to cross-device settings and combining online convex optimization with stochastic optimization.
• Enhancing the stability of EBM training in FedEvg with advanced techniques like MCMC teaching and score-based modeling.
• Empirical evaluation of the proposed methods on text and tabular data, alongside explicit privacy-preserving mechanisms such as differential privacy.

Conclusion

The dissertation presents innovative approaches to address statistical heterogeneity in FL from three different angles. By improving model personalization, adaptive aggregation, and synthetic data generation, it paves the way for more practical and scalable FL systems. These contributions are expected to significantly enhance the effectiveness of collaborative machine learning across data-decentralized environments.