Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification (1909.06335v1)

Abstract: Federated Learning enables visual models to be trained in a privacy-preserving way using real-world data from mobile devices. Given their distributed nature, the statistics of the data across these devices is likely to differ significantly. In this work, we look at the effect such non-identical data distributions has on visual classification via Federated Learning. We propose a way to synthesize datasets with a continuous range of identicalness and provide performance measures for the Federated Averaging algorithm. We show that performance degrades as distributions differ more, and propose a mitigation strategy via server momentum. Experiments on CIFAR-10 demonstrate improved classification performance over a range of non-identicalness, with classification accuracy improved from 30.1% to 76.9% in the most skewed settings.

• The paper shows that non-identical client data distributions drastically lower visual classification accuracy, particularly using CIFAR-10.
• It introduces a method using the Dirichlet distribution to simulate varied client data skewness within the FedAvg framework.
• Results reveal that server momentum (FedAvgM) can improve accuracy from 30.1% to 76.9%, highlighting the importance of precise hyperparameter tuning.

Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification

The paper "Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification" by Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown, explores the performance dynamics of Federated Learning (FL) under non-identical data distributions across clients. The paper is specifically oriented towards visual classification tasks and provides both empirical results and mitigation strategies for performance degradation in non-identical data scenarios.

Introduction

Federated Learning (FL) introduces a framework where models are trained using decentralized data from multiple devices, facilitating privacy-preserving machine learning. However, the inherent heterogeneity of data allocations among different clients raises significant challenges. This paper scrutinizes these challenges by analyzing the impact of varying degrees of non-identical data distributions on visual classification performance using the CIFAR-10 dataset.

Methodology

The authors introduce a method to create synthetic non-identical client data distributions through the Dirichlet distribution, parameterized to simulate a range of identicalness. They run extensive experiments using the Federated Averaging (FedAvg) algorithm under different hyperparameters and distribution skewness settings.

Experimental Setup

• Dataset: CIFAR-10 with 60,000 images (50,000 training, 10,000 testing).
• Clients: 100 clients, each holding 500 images, with data sampled per Dirichlet concentration $\alpha$.
• Evaluation: Performance was measured with varying $\alpha$, number of local epochs (E), and reporting fraction (C).

Results

The paper confirms that performance degrades as the degree of data distribution non-identicalness increases. Specific findings include:

• FedAvg Performance: Test accuracy drops significantly when clients possess highly non-identical data distributions (low $\alpha$). For instance, with highly skewed distributions, accuracy can plummet to about 30.1%.
• Mitigation via Server Momentum: Implementing momentum at the server side (FedAvgM) considerably improves performance across non-identical distributions. For significantly skewed data, the classification accuracy increased from 30.1% to 76.9%.

Hyperparameter Sensitivity

The sensitivity of hyperparameters like client learning rates and effective learning rates is particularly pronounced in scenarios with non-identical data distributions. The paper demonstrates that:

• High $\alpha$ (more identical distributions): A broad range of learning rates yields good performance.
• Low $\alpha$ (non-identical distributions): Requires precise learning rate adjustments to achieve optimal performance.

Practical Implications and Theoretical Considerations

• Federated Learning Robustness: Non-identical data distribution remains a critical factor affecting FL's robustness and efficacy. Techniques like server momentum can mitigate some adverse effects, but precise hyperparameter tuning is crucial.
• System Optimization: Increasing the reporting fraction (C) generally improves performance but shows diminishing returns beyond a certain point.
• Training Stability: Higher frequencies of synchronization (lower E) are not always advantageous under non-identical data distributions, as shown by more volatile training error trends.

Conclusions and Future Directions

This paper provides a detailed investigation into how varying degrees of data heterogeneity affect FL performance in visual classification tasks. The incorporation of server-side momentum demonstrates a tangible strategy for performance enhancement. Future research could explore adaptive methods for hyperparameter tuning in FL systems, explore more sophisticated aggregation techniques, and apply the findings to other domains beyond visual classification to assess generalizability.

Overall, this paper underscores the necessity of addressing data distribution disparities in FL to achieve consistent and reliable performance across diverse applications.