SplitFed: When Federated Learning Meets Split Learning (2004.12088v5)

Abstract: Federated learning (FL) and split learning (SL) are two popular distributed machine learning approaches. Both follow a model-to-data scenario; clients train and test machine learning models without sharing raw data. SL provides better model privacy than FL due to the machine learning model architecture split between clients and the server. Moreover, the split model makes SL a better option for resource-constrained environments. However, SL performs slower than FL due to the relay-based training across multiple clients. In this regard, this paper presents a novel approach, named splitfed learning (SFL), that amalgamates the two approaches eliminating their inherent drawbacks, along with a refined architectural configuration incorporating differential privacy and PixelDP to enhance data privacy and model robustness. Our analysis and empirical results demonstrate that (pure) SFL provides similar test accuracy and communication efficiency as SL while significantly decreasing its computation time per global epoch than in SL for multiple clients. Furthermore, as in SL, its communication efficiency over FL improves with the number of clients. Besides, the performance of SFL with privacy and robustness measures is further evaluated under extended experimental settings.

• The paper introduces SplitFed Learning, a novel framework that merges federated and split learning to enhance model privacy and computational efficiency.
• It employs a dual-server architecture with differential privacy and PixelDP to secure client data while supporting parallel processing of model updates.
• Experiments on diverse datasets show that SplitFed reduces training time and communication costs while maintaining centralized-level model accuracy.

Analysis of SplitFed: Integration of Federated Learning and Split Learning

The paper "SplitFed: When Federated Learning Meets Split Learning" introduces a novel machine learning framework called SplitFed Learning (SFL), which integrates Federated Learning (FL) and Split Learning (SL) to address their respective limitations. The research focuses on enhancing model privacy, communication efficiency, and computation time in distributed collaborative machine learning (DCML) environments. The SFL approach is promising for resource-constrained applications, offering insights into balancing efficiency and security in distributed machine learning.

Key Contributions and Background

FL and SL are popular for enabling model training across distributed data sources without exposing raw data. FL supports parallel client training and aggregation at a central server but is resource-intensive as it requires the full model to be trained locally. SL, conversely, distributes partial models between clients and the server, improving privacy and reducing client resource requirements but suffering from increased training overhead due to sequential client-server interactions.

The primary contribution of this paper is the proposal of SplitFed Learning. SFL capitalizes on the parallel training efficiency of FL while preserving the model-splitting privacy advantages of SL. Additionally, SFL incorporates differential privacy (DP) and PixelDP mechanisms to further enhance data confidentiality and robustness against adversarial attacks.

Technical Details

SplitFed Learning operates using a dual-server architecture comprising a main server and a fed server. The main server handles partial model computation and communication with clients, while the fed server executes Federated Averaging (FedAvg) at the client-side for model synchronization:

• Parallel Client Processing: Unlike SL, SFL enables parallel client-side model updates, reducing idle resource time.
• Enhanced Privacy Measures: Differential privacy ensures protected gradient updates on the client side, and PixelDP adds noise to activation data, safeguarding against inversion attacks.
• Variants: SFLV1 and SFLV2 are two implemented variants. SFLV1 includes FedAvg at both client and server sides, while SFLV2 omits model aggregation at the server side for potentially increased accuracy.

Experimental Evaluation

Empirical analyses are conducted using diverse datasets (MNIST, FMNIST, CIFAR10, and HAM10000) and models (LeNet, AlexNet, VGG16, ResNet18). The experiments validate the communication efficiency and reduced training time of SFL compared to classical SL, while maintaining model accuracy comparable to centralized learning:

• Accuracy: SFL achieves results similar to SL in model accuracy, with improvements noted in specific scenarios due to parallel processing.
• Efficiency: SFL significantly reduces training time compared to SL, owing to elimination of sequential client resources and parallel main server operations.
• Communication Costs: SFL demonstrates effective communication handling, especially with large client numbers, supporting scalability in distributed systems.

Implications and Future Directions

SplitFed Learning constitutes a robust contribution to the domain of privacy-preserving machine learning, with implications for IoT, healthcare, and finance sectors. By addressing computation constraints and ensuring privacy, SFL paves the way for broader adoption of DCML techniques in sensitive data contexts.

Future work includes refining privacy-utility trade-offs, enhancing encryption methods like homomorphic encryption for further security, and exploring non-IID data distributions. Additionally, applying SFL to non-convolutional models could widen its applicability.

Conclusion

The integration of FL and SL in the form of SplitFed Learning presents a balanced trade-off between performance efficiency and data privacy. This work exemplifies the potential advancements in distributed learning frameworks, promoting enhanced model security without compromising computational feasibility.