FedMultimodal: A Benchmark For Multimodal Federated Learning (2306.09486v2)

Abstract: Over the past few years, Federated Learning (FL) has become an emerging machine learning technique to tackle data privacy challenges through collaborative training. In the Federated Learning algorithm, the clients submit a locally trained model, and the server aggregates these parameters until convergence. Despite significant efforts that have been made to FL in fields like computer vision, audio, and natural language processing, the FL applications utilizing multimodal data streams remain largely unexplored. It is known that multimodal learning has broad real-world applications in emotion recognition, healthcare, multimedia, and social media, while user privacy persists as a critical concern. Specifically, there are no existing FL benchmarks targeting multimodal applications or related tasks. In order to facilitate the research in multimodal FL, we introduce FedMultimodal, the first FL benchmark for multimodal learning covering five representative multimodal applications from ten commonly used datasets with a total of eight unique modalities. FedMultimodal offers a systematic FL pipeline, enabling end-to-end modeling framework ranging from data partition and feature extraction to FL benchmark algorithms and model evaluation. Unlike existing FL benchmarks, FedMultimodal provides a standardized approach to assess the robustness of FL against three common data corruptions in real-life multimodal applications: missing modalities, missing labels, and erroneous labels. We hope that FedMultimodal can accelerate numerous future research directions, including designing multimodal FL algorithms toward extreme data heterogeneity, robustness multimodal FL, and efficient multimodal FL. The datasets and benchmark results can be accessed at: https://github.com/usc-sail/fed-multimodal.

Overview of the FedMultimodal Benchmark for Multimodal Federated Learning

In this paper, the authors present FedMultimodal, the first Federated Learning (FL) benchmark dedicated to multimodal applications. Over the last few years, FL has emerged as an essential machine learning paradigm primarily designed to address data privacy concerns by facilitating collaborative model training directly on user devices. Despite its success in unimodal domains like computer vision, audio, and natural language processing, FL's application in multimodal learning remains insufficiently explored. Multimodal learning, which involves processing and learning from multiple diverse data streams, is significant for numerous real-world applications such as emotion recognition, healthcare, and social media analytics, where privacy and robustness are of utmost importance.

FedMultimodal Key Contributions

The paper introduces several key contributions through FedMultimodal:

1. Diverse Multimodal Datasets: FedMultimodal encompasses ten publicly accessible datasets representing five distinct application scenarios: emotion recognition, multimodal action recognition, human activity recognition, healthcare, and social media classification. These datasets include various modalities such as audio, video, accelerometer, gyroscope, electrocardiogram, and textual data.
2. Comprehensive Simulation Framework: FedMultimodal offers a complete end-to-end pipeline for FL research, including non-IID data partitioning, feature extraction using mobile-friendly models, multimodal model training, fusion strategies such as concatenation-based and attention-based fusion, and evaluation against multiple FL optimizers.
3. Robustness Assessment: The benchmark provides robustness evaluation to three prevalent data corruptions encountered in real-world settings: missing modalities, missing labels, and erroneous labels.

Benchmark Results

The authors present benchmark results across various FL algorithms, including FedAvg, FedProx, FedRS, and FedOpt, employing both concatenation-based and attention-based fusion strategies. Notably, attention-based fusion generally yields superior performance compared to concatenation techniques, especially in high-data heterogeneity scenarios. Among FL algorithms, FedOpt demonstrates the best results overall, although it requires additional hyperparameter tuning.

Impact of Missing Data and Labels

The paper highlights the impact of real-world noise factors, including missing modalities, missing labels, and erroneous labels. It is noteworthy that multimodal FL models exhibit resilience to a small percentage of missing modalities, but their performance declines substantially as corruption rates increase. Erroneous labels pose a greater challenge to model robustness compared to missing modalities or labels, underscoring the need for advanced noise handling mechanisms in multimodal FL.

Implications for Future Research

The introduction of FedMultimodal is pivotal for advancing multimodal FL research. While the current benchmark successfully addresses several fundamental issues, it opens up further research directions:

• Development of sophisticated modality fusion strategies tailored for FL.
• Investigation into robust learning techniques that effectively mitigate real-world data noise.
• Exploration of privacy-enhancing methods and security mechanisms in multimodal FL environments.
• Examination of scalability and performance optimization for large-scale multimodal datasets and models.

In conclusion, FedMultimodal serves as a critical resource for the FL community, encouraging innovation in multimodal learning while maintaining user privacy and model robustness. This benchmark lays the foundation for improving multimodal interactions in intelligent systems, facilitating future advancements in AI-driven applications.