FedPIA -- Permuting and Integrating Adapters leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learning (2412.14424v1)

Abstract: Large Vision-LLMs typically require large text and image datasets for effective fine-tuning. However, collecting data from various sites, especially in healthcare, is challenging due to strict privacy regulations. An alternative is to fine-tune these models on end-user devices, such as in medical clinics, without sending data to a server. These local clients typically have limited computing power and small datasets, which are not enough for fully fine-tuning large VLMs on their own. A naive solution to these scenarios is to leverage parameter-efficient fine-tuning (PEFT) strategies and apply federated learning (FL) algorithms to combine the learned adapter weights, thereby respecting the resource limitations and data privacy. However, this approach does not fully leverage the knowledge from multiple adapters trained on diverse data distributions and for diverse tasks. The adapters are adversely impacted by data heterogeneity and task heterogeneity across clients resulting in suboptimal convergence. To this end, we propose a novel framework called FedPIA that improves upon the naive combinations of FL and PEFT by introducing Permutation and Integration of the local Adapters in the server and global Adapters in the clients exploiting Wasserstein barycenters for improved blending of client-specific and client-agnostic knowledge. This layerwise permutation helps to bridge the gap in the parameter space of local and global adapters before integration. We conduct over 2000 client-level experiments utilizing 48 medical image datasets across five different medical vision-language FL task settings encompassing visual question answering as well as image and report-based multi-label disease detection. Our experiments involving diverse client settings, ten different modalities, and two VLM backbones demonstrate that FedPIA consistently outperforms the state-of-the-art PEFT-FL baselines.

• The paper introduces FedPIA, which integrates adapter permutation with Wasserstein barycenters to overcome data heterogeneity in multimodal federated learning.
• It employs server-side alignment and client-side optimization to efficiently fine-tune large vision-language models while preserving data privacy.
• Extensive experiments across 2000+ client trials and 48 medical datasets demonstrate significant gains in accuracy and stability over existing PEFT-FL methods.

Overview of FedPIA: A Novel Approach for Multimodal Federated Learning

The paper introduces FedPIA, a novel framework designed to optimize the fine-tuning of Vision-LLMs (VLMs) in the context of multimodal federated learning (FL). This research addresses the significant challenge of adapting large VLMs, which typically contain millions or billions of parameters, for practical applications that demand privacy-preserving training across multiple decentralized clients, such as those in healthcare settings.

Research Context

The usage of VLMs has surged due to their ability to integrate and process multimodal data (e.g., images and text), achieving notable success in complex tasks like Visual Question Answering (VQA) and Visual Commonsense Reasoning. However, the efficiency of fine-tuning these models is hindered by the necessity to gather extensive training data, often complicated by privacy concerns, particularly in medical applications.

The constraints imposed by collecting sensitive data have pushed researchers to explore decentralized approaches like FL, whereby models are trained locally on end-user devices without sharing raw data. Despite its potential, this approach faces scalability issues in resource-limited devices, where the heterogeneity of data and tasks leads to sub-optimal learning outcomes.

The FedPIA Approach

FedPIA addresses these challenges by redefining the integration and fine-tuning strategies of model adapters through two key mechanisms: permutation and integration of adapters. Central to this is the use of Wasserstein Barycenters to effectively merge knowledge across different local client adapters while adapting to the unique distributions and constraints of each client.

• Server-side Integration: In FedPIA, adapter permutations occur server-side to align the diverse client-specific adapters into a cohesive global adapter. This alignment minimizes the loss of task-specific information that can arise from heterogenous data distributions across clients.
• Client-side Optimization: On the client side, FedPIA integrates global and local adapters, enhancing learning by aligning client-specific knowledge with shared global insights. This dual adaptation bridges the gap between diverse learning environments, facilitating stable convergence and improving performance consistency across tasks and modalities.

Experimental Evaluation

The authors conducted extensive experiments, covering over 2000 client-level trials with 48 medical image datasets across five diverse FL scenarios, including VQA and disease detection tasks. They compared FedPIA to leading PEFT-FL baselines using two backbone VLM architectures: ViLT and ALBEF. The results indicated that FedPIA consistently outperforms state-of-the-art baselines, showcasing significant improvements in accuracy and stability in heterogeneous FL settings.

Implications and Future Directions

FedPIA's robust performance underscores the viability of combining adaptation strategies with FL to maintain model efficacy under privacy and resource constraints. This framework provides valuable insights into designing scalable and efficient model fine-tuning methodologies that incorporate data privacy without compromising performance.

Future advances could explore extending FedPIA's methodology to a broader range of applications in domains where privacy and data heterogeneity are critical concerns. Additionally, leveraging FedPIA's approach in combination with other federated optimization techniques could yield interesting avenues for enhancing model generalization across more complex and nuanced data distributions.