Federated Prompt Tuning for Multimodal Models

• Federated Prompt Tuning is a distributed adaptation strategy that fine-tunes lightweight prompt tokens while keeping pre-trained backbones frozen to preserve privacy.
• It employs dual prompt pools—inter and intra—to address data heterogeneity, missing modalities, and client-specific conditional distributions.
• Robust clustering and regularization techniques enable scalable semantic alignment and significant performance gains in multimodal, edge, and privacy-sensitive deployments.

Federated prompt tuning is an advanced parameter-efficient adaptation strategy that enables distributed clients to collaboratively fine-tune prompt instructions of large (often multimodal) pre-trained models without sharing raw data. This paradigm extends classical federated learning (FL) to the tuning of lightweight prompt tokens, leveraging freezing of backbone parameters, and introduces principled mechanisms to address multimodal, heterogeneous, and incomplete data scenarios common in practical edge or privacy-sensitive deployments. Recent frameworks have focused on overcoming challenges posed by local multimodal data with arbitrary missingness, client-specific conditional distributions, and data-model mismatch, yielding robust and generalizable solutions for real-world FL (Phung et al., 6 Feb 2026).

1. Federated Multimodal Prompt Tuning: System Architecture

Generalized federated prompt-tuning for heterogeneous and incomplete multimodal client data establishes a client-server (centralized) FL loop built upon a frozen pre-trained multimodal transformer (for example, ViLT). Each client locally maintains:

• A small, private classification head $w_c$
• Two prompt pools: $w_p^{\text{inter}}$ (inter-client prompts, cardinality $\tau$) and $w_p^{\text{intra}}$ (intra-client prompts, cardinality $\tau$), designed to separately capture input-level missing-data patterns and modality-agnostic local features.

At each communication round $t$:

1. The server broadcasts the current global sets $(w_g^{\text{inter}}, w_g^{\text{intra}})$ to all clients.
2. Clients perform local fine-tuning of prompt parameters and the classification head using their private, possibly incomplete multimodal data, optimizing only these parameters while keeping backbones $F_p \circ F_e$ frozen.
3. Locally updated $(w_p^{\text{inter}}, w_p^{\text{intra}})$ are returned to the server.
4. The server aggregates:
   • Intra-prompts using straightforward FedAvg,
   • Inter-prompts through a specialized clustering alignment that merges semantically similar prompts with respect to missing data patterns, yielding the updated global prompt pool (Phung et al., 6 Feb 2026).

This dual-pool design allows the system to simultaneously learn modality-agnostic and missing-modality-conditional representations, addressing both inter- and intra-client distribution heterogeneity.

2. Client-Side Optimization: Prompt Selection and Regularization

On each client, prompt tuning is conducted by minimizing a composite loss over private multimodal samples $\{(x(M_{t,s}), z_{t,s})\}_{s=1}^m$, where $M_{t,s} \subseteq \{1,\ldots,r\}$ indexes observed modalities for each instance. The prompt-tuned model is formulated as: $F(x(M); w) = F_c(F_p(F_e(x(M)) \circ w_p); w_c)$ with $w_p$ the selected prompts, concatenated to the token sequence. The loss optimized is: $$L'_t(w') = \sum_{s=1}^m \ell(F(x(M_{t,s}); w'), z_{t,s}) + \sum_{s=1}^m r(x(M_{t,s}), w'_p)$$ where $\ell$ is cross-entropy and $r$ is a regularizer enforcing alignment between input features and prompt selection.

Prompt selection operates by embedding both inputs and prompts into a shared metric space:

• Query $q(x(M))$ for the input, key $k(p)$ for each prompt.
• Distance is $d(x(M), p) = -\cos(q(x(M)), k(p))$.
• For each sample, the $\kappa$ nearest intra- and inter-prompts are retrieved and regularization encourages semantic fit, penalizing large input-prompt distances: $$r(x(M), w'_p) = \sum_{p \in w_p^{\text{inter'} } \cup w_p^{\text{intra'}}} d(x(M), p)$$ This suppresses prompt overloading and ensures only semantically relevant prompts are utilized (Phung et al., 6 Feb 2026).

3. Server-Side Aggregation: Prompt Pool Alignment

• Intra-Prompts: Simple weighted averaging of all updated intra-prompts: $$w_g^{\text{intra}} = (1/n) \sum_{t=1}^n w_t^{\text{intra}}$$.
• Inter-Prompts: Cluster alignment scheme:
  • Each client submits $\tau$ inter-prompts.
  • Server performs discrete assignment $\alpha_t^{p,q} \in \{0,1\}$ of local prompts $p_t^p$ to cluster centers $\theta_q$ (global prompt candidates).
  • Optimization over $(\alpha, \theta, \gamma, \zeta)$ minimizes joint cost $G(\alpha, \theta, \gamma)$ and popularity penalty $R(\alpha, \zeta)$, with alternating updates:
  • 1. Fix $\alpha$, update cluster centers and cost/model parameters by gradient descent.
  • 2. Fix cluster centers, solve optimal assignments using a Hungarian matching algorithm incorporating a log-popularity term.
  • Unassigned clusters are pruned; surviving $\theta_q$ become the new global inter-prompts (Phung et al., 6 Feb 2026).

This clustering enforces semantic alignment by forcing prompts corresponding to similar missing-data patterns to merge across clients, while popularity weighting balances specificity and generality of global prompt instructions.

4. Semantic Alignment, Generalization, and Robustness

The clustering and regularization together provide strong semantic alignment mechanisms:

• Inter-client clustering aligns prompts encoding similar missing-data distributions, fostering transferability and coverage of diverse multimodal input structures.
• The popularity regularizer $R$ promotes retention of "useful" prompts frequently selected by clients, thus balancing the bias–variance tradeoff between highly specialized and general instructions.
• The input-prompt contrastive regularizer ensures that distributed prompt pools do not collapse or become overloaded, maintaining specialization and avoiding prompt drift.

Extensive empirical evaluation demonstrates that, under high rates of missing modalities and across diverse multimodal benchmarks (UPMC Food-101, MM-IMDB), the described federated prompt tuning outperforms state-of-the-art baselines (up to $+107.8\%$ relative improvement), remains robust even at extreme missingness ($\eta \to 1$), and closely matches centralized performance (Phung et al., 6 Feb 2026).

5. Algorithmic Workflow and Implementation

The complete federated multimodal prompt-tuning algorithm is structured as follows:

for round in range(T): broadcast(global_inter_prompts, global_intra_prompts) for each client t in parallel: initialize(local_inter_prompts, local_intra_prompts) # Client-side local update on private data optimize(local_class_head, local_inter_prompts, local_intra_prompts) upload(local_inter_prompts, local_intra_prompts) # Server-side aggregation global_intra_prompts = FedAvg({local_intra_prompts}) global_inter_prompts = ClusterAlign({local_inter_prompts})

The ClusterAlign subroutine alternates between cluster-center update (gradient descent) and prompt assignment (Hungarian optimization), pruning unassigned clusters after convergence (Phung et al., 6 Feb 2026).

6. Challenges, Limitations, and Future Directions

The dual-prompt pool framework directly addresses:

• Inter-client and intra-client heterogeneity in observed modalities and missingness.
• The failure of naive prompt averaging, which leads to loss of specialization.
• The need for scalable unsupervised semantic alignment of highly diverse prompt instructions.

Experimentally, it supports strong performance under both modality-incomplete and non-IID settings. However, current implementations rely on discrete clustering and Hungarian assignment for global prompt alignment, which may face scalability bottlenecks for large numbers of clients and prompt slots. Future work may explore continuous relaxations, asynchronous updates, dynamic client arrival/departure, and extension to additional modalities (e.g., audio, video) and broader classes of foundation models (Phung et al., 6 Feb 2026).

---

References:

• "Federated Prompt-Tuning with Heterogeneous and Incomplete Multimodal Client Data" (Phung et al., 6 Feb 2026)