---
title: Prompt-Based Federated Learning
url: https://www.emergentmind.com/topics/prompt-based-federated-learning
type: topic
---

# Prompt-Based Federated Learning

Prompt-Based Federated Learning (PBFL) is a paradigm that applies prompt learning—parameter-efficient adaptation of large pretrained models—within federated learning (FL) systems, enabling clients to collaboratively tailor frozen foundation models (typically vision, language, or vision-language models) by updating only small prompt parameters rather than full model weights. This approach is motivated by the need for computational, communication, and privacy efficiency, particularly in heterogeneous, resource-constrained, and privacy-sensitive settings. PBFL leverages the strong generalization of large models while allowing for data- and domain-adaptive personalization via prompts, and is rapidly establishing itself as a leading methodology for collaborative adaptation of foundation models.

## 1. Core Principles and System Architecture

At its core, PBFL freezes a pretrained foundation model (e.g., CLIP, RoBERTa, Transformer-based weather model) on all clients and trains only a small set of prompt parameters in a federated manner [2208.11625, 2405.09771, 2503.22263, 2411.10063, 2305.14244]. The standard workflow is:

- **Initialization:** The server maintains a global prompt (or prompt-generator parameters), which is broadcast to selected clients at each communication round.
- **Local Update:** Clients perform task-specific prompt updates via gradient descent using local data, with the backbone kept frozen to maximally exploit pretrained generalization and minimize local computation.
- **Aggregation:** Clients upload their updated prompt parameters (not model weights, activations, or gradients) to the server. The server aggregates (typically via FedAvg) to form a new global prompt.
- **Personalization (optional):** Many advanced variants (e.g., pFedMoAP [2410.10114], FedPGP [2405.09771], FedMGP [2511.00480], SDFed [2602.08590]) introduce local, low-rank, or mixture-of-expert prompt terms to allow per-client adaptation while preserving a global generalization backbone.

PBFL supports both text and visual prompts depending on the foundation model architecture [2505.23024, 2411.10063]. The prompt representation can be a learnable embedding (prepending context tokens for language models, or patch/border tokens for visual models), a specialized prompt-generator network [2310.06123, 2508.12399], or a combination thereof.

## 2. Mathematical Formalisms and Optimization Objectives

Across PBFL, the learning objective is to minimize a weighted average of local, prompt-parameterized losses:

\[
\min_{\theta \in \mathbb{R}^{m \times d}} F(\theta) = \sum_{i=1}^C \frac{n_i}{N} F_i(\theta),
\]
where \(F_i(\theta)\) is the per-client prompt-tuning loss (e.g., cross-entropy, contrastive, or task-specific), \(\theta\) represents one or more prompt matrices/vectors, and \(n_i\) is the local data size [2503.22263, 2208.11625].

- **Vision-Language Models:** For CLIP, the classification probability for class \(j\) given image \(x\) and prompt \(\theta\) is
  \[
  p(y=j \mid x; \theta) = \frac{\exp(\operatorname{sim}(E_\mathrm{image}(x), E_\mathrm{text}(t_j(\theta))) / \tau)}{\sum_{k=1}^K \exp(\operatorname{sim}(E_\mathrm{image}(x), E_\mathrm{text}(t_k(\theta))) / \tau)},
  \]
  where \(t_j(\theta)\) is the prompt-augmented text sequence, and \(\operatorname{sim}\) is cosine similarity [2208.11625].

- **Personalization and Generalization Balance:** Models such as FedPGP [2405.09771] introduce a global prompt \(p_G\) combined with a per-client low-rank adaptation \(\Delta p_i = U_i V_i\), yielding \(p_i = p_G + \Delta p_i\). The loss
  \[
  \mathcal{L}^i(p_G, U_i, V_i) = \mathcal{L}_{\mathrm{cls}}^i(p_G + U_i V_i) + \mu \mathcal{L}_{\mathrm{con}}^i(p_G, U_i, V_i)
  \]
  incorporates both a standard classification loss and a contrastive prompt-wise regularization to inject global knowledge and promote local specificity [2405.09771].

- **Federated Aggregation:** Typically, only prompt parameters (or their generated updates) are aggregated (e.g. server sets
  \[
  \theta^{(t+1)} = \sum_{i \in S_t} \frac{n_i}{\sum_j n_j} \theta_i^{(t+1)},
  \]
  drastically reducing communication overhead compared to full model updates [2208.11625, 2503.22263].

## 3. Design Patterns and Advanced Methodologies

PBFL has rapidly evolved into a vibrant ecosystem of algorithmic innovations:

- **Prompt Modalities:** Both text and visual prompts are supported. Visual prompts may be padding, patch tokens, style-injection vectors, or parameterized functions over client data [2303.08678, 2508.12399]. Federation can optimize either or both modalities independently or jointly (see VLPT in [2505.23024]; PLAN in [2411.10063]).
  
- **Personalization Strategies:** Approaches include low-rank or local additions to a global prompt [2405.09771], multi-group paired prompt "experts" per client [2511.00480], oracle or data-adaptive mixtures (e.g., pFedMoAP [2410.10114]), and subspace projections for divergence control [2602.08590].

- **Prompt Generators:** Instead of explicit prompt vectors per class, some methods federate a prompt-generator network \(G(T; \theta)\) parameterized by client/class-specific textual/task embedding [2310.06123, 2508.12399]. This approach generalizes to unseen classes/domains naturally and is robust to data partitioning [2310.06123].

- **Federated Domain Generalization:** In scenarios where clients represent distinct domains, federated prompt learning replaces sharing of raw statistics or prototypes (potential privacy risk) with indirect knowledge transfer via prompt exchange and attention-based or GAN-based prompt aggregators [2411.10063, 2509.20807].

- **Mixture and Specialist Designs:** Architectures such as FedMGP [2511.00480] distribute prompt capacity across multiple specialist groups, enforcing intra-client prompt diversity and dynamically aggregating only those most semantically aligned with the global prompt.

- **Communication-Efficient Protocols:** Most PBFL algorithms restrict communication to lightweight prompt parameters (typ. <1 MB per round), with strategies to further sparsify or regularize updates via prompt-level similarity graphs [2305.14244], attention-based aggregation [2411.10063], or dynamic selection [2511.00480].

- **Federated Robustness and Security:** Recent work [2508.08040] identifies prompt-tuning as a vulnerable attack surface. Backdoor prompts can propagate through global aggregation and trigger malicious behaviors. Conventional FL defenses (e.g., DP, robust aggregation) show incomplete mitigation.

## 4. Empirical Findings and Benchmarks

PBFL displays strong quantitative and qualitative performance across a diverse benchmark suite, as extensively catalogued in FLIP [2503.22263], PLAN [2411.10063], FedPGP [2405.09771], and related works. Key empirical results include:

| Method      | Global Acc (%) | Personalization (%) | Base/Novel HM (%) | Comms. per Round | Robust to Non-IID |
|-------------|----------------|---------------------|-------------------|------------------|-------------------|
| PromptFL    | 69.6           | 75.2                | 63.8              | ∼2MB             | Yes               |
| FedOTP      | 69.4           | 76.5                | 63.7              | ∼4MB             | Yes               |
| FedPGP      | 87.3 (HM)      | 91.5–99.3           | 81.8 (Novel)      | ∼1MB             | Yes               |
| FedCSAP     | 76.06 (HM)     | –                   | 75.61 (New)       | ∼1–2MB           | Yes               |
| PLAN        | up to 97.4     | –                   | –                 | ∼1.3MB           | Yes               |

- **PromptFL [2208.11625, 2503.22263]** is a strong baseline, matching full CLIP fine-tuning accuracy at less than 5% the communication cost, robust to both IID and non-IID partitions.
- **Personalization and Generalization Trade-off:** FedPGP [2405.09771] and similar models balance local accuracy and out-of-domain performance via low-rank adaptation and contrastive regularization, with harmonic mean of local/base/novel accuracy exceeding all prior methods.
- **Heterogeneous Settings:** SDFed [2602.08590] shows that allowing variable-length local prompts with subspace refinement and divergence control yields consistent gains under strong heterogeneity.
- **Domain Generalization:** PLAN [2411.10063] and FedDSPG [2509.20807] achieve state-of-the-art accuracy in cross-domain benchmarks by learning and aggregating both text and visual prompts using adaptive, privacy-preserving attention mechanisms.
- **Scalability & Efficiency:** Communication and compute are reduced by one to three orders of magnitude relative to full-model FL, enabling practical on-device deployment [2305.14244, 2503.22263].

## 5. Theoretical Foundations and Convergence Analysis

PBFL has been subject to rigorous theoretical analysis [2409.19610, 2602.08590]:

- **Feature-Learning Theory:** Signal learning (task-relevant prompt coefficients) versus noise memorization (prompt drift into spurious/null-space directions) governs test accuracy. The test error is determined by the ratio μ/σ of signal-to-noise in prompt components [2409.19610].
- **Portfolio Analogy:** Optimal mixing of global and local prompts is formalized analogously to mean-variance portfolio optimization, yielding closed-form solutions for the mixing coefficient as a function of client heterogeneity [2409.19610].
- **Subspace Refinement (SDFed):** Local prompt subspace orthogonal to the dominant global prompt ensures efficient knowledge transfer while minimizing prompt conflicts. Convergence to stationarity is guaranteed under standard FL conditions [2602.08590].
- **Dynamic Aggregation (FedMGP):** Softmax-weighted sampling and aggregation, guided by cosine similarity between local and global prompt banks, provably improves generalization and reduces bias compared to indiscriminate averaging [2511.00480].

## 6. Open Challenges, Controversies, and Future Directions

Although PBFL demonstrates parameter/communication efficiency and robust performance across diverse settings, several challenges and research frontiers remain:

- **Attack Surfaces:** PBFL is vulnerable to prompt-level backdoor poisoning [2508.08040]. Current defenses (DP, robust aggregation) are not fully effective and can compromise utility or fail under subtle prompt attacks.
- **Aggregation Sensitivity:** Uniform prompt averaging can under-represent underrepresented clients or encourage prompt collapse in highly non-IID settings. Adaptive aggregation (attention, similarity graphs, dynamic sampling) mitigates but does not eliminate these issues.
- **Task Generalization:** Most work targets classification; extending prompt FL to detection, segmentation, structured prediction, multi-modal generation, or even reinforcement learning is in early stages.
- **Prompt Structure and Scalability:** The trade-off between prompt length/capacity and expressivity versus compute/memory is not fully understood; dynamic prompt sizing and hierarchical prompt banks are active research areas.
- **Privacy Guarantees:** While prompt-based protocols reduce the exposure of sensitive feature/statistics, formal end-to-end privacy guarantees and the integration of DP/secure aggregation into PBFL pipelines remain open challenges.
- **Theoretical Understanding:** Precise convergence rates, generalization bounds under client heterogeneity, and prompt optimization landscapes continue to be developed.

## 7. Practical Implementation and Deployment Considerations

PBFL is well-suited to edge-device and privacy-sensitive deployments. Key practical guidelines extracted from FLIP [2503.22263]>:

- Use one to two prompts of length four to eight for a strong efficiency/performance trade-off.
- Prompt learning converges in 1/4–1/10 as many rounds as full-model FL; mixed-precision acceleration and on-device adaptation are feasible.
- On-device weather forecasting [2305.14244], medical visual QA [2402.09677], and graph learning [2411.02003] benefit significantly from task-specific prompt formats and personalized aggregation strategies.
- When domain-shifts or label-skew dominate, customizing prompt modality (visual vs. text), aggregation protocol, and personalization degree (hybrid global–local) is essential for maximizing transferable performance [2505.23024, 2405.09771].

PBFL is establishing itself as a general, efficient, and effective framework for federated adaptation of large foundation models across vision, language, multi-modal, time-series, and specialized domains, with a rapidly expanding body of theoretical, empirical, and systems-level research underpinning its development [2503.22263].

Source: https://www.emergentmind.com/topics/prompt-based-federated-learning