BadPromptFL in Federated Prompt Learning

• BadPromptFL is a backdoor attack in prompt-based federated learning that leverages prompt aggregation to embed covert triggers in multimodal models.
• It exploits vulnerabilities by allowing compromised clients to jointly optimize triggers and prompt embeddings, resulting in high attack success rates with minimal accuracy loss.
• Emerging defenses and design principles highlight the need for prompt-specific security mechanisms to mitigate attacks and privacy leaks in federated settings.

Searching arXiv for papers on BadPromptFL and closely related federated prompt-learning security work. First, searching for the BadPromptFL backdoor-attack paper by title and related prompt-based federated learning security papers. BadPromptFL denotes a failure mode of prompt-based federated learning. In its strict sense, it names a backdoor attack on prompt-based federated learning in multimodal contrastive models, in which compromised clients jointly optimize local backdoor triggers and prompt embeddings so that poisoned prompts enter global aggregation and enable universal backdoor activation at inference without modifying model parameters (Zhang et al., 11 Aug 2025). In a broader research usage, the term also captures prompt-based federated learning that becomes ineffective, fragile, or privacy-leaking: prompts may overfit to particular clients, fail under heterogeneous aggregation, break under perturbed inputs, or expose membership information through prompt-selection dynamics (Chang et al., 23 Apr 2025, Shi et al., 2024, Nguyen et al., 10 Jan 2026).

1. Prompt-based federated learning as the substrate

Prompt-based federated learning emerged from the convergence of parameter-efficient adaptation and federated optimization. In multimodal CLIP-style systems, prompt tuning replaces full-model fine-tuning with learned contextual prompts while keeping the backbone encoders frozen; in federated settings, clients collaboratively train prompts under data-privacy constraints, and the server aggregates prompt parameters rather than full model weights (Zhang et al., 11 Aug 2025). This paradigm is attractive because prompts are computationally cheap, communication-efficient, and semantically powerful, but these same properties create a distinct attack surface: prompts function as behavioral templates, and their aggregation merges behavioral templates rather than conventional full-network gradients (Zhang et al., 11 Aug 2025).

Several adjacent systems define the design space within which BadPromptFL becomes meaningful. "Visual Prompt Based Personalized Federated Learning" (Li et al., 2023) keeps a shared backbone and learns client-specific visual prompts that are never transmitted; only the backbone is aggregated, while prompts encode local data-distribution information. "Tunable Soft Prompts are Messengers in Federated Learning" (Dong et al., 2023) goes further in the opposite direction: soft prompts become the only shared object, while a proprietary global LLM remains solely on the server and clients train with lightweight auxiliary models. "DP2FL: Dual Prompt Personalized Federated Learning in Foundation Models" (Chang et al., 23 Apr 2025) introduces a global task prompt and client-specific data prompts, explicitly to avoid ineffective prompts that either overfit to particular clients or fail to personalize.

This background matters because BadPromptFL is not an isolated pathology. It arises precisely where prompts are shared, aggregated, or used as the main control surface for foundation models in heterogeneous federated environments. The literature therefore treats prompt design, prompt aggregation, and prompt privacy as first-order systems questions rather than narrow optimization details (Chang et al., 23 Apr 2025, Dong et al., 2023).

2. BadPromptFL as a backdoor attack

In the strict usage introduced by "BadPromptFL: A Novel Backdoor Threat to Prompt-based Federated Learning in Multimodal Models" (Zhang et al., 11 Aug 2025), the setting is prompt-based federated learning for CLIP-style vision-LLMs. CLIP consists of an image encoder $f(\cdot)$ and a text encoder $g(\cdot)$ trained with an InfoNCE-style loss,

$$\mathcal{L}_{\text{CLIP}} = - \log \frac{\exp(\text{sim}(f(x), g(t))/\tau)}{\sum_{t' \in \mathcal{T}} \exp(\text{sim}(f(x), g(t'))/\tau)}.$$

Prompt tuning adapts CLIP by learning a context matrix $\mathcal{P} \in \mathbb{R}^{M \times d}$ while keeping the backbone frozen, and PromptFL-style aggregation updates the global prompt by weighted averaging,

$$\mathbf{p}_{\text{global}}^{(r)} = \sum_{i=1}^{N} w_i \cdot \mathbf{p}_i^{(r)}.$$

BadPromptFL exploits exactly this aggregation mechanism. A subset $\mathcal{A}$ of clients is adversarial, with $\rho \ll 1$, and malicious clients jointly optimize a learnable trigger $\delta$ and prompt vector $\mathbf{p}$. Triggered images are constructed as

$$\tilde{\mathbf{x}} = \mathrm{Clip}(\mathbf{x}_c + \delta),$$

and malicious local training minimizes a mixed objective

$g(\cdot)$0

where the backdoor term aligns the triggered image embedding with the target text embedding. In the detailed formulation, clean and poisoned losses are written as

$g(\cdot)$1

$g(\cdot)$2

with the combined objective

$g(\cdot)$3

The core algorithm uses alternating optimization on malicious clients. First, the trigger is updated while the prompt is fixed; second, the prompt is updated while the trigger is fixed. Benign clients perform standard prompt tuning on clean data. The server remains honest but naive and aggregates client prompts with FedAvg-style averaging. Over many rounds, the aggregated global prompt acquires a universal association of the trigger pattern with an attacker-chosen target class, even though the backbone encoders are never modified (Zhang et al., 11 Aug 2025).

This architecture makes the attack distinct from classical model-poisoning backdoors. The poisoned behavior is encoded in prompt space, not in the frozen model weights. Because prompts are low-dimensional and task-specific, the malicious behavior can be hidden in a narrow subspace while preserving normal downstream accuracy on clean inputs. This is why the paper characterizes prompt aggregation itself as a new security surface in multimodal federated learning (Zhang et al., 11 Aug 2025).

3. Empirical behavior, stealth, and generalizability

BadPromptFL is evaluated on CLIP RN50 and CLIP ViT-B/16 with frozen backbones and contextual prompts, under non-IID label-partitioned client data, full client participation, 100 communication rounds, 5 local epochs per client per round, and few-shot 6-shot tuning (Zhang et al., 11 Aug 2025). The datasets are Caltech101, FGVCAircraft, Stanford Cars, Oxford Pets, Oxford Flowers, EuroSAT, UCF101, and DTD. The reported metrics are clean accuracy (ACC) and attack success rate (ASR).

The central empirical pattern is high ASR with limited loss of clean performance. On RN50, Caltech101 yields clean 6-shot ACC of 91.03%, while 6-shot PromptFL plus BadPromptFL yields ACC = 90.91% and ASR = 90.47%; on ViT, Caltech101 yields clean 6-shot ACC of 95.58%, while 6-shot plus BadPromptFL yields ACC = 95.13% and ASR = 92.17% (Zhang et al., 11 Aug 2025). The same pattern holds across diverse domains, including remote sensing and action recognition.

Dataset	RN50 ACC / ASR	ViT-B/16 ACC / ASR
Caltech101	90.91 / 90.47	95.13 / 92.17
Oxford Flowers	91.84 / 98.58	94.76 / 95.45
EuroSAT	76.32 / 93.23	75.19 / 99.52
UCF101	71.72 / 92.68	77.61 / 95.35

These results support the claim that prompt-space backdoors are both effective and stealthy. The trigger is a learnable additive noise patch constrained by clipping, and client behavior remains indistinguishable from benign participation in timing, message size, and update type (Zhang et al., 11 Aug 2025). The paper also reports that the attack remains effective across multiple aggregation and defense settings, including FedAvg, MKrum, differential privacy, Foolsgold, and Multi Metric, indicating that prompt-space poisoning does not behave like obvious outlier corruption in high-dimensional weight space (Zhang et al., 11 Aug 2025).

A second empirical line comes from "SABRE-FL: Selective and Accurate Backdoor Rejection for Federated Prompt Learning" (Khan et al., 25 Jun 2025). That paper likewise shows that federated prompt learning with CLIP-style models is vulnerable to targeted misclassification under learnable, visually imperceptible triggers while maintaining high clean accuracy. Without defense, its reproduced CA/BA numbers include Flowers 77.9 / 41.7, Pets 94.2 / 16.3, DTD 65.6 / 34.8, Aircraft 32.8 / 93.9, and Food-101 90.0 / 20.6 (Khan et al., 25 Jun 2025). Although the reported scales differ from BadPromptFL because the experimental protocol differs, the common result is that prompt-only federated adaptation is sufficiently expressive to memorize and propagate a backdoor while preserving benign utility.

4. Defenses and the rise of prompt-specific security mechanisms

Existing federated defenses perform unevenly against prompt-space backdoors. In the BadPromptFL evaluation, FedAvg typically yields ASR above 90% with minimal clean-accuracy drop; MKrum reduces ASR only modestly; differential privacy can reduce ASR only at severe cost to ACC; Foolsgold can nearly eliminate the backdoor in some cases, such as RN50 on EuroSAT with ACC = 32.12% and ASR $g(\cdot)$4, but on other datasets ASR remains substantial and ACC often falls below baseline; Multi Metric shows similarly mixed behavior (Zhang et al., 11 Aug 2025). The common finding is that no evaluated defense simultaneously preserves benign PromptFL utility and drives ASR close to zero.

SABRE-FL is the main prompt-specific defense in the supplied literature. Its core premise is that backdoor triggers may be visually imperceptible but not embedding-imperceptible: if $g(\cdot)$5 and $g(\cdot)$6, then there exists a margin $g(\cdot)$7 such that $g(\cdot)$8 consistently over poisoned samples (Khan et al., 25 Jun 2025). SABRE-FL therefore trains a binary detector $g(\cdot)$9 offline on an auxiliary out-of-distribution dataset, using CLIP image embeddings of clean and triggered samples. During federated training, each client sends prompt updates together with a set of embeddings $$\mathcal{L}_{\text{CLIP}} = - \log \frac{\exp(\text{sim}(f(x), g(t))/\tau)}{\sum_{t' \in \mathcal{T}} \exp(\text{sim}(f(x), g(t'))/\tau)}.$$0, and the server computes a client-level anomaly score

$$\mathcal{L}_{\text{CLIP}} = - \log \frac{\exp(\text{sim}(f(x), g(t))/\tau)}{\sum_{t' \in \mathcal{T}} \exp(\text{sim}(f(x), g(t'))/\tau)}.$$1

Given an assumed upper bound $$\mathcal{L}_{\text{CLIP}} = - \log \frac{\exp(\text{sim}(f(x), g(t))/\tau)}{\sum_{t' \in \mathcal{T}} \exp(\text{sim}(f(x), g(t'))/\tau)}.$$2 on malicious clients, the server removes the top-$$\mathcal{L}_{\text{CLIP}} = - \log \frac{\exp(\text{sim}(f(x), g(t))/\tau)}{\sum_{t' \in \mathcal{T}} \exp(\text{sim}(f(x), g(t'))/\tau)}.$$3 clients with highest $$\mathcal{L}_{\text{CLIP}} = - \log \frac{\exp(\text{sim}(f(x), g(t))/\tau)}{\sum_{t' \in \mathcal{T}} \exp(\text{sim}(f(x), g(t'))/\tau)}.$$4 before aggregation (Khan et al., 25 Jun 2025).

Empirically, SABRE-FL substantially reduces backdoor accuracy while preserving clean accuracy. On Flowers it reaches 76.6 / 1.1 CA / BA; on Pets 94.5 / 4.4; on DTD 64.9 / 6.8; on Aircraft 32.1 / 7.6; and on Food-101 90.6 / 1.9, outperforming Trimmed Mean, Median, Norm Bounding, and FLAME on all five datasets (Khan et al., 25 Jun 2025). The defense is server-side only, requires no raw client data or labels, and exploits the fact that CLIP embeddings of triggered samples form well-separated clusters in the frozen representation space (Khan et al., 25 Jun 2025).

Taken together, these results suggest that defenses designed for full-model Byzantine robustness are poorly matched to prompt-level multimodal attacks. Prompt aggregation merges low-dimensional, semantically loaded control vectors, so defense mechanisms that reason directly in embedding space or prompt space appear more appropriate than generic gradient-space filtering (Zhang et al., 11 Aug 2025, Khan et al., 25 Jun 2025).

5. Beyond backdoors: privacy leakage, prompt injection, and fragile prompt behavior

BadPromptFL also has a broader technical meaning in the surrounding literature: prompts in federated systems can be bad not only because they are backdoored, but also because they leak membership, admit injection, or fail under distribution shift. "Leveraging Soft Prompts for Privacy Attacks in Federated Prompt Tuning" (Nguyen et al., 10 Jan 2026) shows that federated prompt tuning introduces a new membership-inference vector. In PromptMIA, a malicious server inserts adversarially crafted keys and prompts into a shared prompt pool and monitors which prompts are updated by clients. The attack is formalized through a security game with advantage

$$\mathcal{L}_{\text{CLIP}} = - \log \frac{\exp(\text{sim}(f(x), g(t))/\tau)}{\sum_{t' \in \mathcal{T}} \exp(\text{sim}(f(x), g(t'))/\tau)}.$$5

and the paper proves $$\mathcal{L}_{\text{CLIP}} = - \log \frac{\exp(\text{sim}(f(x), g(t))/\tau)}{\sum_{t' \in \mathcal{T}} \exp(\text{sim}(f(x), g(t'))/\tau)}.$$6 by construction for the target sample. Empirically, PromptMIA consistently achieves high advantage and ASR above 90% across datasets and models, while differential privacy and classical anomaly detection leave substantial residual leakage or degrade utility sharply (Nguyen et al., 10 Jan 2026).

A different failure mode is prompt fragility under input perturbation. "Robustness-aware Automatic Prompt Optimization" (Shi et al., 2024) is not a federated paper, but it is directly relevant to the broader notion of bad prompts. It shows that prompt optimization on clean validation sets can produce prompts that degrade sharply under typos, synonym replacements, paraphrases, and structural changes, and it proposes BATprompt, which optimizes prompts on adversarially generated perturbations. On XSum under C2 perturbation, BATprompt improves ROUGE-1 / ROUGE-2 / ROUGE-L to 21.68 / 4.76 / 16.42, compared with 17.62 / 3.16 / 14.96 for EvoPrompt and 18.31 / 3.08 / 15.50 for BATprompt without adversarial training (Shi et al., 2024). This suggests that robustness failures in prompt design are not confined to FL, but federated heterogeneity can exacerbate them.

The prompt-injection perspective broadens the threat model still further. "Exploring Potential Prompt Injection Attacks in Federated Military LLMs and Their Mitigation" (Lee et al., 30 Jan 2025) identifies four vulnerabilities in federated military LLMs—secret data leakage, free-rider exploitation, system disruption, and misinformation spread—and treats prompt injection as both an operational-time and training-time risk. The paper proposes a human-AI collaborative framework combining red/blue team wargaming, continuous quality assurance, joint policy development, and verification of security protocols (Lee et al., 30 Jan 2025). Although this work is conceptual rather than empirical, it reinforces the view that prompt failures in federated systems are simultaneously optimization, security, and governance problems.

This broader usage is consistent with the software-engineering taxonomy in "A Taxonomy of Prompt Defects in LLM Systems" (Tian et al., 17 Sep 2025), which organizes prompt defects into six dimensions: Specification and Intent, Input and Content, Structure and Formatting, Context and Memory, Performance and Efficiency, and Maintainability and Engineering. A plausible implication is that backdoored prompts, privacy-leaking prompt pools, and heterogeneity-misaligned prompt aggregators are all instances of prompt defects manifesting at system scale rather than only at single-model inference time (Tian et al., 17 Sep 2025).

6. Design principles, countermeasures, and open problems

The literature suggests that effective prompt-based federated learning depends on explicit role separation among prompts, aggregation rules, and frozen backbones. In DP2FL, the task prompt is global and encodes task-level knowledge, while the data prompt is client-specific and captures local distributional information; aggregation is loss-aware rather than uniform, and a global model consisting of the global task prompt and a global data prompt supports prediction on new data sources and initialization of new clients (Chang et al., 23 Apr 2025). In pFedPT, client-specific visual prompts remain local and only the backbone is aggregated; prompts act as data-level personalization mechanisms rather than globally shared behavioral templates (Li et al., 2023). In FedSP, tunable soft prompts are the only shared object, but they are engineered as messengers between a proprietary server-side LLM and distilled auxiliary client models, specifically to protect the global model and reduce communication and computation cost (Dong et al., 2023).

These designs point to a common lesson: prompt sharing is powerful precisely because prompts are semantically concentrated. That concentration can support personalization and efficiency, but it can also support poisoning and inference. A plausible design principle is therefore to separate shared prompts that encode global task semantics from private prompts that encode local data peculiarities, to aggregate prompts using performance-aware or embedding-aware criteria rather than naive averaging, and to reason about prompt-space security independently of standard weight-space robustness (Chang et al., 23 Apr 2025, Khan et al., 25 Jun 2025).

Open problems remain substantial. BadPromptFL explicitly argues that existing FL defenses, designed for full-model gradient attacks, are not well-suited to prompt-level multimodal backdoors (Zhang et al., 11 Aug 2025). SABRE-FL assumes an upper bound on the number of malicious clients and studies data poisoning with learnable triggers rather than direct prompt manipulation (Khan et al., 25 Jun 2025). PromptMIA shows that even when the backbone is frozen and only a small prompt pool is shared, the server can mount a high-advantage membership-inference attack, and secure aggregation for non-linear prompt aggregation remains unresolved (Nguyen et al., 10 Jan 2026). The broader prompt-injection literature adds governance and audit challenges, especially in multi-party deployments (Lee et al., 30 Jan 2025).

BadPromptFL is therefore best understood as a research nexus rather than a single attack primitive. At its narrowest, it is a prompt-space backdoor on multimodal federated learning (Zhang et al., 11 Aug 2025). At its broadest, it names the failure of prompt-based federation to remain robust, personalized, and private under heterogeneity, adversarial participation, and semantically concentrated control channels (Chang et al., 23 Apr 2025, Nguyen et al., 10 Jan 2026).