Multi-stream Prompt Tuning

• Multi-stream prompt tuning is a parameter-efficient method that adapts frozen backbones using multiple, coordinated prompt pathways instead of a single prompt.
• It integrates diverse streams—such as source-task, textual, visual, and subspace prompts—with fusion strategies like attention weighting and residual connections to improve performance.
• Empirical results show enhanced few-shot transfer and generalization, though challenges like interference among streams and increased memory footprint persist.

Multi-stream prompt tuning denotes a family of parameter-efficient adaptation schemes in which a frozen backbone is conditioned by multiple prompt components rather than by a single learned prompt. In this literature, a “stream” may be a source-task prompt in a prompt bank, a textual or visual prompt branch in a vision-LLM, a task-shared or domain-specific prompt, a learned prompt subspace, or a representation branch derived from prompt tokens. The unifying idea is that downstream adaptation is improved by coordinating, weighting, or fusing several prompt-conditioned pathways instead of optimizing one monolithic prompt in isolation (Asai et al., 2022, Miao et al., 2023, Shen et al., 2022).

1. Definition and conceptual boundaries

Standard soft prompt tuning prepends a trainable prompt to the embedded input of a frozen model and optimizes only the prompt. ATTEMPT states this baseline as

$$\underset{\mathbf{P}}{\max}~ p_{\theta}(\boldsymbol{y} \mid [\mathbf{P};\mathbf{X}]),$$

with a soft prompt $\mathbf{P} \in \mathbb{R}^{m \times d}$ prepended to the input embeddings $\mathbf{X}$ of a frozen LLM $\theta$ (Asai et al., 2022). Multi-stream prompt tuning departs from this single-prompt formulation by introducing several prompt-bearing objects and an explicit mechanism for their interaction.

The term is not used with a single uniform meaning across the literature. In ATTEMPT, the streams are reusable source-task prompts and a target prompt mixed per instance (Asai et al., 2022). In MuDPT, the streams are deep textual prompts, deep visual prompts, and transformed cross-modal prompt states coupled layerwise inside CLIP (Miao et al., 2023). In MVLPT, the primary streams are multiple task streams whose prompts are shared during source multitask learning and optionally during grouped target adaptation (Shen et al., 2022). In MePT, the streams are three complementary visual branches—global, augmented, and vanilla—extracted from one prompted CLIP model and combined by self-ensemble (Wang et al., 2024). In MPrompt, the streams correspond to task-specific, domain-specific, and context-specific prompt components with different semantic scopes (Chen et al., 2023).

Taken together, these works suggest that multi-stream prompt tuning is better understood as a design principle than as a single architecture. A stream can be prompt-space, modality-space, task-space, representation-space, or personalization-space. What remains constant is that adaptation is decomposed across multiple prompt-conditioned paths and those paths are then fused, selected, or jointly optimized (Zhang et al., 9 Apr 2025, Lan et al., 2024, Zheng et al., 21 Oct 2025).

2. Principal structural patterns

One major pattern is the prompt-bank formulation. ATTEMPT pretrains source prompts $\mathbf{P}_1,\ldots,\mathbf{P}_t$, introduces a fresh target prompt $\mathbf{P}_{\textit{target}}$, computes prompt-level attention weights $a_j$, and forms an instance-wise prompt

$$\mathbf{P}_{\textit{instance}(\mathbf{X})} = \mathbf{P}_{\it target} + \sum_{j=1}^{t+1} a_j \mathbf{P}_j.$$

Only the target prompt and the attention module $\mathcal{G}$ are updated; the LLM and all source prompts remain fixed. This makes the stream interpretation explicit: each source task contributes one prompt stream, the target task contributes another stream, and the router produces a prompt-space mixture per example (Asai et al., 2022).

A second pattern is multi-source prompt aggregation. HGPrompt treats each source visual prompt as a branch over a frozen ViT backbone and learns nonnegative convex weights $\alpha_i$ such that

$\mathbf{P} \in \mathbb{R}^{m \times d}$0

Its combination is static at the task level rather than input-adaptive, and the weights are optimized jointly for transferability and stability (Zhang et al., 9 Apr 2025). BMTPT is closely related conceptually, but it is not a live multi-stream inference architecture in the strict sense. It learns a posterior over prompts across source tasks using Stein Variational Gradient Descent, represents that posterior with prompt particles, and then aggregates them into a representative source prompt mean used to initialize and regularize target adaptation (Lee et al., 2024).

A third pattern is subspace or branch decomposition within one prompt learner. EPT decomposes a long prompt into a short prompt $\mathbf{P} \in \mathbb{R}^{m \times d}$1 and low-rank matrices $\mathbf{P} \in \mathbb{R}^{m \times d}$2, then sends the short prompt through several learned subspaces $\mathbf{P} \in \mathbb{R}^{m \times d}$3, weights those spaces with a gate $\mathbf{P} \in \mathbb{R}^{m \times d}$4, and reconstructs a final prompt via

$\mathbf{P} \in \mathbb{R}^{m \times d}$5

Here the streams are learned prompt subspaces plus a low-rank fusion branch (Lan et al., 2024).

A fourth pattern is representation branching inside a prompted model. MePT extracts three image representations from a frozen CLIP backbone under deep prompt tuning: a prompted global representation, augmented representations from final visual-prompt outputs, and the original vanilla CLIP representation. These three branches are combined by logit-level self-ensemble, with equal-weight mean reported as the best ensemble strategy (Wang et al., 2024).

3. Multimodal interaction and dual-stream prompting

The most direct multimodal interpretation appears in MuDPT. The method introduces deep textual prompts

$\mathbf{P} \in \mathbb{R}^{m \times d}$6

and deep visual prompts

$\mathbf{P} \in \mathbb{R}^{m \times d}$7

then applies an Injection Model

$\mathbf{P} \in \mathbb{R}^{m \times d}$8

followed by hierarchical bi-directional fusion

$\mathbf{P} \in \mathbb{R}^{m \times d}$9

The paper calls this “deep-symphysis,” defined as deep hierarchical bi-directional prompt fusion. The core claim is that CLIP was pretrained by aligning visual and textual representations in a common embedding space, so downstream prompt adaptation should preserve or restore that alignment rather than perturb one side in isolation (Miao et al., 2023).

ProMPT also treats prompt tuning as a dual-stream problem, but it uses recurrent cross-stream conditioning rather than residual prompt fusion. After an initialization stage and a top-$\mathbf{X}$0 feature filter, filtered text features generate class-conditional vision prompts

$\mathbf{X}$1

and the updated image feature generates instance-conditional text prompts

$\mathbf{X}$2

This loop is repeated for $\mathbf{X}$3 iterations inside the Multi-modal Iterative Evolution module. ProMPT’s stream structure is therefore bidirectional, selective, and iterative: text conditions vision prompting, vision conditions text prompting, and feature filtering keeps only the most image-relevant text features at each step (Qiu et al., 2024).

MePT is also multimodal in the sense that it employs deep visual and text prompt tuning on frozen CLIP, but its multi-stream novelty lies mainly on the visual side. The global branch uses the prompted class token, the augmented branch uses projected final visual-prompt token outputs, and the vanilla branch preserves the original CLIP image representation. The paper’s description ties these branches to conditional, marginal, and fine-grained distributions, although this association is conceptual rather than a formal probabilistic decomposition (Wang et al., 2024).

FedDEAP extends the dual-stream idea into federated, multi-domain CLIP adaptation. It learns a global semantic prompt $\mathbf{X}$4, a local domain prompt $\mathbf{X}$5, a semantic transformation network $\mathbf{X}$6, and a domain transformation network $\mathbf{X}$7. Only the semantic prompt is globally aggregated,

$\mathbf{X}$8

whereas the domain prompt remains local. The resulting architecture is explicitly “dual-prompt,” but structurally it is a two-stream prompt tuning system separating shared semantics from domain-specific personalization (Zheng et al., 21 Oct 2025).

4. Hierarchical, multitask, and granularity-aware variants

MVLPT reinterprets prompt tuning as cross-task prompt sharing. It first learns one transferable prompt across multiple source tasks and then uses that prompt to initialize each target task. It further shows that many target tasks can benefit from jointly learning shared prompt vectors during target adaptation itself. This recipe is instantiated for text prompt tuning, visual prompt tuning, and unified vision-language prompt tuning, yielding the multitask variants MCoOp, MVPT, and MUPT (Shen et al., 2022).

The unified variant in MVLPT is particularly relevant because it couples task-stream sharing with modality-stream interaction. Its prompt is partitioned into textual and visual parts,

$\mathbf{X}$9

and a lightweight Transformer layer applies self-attention and a feed-forward network before splitting the transformed prompt back into the text and visual encoders. This makes MUPT simultaneously multitask and multimodal (Shen et al., 2022).

MPrompt introduces a different notion of stream: semantic granularity. It uses task-specific prompts in encoder self-attention, decoder masked self-attention, and decoder cross-attention; domain-specific prompts indexed by unsupervised context clusters; and context-specific prompts transformed by a prompt generator conditioned on the current context. The selected domain prompt $\theta$0 and shared context prompt $\theta$1 are concatenated as

$\theta$2

and passed, together with the context $\theta$3, through a prompt generator to produce the actual generated prompt $\theta$4 inserted into the QA encoder. Domain prompts are regularized by an independence constraint based on HSIC and CKA so that different domain prompts focus on intra-domain information rather than collapsing into redundant copies (Chen et al., 2023).

BMTPT and HGPrompt occupy an intermediate position between multitask transfer and multi-stream prompting. BMTPT argues that independently training source prompts and aggregating them later ignores positive and negative interference among source tasks, so it models a joint posterior over source prompts instead (Lee et al., 2024). HGPrompt makes the interference problem explicit in visual prompt transfer and adds Gradient Alignment Regularization,

$\theta$5

to suppress conflicts among prompt branches (Zhang et al., 9 Apr 2025). A plausible implication is that stream multiplicity alone is insufficient; the streams must also be made compatible.

5. Optimization principles and empirical record

Across these methods, the backbone is typically frozen and only prompts and small auxiliary modules are updated. This is explicit in ATTEMPT, MuDPT, HGPrompt, MVLPT, BMTPT, MePT, ProMPT, EPT, and MPrompt, although the precise trainable components differ by architecture (Asai et al., 2022, Miao et al., 2023, Zhang et al., 9 Apr 2025, Shen et al., 2022, Lee et al., 2024, Wang et al., 2024, Qiu et al., 2024, Lan et al., 2024, Chen et al., 2023).

The empirical evidence is broad but benchmark-specific. On few-shot CLIP adaptation, MuDPT reports 81.38 average on the 11-dataset 16-shot benchmark, compared with 64.65 for CLIP, 73.18 for CoOp, and 75.07 for CoCoOp; on base-to-new evaluation it reports base 82.71, new 74.5, and average 78.39 (Miao et al., 2023). ProMPT reports base 80.94, new 74.89, and harmonic mean 77.80 on the 11-dataset base-to-novel benchmark, improving over CoCoOp’s 75.83 harmonic mean; it also reports 66.25 average in cross-dataset transfer and 60.25 average in domain generalization (Qiu et al., 2024). MePT reports base 84.63, novel 76.30, and harmonic mean 80.25, slightly above PromptSRC’s 79.97, and 66.11 average in cross-dataset generalization after ImageNet training (Wang et al., 2024).

In multitask and transfer-oriented vision-language prompting, MVPT reaches 74.13% average on the 20-shot ELEVATER benchmark with CLIP ViT-B/16, with the table reporting the best multitask variant’s gain over the best baseline as $\theta$6 (Shen et al., 2022). HGPrompt reports 59.6 average on VTAB-1k, ahead of PANDA’s 58.7, SPoT’s 58.5, ATTEMPT’s 56.9, and VPT’s 52.3 (Zhang et al., 9 Apr 2025).

In NLP, ATTEMPT-m reports 85.8 average on the 8 GLUE tasks shown, 74.1 on SuperGLUE average, 72.8 average F1 on MRQA, and 85.6 average on WinoGrande/Yelp/SciTail/PAWS, while updating far fewer parameters than full fine-tuning or adapters (Asai et al., 2022). BMTPT reports 88.7 on GLUE average and 74.6 on SuperGLUE average with T5-base, and states that it trains only 0.035% of the parameters compared to full fine-tuning (Lee et al., 2024). EPT reports 86.8 on GLUE and 77.3 on SuperGLUE, with training time decreased by 14% relative to vanilla prompt tuning at the selected configuration (Lan et al., 2024). MPrompt reports an average improvement of 1.94\% over state-of-the-art methods across 12 QA benchmarks and improves over Prefix-tuning by +2.17% on UnifiedQA-Base, +1.85% on UnifiedQA-Large, and +1.82% on UnifiedQA-XL (Chen et al., 2023).

These figures support the central claim shared across the literature: when prompt-conditioned pathways are explicitly separated and then coordinated—by attention, convex weighting, residual fusion, recurrence, task sharing, subspace gating, or ensemble—the resulting model often yields better few-shot transfer, base-to-novel generalization, or cross-task robustness than single-stream prompt tuning (Asai et al., 2022, Miao et al., 2023, Qiu et al., 2024).

6. Limitations, misconceptions, and open questions

A common misconception is that multi-stream prompt tuning always implies input-adaptive routing. ATTEMPT is input-adaptive because it computes instance-level prompt weights $\theta$7 (Asai et al., 2022), whereas HGPrompt uses a single task-level weight vector $\theta$8 learned for the target task and applies it across samples (Zhang et al., 9 Apr 2025). BMTPT is multi-stream during source posterior learning but compresses that structure into a single representative prompt for target deployment, so it is better characterized as a multi-source prompt transfer framework than as a persistent multi-stream inference architecture (Lee et al., 2024).

Another misconception is that more streams necessarily help. ATTEMPT’s modularity analysis shows that some prompts can cause negative transfer, such as QA prompts for RTE (Asai et al., 2022). MVLPT reports that bad task grouping can be actively harmful: for MUPT, “Worst M” drops to 69.94 versus 73.39 for “Best M” (Shen et al., 2022). HGPrompt is motivated precisely by the fact that naive prompt aggregation can lead to representation collapse and mutual interference (Zhang et al., 9 Apr 2025). BMTPT reports that 10 particles did not improve over 5, attributing this to SVGD instability or kernel sensitivity (Lee et al., 2024).

The cost profile is also uneven. ATTEMPT is parameter-efficient but increases memory footprint: on T5-base it reports 13.7 GB for ATTEMPT versus 8.5 GB for prompt tuning, and on T5-XL 16.1 GB versus 15.9 GB (Asai et al., 2022). ProMPT adds recurrent iterations, feature filtering, and cross-modal generators (Qiu et al., 2024). HGPrompt requires multiple prompt-conditioned forward passes, prompt-specific gradients, and weight optimization over source branches (Zhang et al., 9 Apr 2025). EPT reduces explicit prompt length but still introduces multi-space projections and a two-learning-rate tuning burden noted by the authors as future work (Lan et al., 2024). MVLPT adds a one-time compute cost for multitask source prompt initialization (Shen et al., 2022).

Several papers also report robustness caveats. MuDPT notes that on 3 of 11 datasets in base-to-new evaluation it still lags behind zero-shot CLIP (Miao et al., 2023). MePT reports mixed domain-generalization evidence: on the four ImageNet variants, PromptSRC’s average is 60.65 while MePT’s is 60.36, although MePT achieves the best result on ImageNetV2 (Wang et al., 2024). ProMPT depends on an initial top-$\theta$9 filter; if the early ranking is poor, later cross-stream prompting may begin from noisy candidates (Qiu et al., 2024). MPrompt relies on unsupervised clustering for latent domains, so domain assignments can be imperfect (Chen et al., 2023). FedDEAP leaves the behavior of personalized domain prompts on unseen clients or domains largely unexplored (Zheng et al., 21 Oct 2025).

These limitations suggest several open directions already implicit in the literature: larger prompt banks with scalable routing, more explicit analyses of cross-stream compatibility, extensions beyond CLIP-style dual encoders and classification settings, and stronger methods for deciding which prompt factors should be shared, which should remain private, and which should be dynamically composed (Zhang et al., 9 Apr 2025, Shen et al., 2022, Zheng et al., 21 Oct 2025).