Synchronous Dual Prompt Tuning (SDPT)

• SDPT is a parameter-efficient fine-tuning method for visual-language models that unifies prompt injection across text and image modalities.
• It leverages inverse linear projections to map a single set of learnable prototype tokens into both modalities at every fusion layer, ensuring synchronous semantic alignment.
• The approach outperforms previous dual-prompt techniques in detection and grounding tasks while maintaining minimal additional parameters.

Synchronous Dual Prompt Tuning (SDPT) is a parameter-efficient fine-tuning approach for visual–LLMs, designed to jointly optimize text and image modalities within fusion-based VLPMs. SDPT achieves cross-modal alignment by injecting a single, learnable set of prototype tokens—shared across modalities—at multiple fusion layers, guaranteeing that identical semantic content is visible and tunable in both text and image branches during every stage of deep multimodal interaction. By leveraging inverse linear projections derived from the pretrained model's internal query maps, SDPT inserts this unified prompt directly into the pre-fusion latent spaces, avoiding any additional model parameters. This approach yields state-of-the-art transfer performance in scenarios demanding generalization across both modalities, particularly for detection and grounding tasks in VLPM architectures such as GLIP (Zhou et al., 2024). The SDPT mechanism substantially outperforms previous prompt tuning and dual-modal adaptation methods, establishing new standards for parameter-efficient cross-modal adaptation.

1. Architecture of Synchronous Dual Prompt Tuning

SDPT operates within a dual-encoder/fusion architecture typical of modern VLPMs such as GLIP, which comprises separate $L$-layer Transformer stacks for text and image (potentially with BERT and Swin backbones, respectively), interconnected by multi-layer Cross Multi-Head Attention (X-MHA) modules. At each X-MHA layer, input text and image embeddings ($P^i\in \mathbb{R}^{n\times d_T}$, $R^i \in \mathbb{R}^{m\times d_I}$) are processed via modality-specific downstream encoders, with the X-MHA module facilitating inter-modal information exchange. The SDPT construction interposes $k$ learnable, unified prototype tokens $Z^i\in\mathbb{R}^{k\times d}$—directly parameterized in the fusion (cross-modal) latent space—at the input of every X-MHA, synchronously appended to text and image streams after inverse-projection into each modality’s input space.

Mathematically, at each layer $i$: $$\widehat{P}^i = [Z^{i, (T)}, P^i] \in \mathbb{R}^{(k + n) \times d_T}, \quad \widehat{R}^i = [Z^{i, (I)}, R^i] \in \mathbb{R}^{(k + m) \times d_I}$$ These augmented sequences are then passed to X-MHA and encoders, with the prompt token rows trimmed post-attention. Across the full model, all original weights remain frozen; only the unified prompt parameters $Z^i$ are updated (Zhou et al., 2024).

2. Unified Prototype Tokens and Inverse Linear Projections

A core innovation in SDPT is the representation of the cross-modal prompt as a single, learnable matrix in the joint fusion space, alongside a deterministic mechanism for projecting this matrix into the native input spaces of each encoder. Specifically, given pretrained query projection matrices and biases $(W^{(q,T)}, B^{(q,T)})$ and $(W^{(q,I)}, B^{(q,I)})$, SDPT computes their Moore–Penrose pseudoinverses $W^{(q,T)\,\dagger}$ and $W^{(q,I)\,\dagger}$ offline.

For each fusion-layer $i$, prompt tokens for each modality are synthesized as: $$Z^{i, (T)} = [Z^i - \mathbf{1}B^{(q,T)}]W^{(q,T)\,\dagger} \in \mathbb{R}^{k\times d_T}\ Z^{i, (I)} = [Z^i - \mathbf{1}B^{(q,I)}]W^{(q,I)\,\dagger} \in \mathbb{R}^{k\times d_I}$$ This strictly non-trainable mapping ensures all trainable semantics reside in $Z^i$ itself, fully in the model’s existing fusion space, which encodes pre-aligned text-image relationships (Zhou et al., 2024).

3. Synchronous Modal Injection and Training Procedure

SDPT enforces strict synchrony between text and image prompt injection: the same $Z^i$ is mapped via fixed inverse linear transformations and simultaneously inserted into both the text and image encoder input streams at every X-MHA layer. No parameters besides the prototype tokens are updated. Optimization uses the original VLPM loss (e.g., detection/classification, bounding-box regression, IoU), identical batch size, learning rate, and schedule as full model fine-tuning. All heavy encoder and fusion weights are frozen. For GLIP-L, tuning all $Z^i$ across $L=8$ layers with $k=10$ and $d=1024$ yields approximately 0.04% of the full parameter count; with $k=120$, this allocation rises to 0.5% ($\sim$2M parameters) (Zhou et al., 2024).

The effectiveness of strict synchrony is supported by ablation: separating text and image prompt learning results in drops of several points in mAP (e.g., 57.7$\rightarrow$54.8), and non-shared tokens fall further (53.3). Thus, enforcing a single cross-modal semantics yields significant empirical gains over asynchronous or non-shared prompt regimes (Zhou et al., 2024).

4. Comparison with Dual-Modality Approaches and Extensions

SDPT is a distinct advance over earlier dual-prompt techniques. Traditional methods such as DPT (Dual-Modality Prompt Tuning) (Xing et al., 2022) independently learn text and visual prompts (either naive or class-aware), often combined with lightweight trainable cross-attention heads. DPT’s synchronous training adjusts both prompt sets via the same loss, but keeps textual and visual prompts physically separated and learns separate parameters for each branch. SDPT, in contrast, encodes all cross-modal semantics in a single prompt matrix and synchronously projects it into both streams, minimizing tunable parameters and enforcing deep fusion-layer alignment.

Mechanistically, DPT’s visual prompts operate within the ViT patch sequence and may incorporate class semantics via class-aware cross-attention. SDPT operates at a deeper fusion level and leverages pretrained cross-attention query projections for semantic injection, enabling direct control over the joint space where modalities interact most tightly.

Other contemporary works such as DPC (Dual-Prompt Collaboration) (Li et al., 17 Mar 2025) clone and fine-tune a parallel text-side prompt, mix base and new class optimizations via a weighting/decoupling scheme, and apply hard negative mining; this architecture remains entirely text-domain and orthogonal to SDPT’s design, which handles full cross-modal prompt alignment in fusion-type backbones (Zhou et al., 2024).

5. Empirical Results and Ablations

SDPT demonstrates superior parameter efficiency and accuracy on diverse detection benchmarks. On COCO, LVIS, and ODinW13, when compared with full fine-tuning (FT), LoRA, MaPLe, and UPT, SDPT with $k=120$ achieves the best or highly competitive mAP/AP:

Method	COCO mAP	LVIS AP	ODinW13 full-shot	Tunable #Params (%)
Full FT	60.8	41.2	68.9	397.6M (100%)
MaPLe	57.2	40.8	68.7	2.96M (0.74%)
SDPT ($k$=10)	57.6	41.2	69.5	0.16M (0.04%)
SDPT ($k$=120)	58.0	41.4	71.2	1.97M (0.50%)

Notably, SDPT with only 0.5% tunable parameters outperforms full fine-tuning on LVIS and ODinW13 and exceeds all tested PEFT (parameter-efficient fine-tuning) baselines. Ablations further show SDPT’s performance is robust to prompt length $k$ (stable 57.1–57.7 mAP for $k\in\{10,100,200,400\}$), and nearly optimal if prototype tokens are inserted only in the first and last X-MHA layers (Zhou et al., 2024).

When contrasted with DPT (Xing et al., 2022), SDPT avoids the need for additional trainable cross-attention and instead leverages fixed inverse projections, thus reducing parameter count and increasing training stability.

6. Implementation and Practical Considerations

SDPT’s implementation wraps each X-MHA in a module hosting an $nn.Parameter$ $Z^i$ and stores precomputed pseudoinverse and bias terms for projection. Forward passes compute and inject projected prototype tokens, perform cross-modal attention as in the original model, and slice prompt rows prior to the next layer. Data preprocessing exactly follows GLIP: resizing, normalization, and padding. AdamW is used with typical learning rates of $1\mathrm{e}{-4}$ for $Z^i$ and batch sizes $16$–$32$ per GPU, with convergence in as few as 12 epochs.

No changes to the detection/classification head or loss function are necessary, supporting straightforward integration into established VLPM codebases. For few-shot settings, results are typically averaged over three random seeds (Zhou et al., 2024).

7. Significance and Broader Context

SDPT extends the frontier of parameter-efficient cross-modal adaptation for fusion-based VLPMs. By unifying prompt semantics, enforcing synchrony across branches, and projecting into pre-trained alignment subspaces, SDPT addresses the modal-mapping/aligning issues that limit the transfer capacity of previous prompt tuning techniques—especially in deep interleaved fusion models as opposed to parallel dual-encoder structures. Empirical improvements are substantial, and robust parameter-efficiency makes SDPT well-suited for scalable or resource-constrained deployment. In comparison to prompt-splitting or weighting in text-focused dual-prompt designs (Li et al., 17 Mar 2025), SDPT provides a fusion-centric paradigm for joint multimodal adaptation.

Ongoing work in prompt-based PEFT continues to refine the balance between specialization and generalization across both modalities and tasks. The SDPT architecture demonstrates that synchrony and deep fusion-level prompt injection yield strong generalization and transfer, suggesting further research into projection schemes and the interplay between prompt position, length, and sharing strategies in multimodal backbones (Zhou et al., 2024, Xing et al., 2022, Li et al., 17 Mar 2025).