Dynamic Prompt Module (DPM)
- Dynamic Prompt Module (DPM) is an adaptive control pattern that conditionally constructs or modifies prompt representations based on input, task, or temporal conditions.
- It employs techniques such as policy-based selection, cross-attentive fusion, and online memory management to dynamically tailor prompt guidance.
- Empirical results show that DPM enhances performance in tasks like lifelong person re-identification, vision-language adaptation, and diffusion model patching.
Searching arXiv for papers on dynamic prompt modules and related prompt-adaptation frameworks. {"16query16 prompt\"16 OR \16"dynamic prompt module\"16 OR \16"dynamic prompt fusion\"16 OR \16"dynamic test-time prompt tuning\"16 OR \16"prompt learning\"16)16 {"16query16 OR \16"Dynamic Textual Prompt For Rehearsal-free Lifelong Person Re-identification\" OR 16all:(\16 OR \16"DynaPrompt: Dynamic Test-Time Prompt Tuning\" OR 16all:(\16 OR \16"UniCell: Universal Cell Nucleus Classification via Prompt Learning\"","16max_results16 OR \16,"16sort_by16 Dynamic Prompt Module (DPM16)16^ denotes a class of mechanisms that make prompt construction, selection, fusion, or application conditional on the current input, task, domain, timestep, or interaction state, rather than treating prompts as fixed parameters shared uniformly across all examples. Across recent work, the term does not name a single canonical architecture: in some papers it is an explicit module, as in UniCell and ProDA; in others it is a generic label that maps to a differently named component such as Dynamic Prompt Fusion in lifelong person re-identification or Dynamic PRC in prompt middleware. Despite this terminological variation, the common objective is stable: convert prompts from static conditioning artifacts into adaptive control variables that modulate representation learning, inference-time behavior, or user interaction (&&&16query16&&&
16all:(\16. Conceptual scope and terminology
The modern DPM literature spans several distinct problem settings. In lifelong person re-identification, the dynamic prompt component is officially named the Dynamic Prompt Fusion module inside the Dynamic Textual Prompt framework, where it fuses invariant textual descriptions and identity-specific prompt tokens into instance-wise semantic anchors (&&&16all:(\16&&& In test-time adaptation for vision-LLMs, DynaPrompt uses a Dynamic Prompt Module to maintain a prompt buffer, select relevant prompts online, update only the selected subset, and append fresh prompts when no existing prompt is suitable (&&&16 OR \16&&&16)16 In universal nuclei classification, DPM is a dataset- and category-conditioned prompt mechanism inserted after a deformable Transformer encoder to refine image features via cross-attention (&&&16 OR \16&&&16)16 In diffusion model patching, the “mixture-of-prompts” design plays the role of a DPM by dynamically gating learnable prompt embeddings at each reverse denoising timestep while the backbone remains frozen (&&&16 OR \16&&&16)16
| Setting | DPM form | Representative paper |
|---|---|---|
| Prompt tuning in frozen backbones | Instance-dependent position, length, or representation | (&&&16query16&&& |
| Rehearsal-free lifelong ReID | Cross-attentive fusion of invariant and identity prompts | (&&&16all:(\16&&& |
| Test-time VLM adaptation | Online prompt buffer with selection, update, append, delete | (&&&16 OR \16&&&16)16 |
| Universal pathology | Dataset/category prompt integration for feature refinement | (&&&16 OR \16&&&16)16 |
| Frozen diffusion patching | Timestep-gated mixture-of-prompts | (&&&16 OR \16&&&16)16 |
| Multimodal continual instruction tuning | Cross-modal generator producing instance-specific soft prompts | (&&&16all:(\16query16&&& |
This diversity clarifies a central terminological point. “Prompt” may refer to natural-language descriptions, continuous prompt embeddings, dataset/category prompt tensors, multi-hot action specifications, or dynamically generated interface controls. Likewise, “dynamic” may mean per-instance selection, per-domain fusion, per-timestep gating, per-user control generation, or prompt lifecycle management over a stream. A plausible implication is that DPM is better understood as a functional design pattern than as a fixed layer type.
16 OR \16. Core mechanisms
One major design line treats DPM as a policy over prompt factors. Dynamic Prompting formalizes prompt tuning as a problem of choosing prompt position, length, and representation per instance or task, using lightweight policy networks and Gumbel-Softmax. The augmented input is formed as PRESERVED_PLACEHOLDER_16query16, and the framework argues that allowing prompts to “encompass” the input unlocks attention routes unavailable to purely prefix or postfix prompting (&&&16query16&&& In this formulation, the module is dynamic because it emits discrete structural decisions rather than only learning a static soft prompt.
A second line treats DPM as a fusion operator over heterogeneous prompt sources. In rehearsal-free lifelong person re-identification, the Dynamic Prompt Fusion module combines Invariant Prompts generated by BLIP with Person Knowledge Prompts adapted from CLIP-ReID. The fused dynamic prompt is produced by an encoder-decoder with cross-attention,
PRESERVED_PLACEHOLDER_16all:(\16^
where the fusion weights are implicit in cross-attention rather than fixed by a scalar mixing coefficient. The resulting text feature is aligned globally and locally with image features so that pedestrian images from different domains are drawn into a unified semantic space (&&&16all:(\16&&&
A third line treats DPM as an online memory-and-selection mechanism. DynaPrompt maintains a prompt buffer PRESERVED_PLACEHOLDER_16 OR \16^ and, for each test sample, computes two adaptive selection criteria relative to the initial prompt PRESERVED_PLACEHOLDER_16 OR \16: entropy-based confidence and probability-difference-based sensitivity. The selected set is
PRESERVED_PLACEHOLDER_16 OR \16^
where PRESERVED_PLACEHOLDER_16 OR \16^ contains prompts with entropy no worse than PRESERVED_PLACEHOLDER_16)16, and PRESERVED_PLACEHOLDER_16max_results16^ contains prompts whose probability difference is no smaller than that of PRESERVED_PLACEHOLDER_16sort_by16. Only prompts in PRESERVED_PLACEHOLDER_16submittedDate16^ are updated; the rest are frozen. If PRESERVED_PLACEHOLDER_16all:(\16query16, a fresh prompt is appended and inactive prompts can later be deleted (&&&16 OR \16&&&16)16
A fourth line treats DPM as a generator rather than a selector. In DRAPE, the module derives instruction-aware prompt queries from the instruction tokens, cross-attends them to visual patch features, and outputs continuous prompt tokens prepended to the frozen LLM. In diffusion model patching, the prompt generator is timestep-aware: a gating network uses the timestep embedding and block depth to form
PRESERVED_PLACEHOLDER_16all:(\16all:(\16^
which is then additively injected into each frozen DiT block (&&&16all:(\16query16&&&, &&&16 OR \16&&&16)16
16 OR \16. Architectural instantiations across domains
In vision-language and continual-learning settings, DPM often functions as a semantic anchor or anti-forgetting device. The Dynamic Textual Prompt framework for lifelong person re-identification uses CLIP image and text encoders, BLIP-generated attribute descriptions, a two-layer transformer encoder plus decoder with cross-attention for prompt fusion, a Text-Visual Feature Alignment module that partitions image and text features into head, upper-body, lower-body, and foot components, and a Learnable Knowledge Distillation module with learnable temperatures PRESERVED_PLACEHOLDER_16all:(\16 OR \16^ to balance plasticity and stability across domains (&&&16all:(\16&&& DRAPE adopts a different but related strategy in Multimodal Continual Instruction Tuning: one prompt generator is trained per task, the shared projector is protected by null-space gradient projection, and CLIP-based prototype routing selects the correct generator without task labels at inference (&&&16all:(\16query16&&&
In structured recognition, DPM frequently refines intermediate representations rather than only text prompts. UniCell uses dataset prompts and a category memory bank. Dataset prompt sequences are built by prepending PRESERVED_PLACEHOLDER_16all:(\16 OR \16^ learnable context tokens to dataset names, while category names form token-level memory banks. A visibility mask limits attention to categories present in the current dataset, after which masked cross-attention updates dataset prompts and a second cross-attention stage refines encoder features token-wise. The refined features feed both first-stage detection-classification heads and the deformable decoder (&&&16 OR \16&&&16)16 ProDA defines DPM at the node level over a spatio-temporal scene graph: an action specification, represented as a multi-hot vector, is combined with each node feature to generate weights over candidate prompt embeddings, producing prompted node features that guide downstream graph parsing and action disentanglement (&&&16all:(\16submittedDate16&&&
In scene-adaptive single-domain generalization, prompt guidance takes the form of gating rather than prompt-token insertion. The prompt-driven object-centric perception network uses hand-designed scene prompts such as “an image taken in {scene name},” encodes them with a frozen CLIP text encoder, fuses them with backbone features via Slot Attention, and derives spatial and channel masks through Gumbel-Softmax-relaxed binary gating. These masks are then applied inside backbone blocks by the Dynamic Selective Module, which filters both channels and spatial regions (&&&16 OR \16query16&&&16)16
In generative modeling, DPM is stage-aware. Diffusion Model Patching inserts prompts additively into the token sequence of every DiT block, keeping sequence length unchanged and preserving throughput. Prompts are distinct per block, zero-initialized for stability, and dynamically gated at each denoising step. Because the backbone remains frozen, the added parameters are confined to the prompts and the small gating network (&&&16 OR \16&&&16)16
In interactive systems, the term extends beyond neural prompt tensors. Dynamic Prompt Middleware instantiates DPM as Dynamic Prompt Refinement Controls: an Option Module analyzes the user prompt, conversation history, and session options to generate 16 OR \16–16 OR \16^ context-sensitive UI controls with 16 OR \16–16 OR \16^ options each, which are serialized into textual refinements for a reactive Chat Module. Inline controls are per-response; session controls persist across the interaction, and users can pin inline options or edit JSON directly (&&&16 OR \16 OR \16&&&16)16
16 OR \16. Optimization objectives and training regimes
The loss functions associated with DPMs are highly task-specific, but they typically combine a task loss with a prompt-specific regularizer, alignment loss, or routing constraint. In lifelong person re-identification, Stage-I freezes the CLIP image encoder and optimizes only the global supervised contrastive alignment
PRESERVED_PLACEHOLDER_16all:(\16 OR \16^
while Stage-II jointly optimizes
PRESERVED_PLACEHOLDER_16all:(\16 OR \16^
where PRESERVED_PLACEHOLDER_16all:(\16)16^ is cosine-distance-based local alignment and PRESERVED_PLACEHOLDER_16all:(\16max_results16^ uses learnable student and teacher temperatures to retain prior-stage knowledge (&&&16all:(\16&&&
DynaPrompt uses entropy minimization over augmentations and selected prompts: PRESERVED_PLACEHOLDER_16all:(\16sort_by16^ Only the top 16all:(\16query16% most confident augmentations are used to compute the loss, and only prompts in the selected set receive a gradient update. This objective is coupled tightly to the buffer maintenance policy: selective updates, appending when no prompt qualifies, and deleting the most inactive prompt when the buffer is full (&&&16 OR \16&&&16)16
UniCell follows DETR-style supervision with dataset-specific losses based on Hungarian matching, centroid regression, and Focal classification loss, summed across datasets with balancing weights PRESERVED_PLACEHOLDER_16all:(\16submittedDate16. DPM itself is trained end-to-end without extra prompt regularization because the visibility mask already enforces label-space alignment (&&&16 OR \16&&&16)16 In diffusion model patching, prompt parameters and gating parameters are optimized against the standard diffusion noise-prediction loss while the backbone is frozen, optionally augmented by importance and load-balancing terms to prevent prompt collapse (&&&16 OR \16&&&16)16
Other domains introduce additional structure. The object-centric SDG model adds sparsity-oriented bound losses on spatial and channel mask densities, with an annealing factor PRESERVED_PLACEHOLDER_16 OR \16query16, yielding
PRESERVED_PLACEHOLDER_16 OR \16all:(\16^
ProDA combines three BCE terms over disentangled outputs with a disentanglement loss and a reconstruction loss: PRESERVED_PLACEHOLDER_16 OR \16 OR \16^ By contrast, Dynamic Prompt Middleware explicitly does not propose a formal mathematical model; its behavior is specified procedurally through schema-driven option generation and deterministic serialization of control selections into prompt refinements (&&&16 OR \16query16&&&, &&&16all:(\16submittedDate16&&&, &&&16 OR \16 OR \16&&&16)16
16 OR \16. Empirical behavior and reported gains
The reported empirical gains indicate that DPMs are most valuable where static prompts or static networks fail under heterogeneity, sequential drift, or test-time uncertainty. In rehearsal-free lifelong person re-identification, the Dynamic Textual Prompt framework reports gains over rehearsal-based state of the art of 16all:(\16 OR \16.16submittedDate16% and 16all:(\16 OR \16.16sort_by16% on seen-dataset mAP and Rank-16all:(\16, and 16all:(\16max_results16.16 OR \16% and 16all:(\16max_results16.16submittedDate16 on unseen datasets across four orders. On Order-16 OR \16, for example, it reaches 16max_results16max_results16.16 OR \16^ mAP and 16sort_by16 OR \16.16 OR \16^ Rank-16all:(\16^ on seen datasets and 16max_results16query16.16max_results16^ mAP and 16)16sort_by16.16sort_by16^ Rank-16all:(\16^ on unseen datasets (&&&16all:(\16&&&
In test-time prompt tuning, DynaPrompt improves OoD mean accuracy from 16)16query16.16sort_by16all:(\16^ to 16)16all:(\16.16sort_by16all:(\16^ over TPT under handcrafted prompts on ImageNet shifts, and from 16)16 OR \16.16sort_by16 OR \16^ to 16)16 OR \16.16 OR \16all:(\16^ with pretrained CoOp prompts. Its most pointed result is the collapse analysis on ImageNet-A: Online TPT falls to 16)16.16submittedDate16)16 TPT reaches 16 OR \16 OR \16.16max_results16max_results16, Oracle 16 OR \16submittedDate16.16 OR \16sort_by16, and DynaPrompt 16 OR \16)16.16all:(\16max_results16, indicating that selective prompt updates plus buffer management reduce error accumulation while still exploiting stream information (&&&16 OR \16&&&16)16
UniCell reports state-of-the-art detection F-score and average classification F-score on CoNSeP, MoNuSAC, Lizard, and OCELOT, with detection gains over the second-best method of +16 OR \16.16max_results16%, +16all:(\16.16submittedDate16 +16all:(\16.16 OR \16%, and +16all:(\16.16 OR \16%, and classification gains of +16sort_by16.16 OR \16%, +16 OR \16.16query16%, +16all:(\16 OR \16.16 OR \16%, and +16 OR \16.16)16%, respectively. Ablation shows that removing DPM lowers PRESERVED_PLACEHOLDER_16 OR \16 OR \16^ by 16all:(\16.16sort_by16 and PRESERVED_PLACEHOLDER_16 OR \16 OR \16^ by 16 OR \16.16sort_by16% (&&&16 OR \16&&&16)16
In diffusion, DMP improves a converged DiT-L/16 OR \16^ on FFHQ from FID-16 OR \16query16K 16)16.16 OR \16)16^ to 16 OR \16.16)16all:(\16^ after 16 OR \16query16K further iterations, corresponding to a 16all:(\16query16.16 OR \16sort_by16% improvement with only a 16all:(\16.16 OR \16 OR \16% parameter increase. On MS-COCO 16 OR \16 OR \16)16×16 OR \16 OR \16)16, DiT-B/16 OR \16^ improves from 16max_results16.16 OR \16 OR \16^ to 16max_results16.16all:(\16 OR \16, and on ImageNet 16 OR \16 OR \16)16×16 OR \16 OR \16)16, DiT-XL/16 OR \16^ improves from 16 OR \16.16 OR \16submittedDate16^ to 16 OR \16.16 OR \16 OR \16^ (&&&16 OR \16&&&16)16
DPM-style conditioning also improves structured and object-centric recognition. The prompt-driven SDG network reaches 16max_results16all:(\16.16all:(\16 OR \16% average accuracy on PACS single-source SDG versus 16)16submittedDate16.16sort_by16)16 for Meta-Casual, and improves Day→Night Clear detection mAP from 16 OR \16)16.16submittedDate16^ to 16 OR \16sort_by16.16 OR \16^ over CLIP-Gap. ProDA reaches 16max_results16all:(\16.16all:(\16^ mAP on Charades with RGB+SSG input, exceeding OR16 OR \16G+NL at 16)16max_results16.16 OR \16, and its DPM outperforms a SimplePrompt variant by +16 OR \16.16query16 OR \16^ on PRESERVED_PLACEHOLDER_16 OR \16 OR \16^ mAP (&&&16 OR \16query16&&&, &&&16all:(\16submittedDate16&&&
In multimodal continual instruction tuning, DRAPE achieves 16)16max_results16.16 OR \16sort_by16^ average accuracy on CoIN, surpassing ProgLoRA’s 16 OR \16submittedDate16.16query16submittedDate16^ by +16sort_by16.16 OR \16submittedDate16, and 16)16submittedDate16.16 OR \16all:(\16^ on UCIT, surpassing SEFE’s 16)16)16.16 OR \16 OR \16^ by +16 OR \16.16sort_by16max_results16. The ablation “w/o Cross-Modal Attn.” drops average accuracy from 16)16max_results16.16 OR \16sort_by16^ to 16)16)16.16 OR \16all:(\16, and “w/o Null-Space Proj.” drops it to 16)16 OR \16.16sort_by16)16, indicating that both prompt synthesis and projector protection are operationally important (&&&16all:(\16query16&&&
Human-centered results show a parallel pattern. In Dynamic Prompt Middleware, participants preferred Dynamic PRC over Static PRC, rated it higher for controlling explanations (mean 16)16.16 OR \16 OR \16^ versus 16 OR \16.16sort_by16all:(\16; PRESERVED_PLACEHOLDER_16 OR \16)16, PRESERVED_PLACEHOLDER_16 OR \16max_results1616)16 and reported less need for additional control (mean 16 OR \16.16max_results16 OR \16^ versus 16 OR \16.16 OR \16 OR \16; PRESERVED_PLACEHOLDER_16 OR \16sort_by16, PRESERVED_PLACEHOLDER_16 OR \16submittedDate1616)16 though reasoning about the exact effects of generated controls remained difficult (&&&16 OR \16 OR \16&&&16)16
16)16. Limitations, misconceptions, and future directions
A recurrent limitation is prompt quality. In lifelong ReID, BLIP-generated invariant prompts can be noisy or incomplete, and very similar individuals may remain hard to distinguish unless Person Knowledge Prompts compensate; excessive emphasis on PKP can in turn overfit current-domain bias (&&&16all:(\16&&& In single-domain generalization, effectiveness depends on suitable scene prompts and on CLIP’s representational alignment with those scenes, while latency and parameter overhead from the gating modules are not reported (&&&16 OR \16query16&&&16)16 In ProDA, distractor-injected action prompts improve robustness but exhibit non-monotonic behavior as prompt noise changes, and imperfect scene graph construction can misalign prompt guidance with actual video content (&&&16all:(\16submittedDate16&&&
Another limitation is stability under adaptation. DynaPrompt reports that removing appending and deletion causes severe collapse, reducing mean accuracy to 16 OR \16 OR \16.16)16 OR \16, and its performance still shows some sensitivity to sample order and buffer size. Dynamic Prompting notes that joint optimization of multiple prompt factors can be harder than optimizing a single factor, and that the straight-through estimator introduces stochasticity that benefits from temperature annealing (&&&16 OR \16&&&, &&&16query16&&& In diffusion model patching, hard prompt selection and attention-based gating underperform soft linear gating, while prepending prompts destabilizes training relative to additive injection (&&&16 OR \16&&&16)16
A common misconception is that DPM necessarily means text prompt tuning for a LLM. The surveyed literature contradicts this directly. DPM may refine detector features with dataset/category prompts, generate action-conditioned node embeddings from multi-hot labels, add prompt tensors to diffusion tokens, or render interactive UI controls that are later serialized into text (&&&16 OR \16&&&, &&&16all:(\16submittedDate16&&&, &&&16 OR \16 OR \16&&&16)16 This suggests that the core invariant of DPM is adaptive conditioning, not any particular prompt substrate.
Open directions in the literature follow this broader view. The ReID work highlights robust captioning under occlusion, privacy-preserving prompt generation, adaptive part parsing, and prompt uncertainty modeling; DynaPrompt proposes confidence-aware extensions and multimodal prompt settings; DRAPE points toward richer cross-modal fusion, soft generator blending, and more robust routing; UniCell raises class-frequency-aware weighting and prompt sparsification; middleware work emphasizes response diffs, provenance-aware controls, and direct manipulation of generated options (&&&16all:(\16&&&, &&&16 OR \16&&&, &&&16all:(\16query16&&&, &&&16 OR \16&&&, &&&16 OR \16 OR \16&&&16)16 Across these directions, DPM research is moving away from static prompt parameters and toward adaptive control systems that couple prompts tightly to the structure of data, tasks, and interaction contexts.