Multi-dimensional Dynamic Prompt Routing

• Multi-dimensional Dynamic Prompt Routing (MDPR) is a framework that decomposes class semantics into five visual dimensions for dynamic, instance-aware prompt selection.
• It constructs an LLM-generated semantic library and employs a two-stage architecture to fuse routed instance features with base VLM logits.
• MDPR addresses double imbalance from pre-training and downstream data, achieving improved tail class recognition with minimal trainable parameters.

Multi-dimensional Dynamic Prompt Routing (MDPR) is a prompt-based adaptation framework in which class semantics are represented as a structured, multi-dimensional prompt library and the prompt pathway is selected dynamically according to the input instance rather than fixed at class level. In the formulation introduced for long-tailed vision-LLM fine-tuning, MDPR is a plug-and-play framework for models such as CLIP, designed for settings where head classes dominate optimization and tail classes are under-learned. Its central claim is that long-tailed tuning of pre-trained vision-LLMs suffers from a double imbalance: an implicit imbalance in VLM pre-training and an explicit imbalance in downstream data. MDPR addresses this by constructing an LLM-generated class knowledge base spanning five visual-semantic dimensions, then performing online instance-aware routing and logits fusion during fine-tuning and inference (Jia et al., 21 Aug 2025).

1. Problem formulation and motivation

The motivating problem is long-tailed recognition for pre-trained vision-LLMs. The framework is explicitly designed for the regime in which some classes receive many optimization updates while tail classes have limited samples and weaker adaptation. The paper further argues that this downstream imbalance compounds biases already present in pre-training: some visual concepts are semantically represented better than others before fine-tuning begins. In that sense, MDPR targets not only class-frequency skew in the target dataset, but also inherited representational asymmetry in the base VLM (Jia et al., 21 Aug 2025).

This diagnosis is positioned against prior prompt-based VLM tuning methods such as CoOp and MaPLe. Those methods learn prompts, but the prompts are described as static or only mildly adaptive, single-view in semantics, and not explicitly designed to counter class confusion or tail-class weakness. LLM-enhanced tuning methods are likewise described as limited when they generate only one refined description per class, rely on costly sample synthesis, or do not address bias accumulation during fine-tuning. MDPR therefore redefines the prompt as a routed semantic resource rather than a single learned token sequence.

A common misconception is that MDPR merely adds more text to the class description. The ablation evidence argues otherwise: the reported gains depend on structured semantic diversity plus routing constraints, not on textual augmentation in the abstract. This suggests that the essential object is a controlled semantic library whose use is conditioned by image content and regularized against instability, rather than a larger prompt inventory alone (Jia et al., 21 Aug 2025).

2. Two-stage architecture and the five-dimensional semantic library

MDPR is organized into two stages. Stage A is offline multi-dimensional prompt construction. For each class, an LLM generates prompts across multiple semantic dimensions, and CLIP’s text encoder converts them into a class-specific prompt bank or semantic library. Stage B is online dynamic prompt routing. During fine-tuning, the model encodes the input image, compares the image feature with the class prompt library, routes attention over the class’s multi-dimensional prompts using a class-specific multi-head attention module, produces dynamic semantic features, predicts class logits from those features, and fuses them with the base VLM logits for final prediction. The framework is explicitly described as attachable to existing prompt-tuning backbones, especially CoOp and MaPLe, without changing the backbone (Jia et al., 21 Aug 2025).

The class knowledge base spans five visual-semantic dimensions.

Dimension	Description	Role
General Appearance (GA)	Typical shape, color, size, and broad visual attributes	Core visual identity
Fine-grained Appearance (FA)	Local details, texture, or parts	Distinguishes similar classes
Functionality (FT)	Purpose, use, or role	Adds use-oriented semantics
Contextual Information (CI)	Surroundings, scene context, or co-occurring objects	Adds scene priors
Differential Features (DF)	Features distinguishing a class from its most confusing counterpart	Reduces class confusion

The fifth dimension, Differential Features, is especially distinctive. MDPR first builds a confusion matrix using CLIP zero-shot predictions on the training set, identifies the most frequently confused class for each class, and then asks the LLM to generate contrastive descriptions that highlight the differences. The stated purpose is to counter inherited bias and improve separation among hard-to-distinguish categories. The result is a class prompt set over $V_{dim}=5$ dimensions rather than a single template or one refined class description (Jia et al., 21 Aug 2025).

This design gives MDPR its “multi-dimensional” character in the strict sense used by the original framework: semantic knowledge is explicitly decomposed into multiple class attributes, and routing operates over that decomposition rather than over a flat prompt pool.

3. Dynamic routing, prior alignment, and fused prediction

Each prompt $\text{p}_{c,v}$ is encoded by CLIP’s text encoder as

$\mathbf{f}^{cv}_{p} = E_t(\text{p}_{c,v}).$

These prompt features are stacked into a prompt tensor of shape $\mathbb{R}^{C \times V_{dim} \times d}$, and the class-level average semantic representation is computed by averaging over the five dimensions. MDPR also constructs a fixed prior importance matrix $\mathbf{M}\in\mathbb{R}^{C\times V_{dim}}$, where each entry measures the similarity between a dimension-specific prompt embedding and a generic class template such as “a photo of a [class name].” This matrix is not learned; it is precomputed and fixed, and acts as a prior over which semantic dimensions are generally useful for each class (Jia et al., 21 Aug 2025).

For an input image $x_b$, the CLIP image encoder yields $\mathbf{f}_{ib} = E_i(x_b)$. A class-specific multi-head attention (C-MHA) module then uses the image feature as query and the class prompt features as key and value: $$\mathbf{f}^{c}_{rb}, \mathbf{W}^{c}_{r} = \operatorname{C\text{-}MHA}(\mathbf{f}_{ib},\mathbf{f}^{c}_{p},\mathbf{f}^{c}_{p}).$$ Here, $\mathbf{W}^{c}_{r}$ are routing weights and $\mathbf{f}^{c}_{rb}$ is the routed semantic feature for that class and image. The routing is therefore instance-aware: even within one class, different images can emphasize different semantic dimensions. The paper’s own example is that one image of a bird may depend more on contextual cues, while another may depend more on fine-grained appearance (Jia et al., 21 Aug 2025).

MDPR does not use the routed branch alone for classification. It combines the base class-name-driven logits $\text{p}_{c,v}$0 with routing-based logits $\text{p}_{c,v}$1, where

$\text{p}_{c,v}$2

and $\text{p}_{c,v}$3 is a learnable temperature scalar. Final inference uses

$\text{p}_{c,v}$4

with $\text{p}_{c,v}$5, typically set to $\text{p}_{c,v}$6. The intended effect is a regularizing ensemble: the base branch contributes general, globally learned class discrimination, while the routed branch contributes instance-specific semantic refinement. In long-tailed settings, where tail-class predictions can be unstable when only a few samples drive gradients, this fusion is presented as a stability mechanism rather than merely a score average (Jia et al., 21 Aug 2025).

4. Optimization, regularization, and offline LLM prompt generation

MDPR uses a multi-task objective

$\text{p}_{c,v}$7

with

$\text{p}_{c,v}$8

The base loss $\text{p}_{c,v}$9 is the main classification loss of the underlying prompt-learning framework. The semantic branch is supervised with Compensated Cross Entropy through

$\mathbf{f}^{cv}_{p} = E_t(\text{p}_{c,v}).$0

which is explicitly chosen for imbalanced data. Two regularizers stabilize the routed semantics. The prior alignment loss aligns learned routing weights with the fixed prior importance matrix, encouraging routing to respect class-specific semantic priors rather than sample-frequency skew. The knowledge alignment loss uses a KL term to align the instance-aware routed semantic feature with the class’s averaged semantic feature after a learnable linear projection. In the paper’s interpretation, this prevents the routed representation from drifting too far from the class’s global semantic center (Jia et al., 21 Aug 2025).

The trainable parameters are limited to the prompt-learning parameters of CoOp or MaPLe, the C-MHA parameters, and the projection layer used for KL alignment. The prompt embeddings, averaged semantic features, and prior matrix $\mathbf{f}^{cv}_{p} = E_t(\text{p}_{c,v}).$1 remain fixed. The paper also notes warm-up for $\mathbf{f}^{cv}_{p} = E_t(\text{p}_{c,v}).$2 and $\mathbf{f}^{cv}_{p} = E_t(\text{p}_{c,v}).$3 over the first 5 epochs to stabilize training. This division between fixed semantic memory and lightweight trainable routing is part of the framework’s parameter-efficient character.

The LLM is used offline to generate the semantic prompt pool. The appendix reports explicit prompt templates for visual features, functional use, contextual scene, differential comparison, and fine-grained attributes. The paper evaluates Qwen2.5, LLaMa4, and DeepSeek-V3 as prompt generators, selecting Qwen2.5 as the default on the basis of semantic similarity statistics and qualitative prompt quality. On CIFAR-100, the reported similarity statistics are Qwen2.5: mean 0.8371, std 0.0370, median 0.8452; LLaMa4: mean 0.8360, std 0.0371, median 0.8457; and DeepSeek-V3: mean 0.8354, std 0.0373, median 0.8438. The differences are stated to be small, but Qwen2.5 is selected for overall balance (Jia et al., 21 Aug 2025).

5. Benchmarks, results, ablations, and efficiency

MDPR is evaluated on CIFAR-100-LT, ImageNet-LT, and Places-LT, with overall accuracy, many-shot accuracy, medium-shot accuracy, and few-shot accuracy as reported metrics. The split follows the common protocol: Many: >100 images, Medium: 20–100 images, Few: <20 images. Baselines include Zero-shot CLIP, CoOp, CoCoOp, MaPLe, MaPLe + LA, PLOT++, LASP, TextRefiner, CPRL, and Candle (Jia et al., 21 Aug 2025).

Benchmark	Selected result	Comparison reported
CIFAR-100-LT, IR=100	Ours (MaPLe): 79.25 overall, 87.60 many, 81.26 medium, 67.17 few	MaPLe: 74.09 overall, 88.14 many, 73.46 medium, 58.43 few
CIFAR-100-LT, IR=50	Ours (MaPLe): 81.38 overall, 88.68 many, 76.90 medium, 74.94 few	MaPLe: 77.09 overall, 87.34 many, 71.98 medium, 65.39 few
ImageNet-LT / Places-LT	Ours (CoOp): 74.57 / 48.89 overall; Ours (MaPLe): 75.57 / 50.94 overall	Baselines are notably lower on few-shot classes; scene-centric semantics matter strongly on Places-LT

The reported pattern is consistent across datasets: the strongest gains appear on tail classes. On ImageNet-LT, Ours (CoOp) obtains 74.57 overall, 77.67 many, 73.42 medium, 69.87 few, and Ours (MaPLe) obtains 75.57 overall, 79.42 many, 74.02 medium, 70.04 few. On Places-LT, Ours (CoOp) obtains 48.89 overall, 49.45 many, 48.72 medium, 47.96 few, and Ours (MaPLe) obtains 50.94 overall, 50.32 many, 51.42 medium, 50.99 few. The paper’s summary is that MDPR achieves state-of-the-art or highly competitive performance, with its strongest improvements consistently appearing in tail classes (Jia et al., 21 Aug 2025).

The ablation studies are used to distinguish the contribution of semantic diversity from the contribution of routing regularization. The compared variants are zero-shot CLIP, base MaPLe, MaPLe + semantic prompts, and full MDPR = base + semantic prompts + regularization. The results show progressive improvement: adding semantic prompts gives a large tail-class jump, while adding regularization further improves stability and balanced accuracy. In the dimension ablation on Places-LT, removing any of GA, FA, FT, CI, or DF reduces accuracy, with Differential Features causing the largest overall decline and Contextual Information also producing a significant drop. The paper’s interpretation is that DF is particularly important for confusion reduction, CI is important because scene context matters in Places-LT, GA provides core object identity, and FA and FT are complementary refinements (Jia et al., 21 Aug 2025).

MDPR is presented as lightweight. The reported overhead is about 1.1M trainable parameters, less than 0.74% of CLIP ViT-B/16. On ImageNet-LT, training time is reported as 1115 s/epoch for CoOp, 1229 s/epoch for CoOp + MDPR, 1361 s/epoch for MaPLe, and 1375 s/epoch for MaPLe + MDPR. The paper emphasizes that this is much cheaper than sample-generation approaches requiring diffusion models or other generative pipelines (Jia et al., 21 Aug 2025).

6. Broader interpretations and related prompt-routing frameworks

Within the literature provided here, MDPR is both a specific framework and a broader interpretive template. The explicit framework is the long-tailed VLM method described above. At the same time, later systems in other subfields can plausibly be viewed as instantiating the same general principle: prompt adaptation is distributed across more than one conditioning axis, and routing combines global priors with instance- or domain-specific signals.

In DRAPE, proposed for Multimodal Continual Instruction Tuning, routing is argued to be too coarse when it operates only at the task level. The method uses a task-specific cross-modal prompt generator, but the generator synthesizes continuous instance-specific soft prompts from the current query-image pair rather than selecting prompts from a fixed pool. Its relevant dimensions are described as task, instance, modality, prototype / semantic, and optimization/subspace. The paper explicitly states that DRAPE is highly relevant to MDPR and is “a strong example of multi-dimensional dynamic prompt routing,” while also noting that its own emphasis is on task-specific generators + instance-specific prompts rather than on the MDPR name itself (Hu et al., 11 May 2026).

In DyGFM, for multi-domain dynamic graph foundation models, the connection is made through semantic-temporal decoupling, divergence-aware cross-domain routing over source experts, and divergence-conditioned prompt generation. The routing dimensions are semantic divergence, temporal divergence, target semantic traits, and target temporal traits. The paper states that DyGFM can be viewed as a form of MDPR because prompts are dynamic, conditioned on multiple signals, and routed across source-domain experts based on domain divergence, even though the authors do not frame the method by that name (Yuan et al., 13 May 2026).

In FineRouter, prompt routing is transferred from model adaptation to model selection over a pool of frontier LLMs. The system combines semantic task structure and model-preference structure in graph-based latent task discovery, then combines task-level prior quality values with prompt-specific quality estimates in a task-aware mixture-of-experts router. The paper explicitly characterizes this as a strong instance of what one could call MDPR, because routing is dynamic at inference time and depends on several interacting evidence sources rather than on a monolithic classifier or a manually defined taxonomy (Zhang et al., 19 Mar 2026).

Taken together, these works suggest a broader research pattern. In the narrow sense, MDPR denotes the five-dimension, class-semantic routing framework for long-tailed VLM tuning. In a wider, interpretive sense, the term can describe prompt-routing systems in which adaptation depends simultaneously on multiple semantic, structural, or optimization dimensions, and in which routing decisions are stabilized by priors, fusion, or structured regularization. The common denominator is not simply dynamic prompting, but dynamic prompting under explicitly factored conditioning signals.