---
title: Multi-dimensional Dynamic Prompt Routing
url: https://www.emergentmind.com/topics/multi-dimensional-dynamic-prompt-routing-mdpr
type: topic
---

# Multi-dimensional Dynamic Prompt Routing

Multi-dimensional Dynamic Prompt Routing (MDPR) is a prompt-based adaptation framework in which class semantics are represented as a structured, multi-dimensional prompt library and the prompt pathway is selected dynamically according to the input instance rather than fixed at class level. In the formulation introduced for long-tailed vision-language model fine-tuning, MDPR is a plug-and-play framework for models such as CLIP, designed for settings where head classes dominate optimization and tail classes are under-learned. Its central claim is that long-tailed tuning of pre-trained vision-language models suffers from a **double imbalance**: an **implicit imbalance in VLM pre-training** and an **explicit imbalance in downstream data**. MDPR addresses this by constructing an LLM-generated class knowledge base spanning five visual-semantic dimensions, then performing online instance-aware routing and logits fusion during fine-tuning and inference [2508.15688].

## 1. Problem formulation and motivation

The motivating problem is long-tailed recognition for pre-trained vision-language models. The framework is explicitly designed for the regime in which some classes receive many optimization updates while tail classes have limited samples and weaker adaptation. The paper further argues that this downstream imbalance compounds biases already present in pre-training: some visual concepts are semantically represented better than others before fine-tuning begins. In that sense, MDPR targets not only class-frequency skew in the target dataset, but also inherited representational asymmetry in the base VLM [2508.15688].

This diagnosis is positioned against prior prompt-based VLM tuning methods such as **CoOp** and **MaPLe**. Those methods learn prompts, but the prompts are described as **static** or only mildly adaptive, **single-view** in semantics, and not explicitly designed to counter class confusion or tail-class weakness. LLM-enhanced tuning methods are likewise described as limited when they generate only one refined description per class, rely on costly sample synthesis, or do not address bias accumulation during fine-tuning. MDPR therefore redefines the prompt as a routed semantic resource rather than a single learned token sequence.

A common misconception is that MDPR merely adds more text to the class description. The ablation evidence argues otherwise: the reported gains depend on **structured semantic diversity** plus **routing constraints**, not on textual augmentation in the abstract. This suggests that the essential object is a controlled semantic library whose use is conditioned by image content and regularized against instability, rather than a larger prompt inventory alone [2508.15688].

## 2. Two-stage architecture and the five-dimensional semantic library

MDPR is organized into two stages. **Stage A** is **offline multi-dimensional prompt construction**. For each class, an LLM generates prompts across multiple semantic dimensions, and CLIP’s text encoder converts them into a class-specific prompt bank or semantic library. **Stage B** is **online dynamic prompt routing**. During fine-tuning, the model encodes the input image, compares the image feature with the class prompt library, routes attention over the class’s multi-dimensional prompts using a class-specific multi-head attention module, produces dynamic semantic features, predicts class logits from those features, and fuses them with the base VLM logits for final prediction. The framework is explicitly described as attachable to existing prompt-tuning backbones, especially **CoOp** and **MaPLe**, without changing the backbone [2508.15688].

The class knowledge base spans five visual-semantic dimensions.

| Dimension | Description | Role |
|---|---|---|
| General Appearance (GA) | Typical shape, color, size, and broad visual attributes | Core visual identity |
| Fine-grained Appearance (FA) | Local details, texture, or parts | Distinguishes similar classes |
| Functionality (FT) | Purpose, use, or role | Adds use-oriented semantics |
| Contextual Information (CI) | Surroundings, scene context, or co-occurring objects | Adds scene priors |
| Differential Features (DF) | Features distinguishing a class from its most confusing counterpart | Reduces class confusion |

The fifth dimension, **Differential Features**, is especially distinctive. MDPR first builds a confusion matrix using CLIP zero-shot predictions on the training set, identifies the most frequently confused class for each class, and then asks the LLM to generate contrastive descriptions that highlight the differences. The stated purpose is to counter inherited bias and improve separation among hard-to-distinguish categories. The result is a class prompt set over \(V_{dim}=5\) dimensions rather than a single template or one refined class description [2508.15688].

This design gives MDPR its “multi-dimensional” character in the strict sense used by the original framework: semantic knowledge is explicitly decomposed into multiple class attributes, and routing operates over that decomposition rather than over a flat prompt pool.

## 3. Dynamic routing, prior alignment, and fused prediction

Each prompt \(\text{p}_{c,v}\) is encoded by CLIP’s text encoder as
\[
\mathbf{f}^{cv}_{p} = E_t(\text{p}_{c,v}).
\]
These prompt features are stacked into a prompt tensor of shape \(\mathbb{R}^{C \times V_{dim} \times d}\), and the class-level average semantic representation is computed by averaging over the five dimensions. MDPR also constructs a fixed prior importance matrix \(\mathbf{M}\in\mathbb{R}^{C\times V_{dim}}\), where each entry measures the similarity between a dimension-specific prompt embedding and a generic class template such as “a photo of a [class name].” This matrix is not learned; it is precomputed and fixed, and acts as a prior over which semantic dimensions are generally useful for each class [2508.15688].

For an input image \(x_b\), the CLIP image encoder yields \(\mathbf{f}_{ib} = E_i(x_b)\). A **class-specific multi-head attention (C-MHA)** module then uses the image feature as query and the class prompt features as key and value:
\[
\mathbf{f}^{c}_{rb}, \mathbf{W}^{c}_{r} = \operatorname{C\text{-}MHA}(\mathbf{f}_{ib},\mathbf{f}^{c}_{p},\mathbf{f}^{c}_{p}).
\]
Here, \(\mathbf{W}^{c}_{r}\) are routing weights and \(\mathbf{f}^{c}_{rb}\) is the routed semantic feature for that class and image. The routing is therefore **instance-aware**: even within one class, different images can emphasize different semantic dimensions. The paper’s own example is that one image of a bird may depend more on contextual cues, while another may depend more on fine-grained appearance [2508.15688].

MDPR does not use the routed branch alone for classification. It combines the base class-name-driven logits \(\hat{\mathbf{y}}_{cb}\) with routing-based logits \(\hat{\mathbf{y}}_{rb}\), where
\[
\hat{y}_{rb} = s \cdot \langle \mathbf{f}_{rb}, \mathbf{f}_{ib} \rangle
\]
and \(s\) is a learnable temperature scalar. Final inference uses
\[
\hat{\mathbf{y}}_{fuse} = (1-\beta)\cdot \hat{\mathbf{y}}_{cb} + \beta\cdot \hat{\mathbf{y}}_{rb},
\]
with \(\beta \in [0,1]\), typically set to \(0.5\). The intended effect is a regularizing ensemble: the base branch contributes general, globally learned class discrimination, while the routed branch contributes instance-specific semantic refinement. In long-tailed settings, where tail-class predictions can be unstable when only a few samples drive gradients, this fusion is presented as a stability mechanism rather than merely a score average [2508.15688].

## 4. Optimization, regularization, and offline LLM prompt generation

MDPR uses a multi-task objective
\[
\mathcal{L}_{total} = \lambda_{base}\mathcal{L}_{base} + \lambda_{sem}\mathcal{L}_{sem} + \mathcal{L}_{reg},
\]
with
\[
\mathcal{L}_{reg} = \lambda_{pa}\mathcal{L}_{pa} + \lambda_{ka}\mathcal{L}_{ka}.
\]
The base loss \(\mathcal{L}_{base}\) is the main classification loss of the underlying prompt-learning framework. The semantic branch is supervised with **Compensated Cross Entropy** through
\[
\mathcal{L}_{sem} = \text{CLA}(\mathbf{z}_{sem}, \mathbf{y}),
\]
which is explicitly chosen for imbalanced data. Two regularizers stabilize the routed semantics. The **prior alignment loss** aligns learned routing weights with the fixed prior importance matrix, encouraging routing to respect class-specific semantic priors rather than sample-frequency skew. The **knowledge alignment loss** uses a KL term to align the instance-aware routed semantic feature with the class’s averaged semantic feature after a learnable linear projection. In the paper’s interpretation, this prevents the routed representation from drifting too far from the class’s global semantic center [2508.15688].

The trainable parameters are limited to the prompt-learning parameters of **CoOp** or **MaPLe**, the **C-MHA** parameters, and the projection layer used for KL alignment. The prompt embeddings, averaged semantic features, and prior matrix \(\mathbf{M}\) remain fixed. The paper also notes warm-up for \(\lambda_{sem}\) and \(\lambda_{ka}\) over the first **5 epochs** to stabilize training. This division between fixed semantic memory and lightweight trainable routing is part of the framework’s parameter-efficient character.

The LLM is used **offline** to generate the semantic prompt pool. The appendix reports explicit prompt templates for visual features, functional use, contextual scene, differential comparison, and fine-grained attributes. The paper evaluates **Qwen2.5**, **LLaMa4**, and **DeepSeek-V3** as prompt generators, selecting **Qwen2.5** as the default on the basis of semantic similarity statistics and qualitative prompt quality. On **CIFAR-100**, the reported similarity statistics are **Qwen2.5: mean 0.8371, std 0.0370, median 0.8452**; **LLaMa4: mean 0.8360, std 0.0371, median 0.8457**; and **DeepSeek-V3: mean 0.8354, std 0.0373, median 0.8438**. The differences are stated to be small, but Qwen2.5 is selected for overall balance [2508.15688].

## 5. Benchmarks, results, ablations, and efficiency

MDPR is evaluated on **CIFAR-100-LT**, **ImageNet-LT**, and **Places-LT**, with **overall accuracy**, **many-shot accuracy**, **medium-shot accuracy**, and **few-shot accuracy** as reported metrics. The split follows the common protocol: **Many: >100 images**, **Medium: 20–100 images**, **Few: <20 images**. Baselines include **Zero-shot CLIP**, **CoOp**, **CoCoOp**, **MaPLe**, **MaPLe + LA**, **PLOT++**, **LASP**, **TextRefiner**, **CPRL**, and **Candle** [2508.15688].

| Benchmark | Selected result | Comparison reported |
|---|---|---|
| CIFAR-100-LT, IR=100 | **Ours (MaPLe)**: 79.25 overall, 87.60 many, 81.26 medium, 67.17 few | **MaPLe**: 74.09 overall, 88.14 many, 73.46 medium, 58.43 few |
| CIFAR-100-LT, IR=50 | **Ours (MaPLe)**: 81.38 overall, 88.68 many, 76.90 medium, 74.94 few | **MaPLe**: 77.09 overall, 87.34 many, 71.98 medium, 65.39 few |
| ImageNet-LT / Places-LT | **Ours (CoOp)**: 74.57 / 48.89 overall; **Ours (MaPLe)**: 75.57 / 50.94 overall | Baselines are notably lower on few-shot classes; scene-centric semantics matter strongly on Places-LT |

The reported pattern is consistent across datasets: the strongest gains appear on **tail classes**. On **ImageNet-LT**, **Ours (CoOp)** obtains **74.57 overall, 77.67 many, 73.42 medium, 69.87 few**, and **Ours (MaPLe)** obtains **75.57 overall, 79.42 many, 74.02 medium, 70.04 few**. On **Places-LT**, **Ours (CoOp)** obtains **48.89 overall, 49.45 many, 48.72 medium, 47.96 few**, and **Ours (MaPLe)** obtains **50.94 overall, 50.32 many, 51.42 medium, 50.99 few**. The paper’s summary is that MDPR achieves **state-of-the-art or highly competitive performance**, with its strongest improvements consistently appearing in tail classes [2508.15688].

The ablation studies are used to distinguish the contribution of semantic diversity from the contribution of routing regularization. The compared variants are **zero-shot CLIP**, **base MaPLe**, **MaPLe + semantic prompts**, and **full MDPR = base + semantic prompts + regularization**. The results show progressive improvement: adding semantic prompts gives a large tail-class jump, while adding regularization further improves stability and balanced accuracy. In the dimension ablation on **Places-LT**, removing any of **GA**, **FA**, **FT**, **CI**, or **DF** reduces accuracy, with **Differential Features** causing the largest overall decline and **Contextual Information** also producing a significant drop. The paper’s interpretation is that **DF** is particularly important for confusion reduction, **CI** is important because scene context matters in Places-LT, **GA** provides core object identity, and **FA** and **FT** are complementary refinements [2508.15688].

MDPR is presented as lightweight. The reported overhead is about **1.1M trainable parameters**, less than **0.74%** of **CLIP ViT-B/16**. On **ImageNet-LT**, training time is reported as **1115 s/epoch** for **CoOp**, **1229 s/epoch** for **CoOp + MDPR**, **1361 s/epoch** for **MaPLe**, and **1375 s/epoch** for **MaPLe + MDPR**. The paper emphasizes that this is much cheaper than sample-generation approaches requiring diffusion models or other generative pipelines [2508.15688].

## 6. Broader interpretations and related prompt-routing frameworks

Within the literature provided here, MDPR is both a specific framework and a broader interpretive template. The explicit framework is the long-tailed VLM method described above. At the same time, later systems in other subfields can plausibly be viewed as instantiating the same general principle: prompt adaptation is distributed across more than one conditioning axis, and routing combines global priors with instance- or domain-specific signals.

In **DRAPE**, proposed for **Multimodal Continual Instruction Tuning**, routing is argued to be too coarse when it operates only at the task level. The method uses a task-specific cross-modal prompt generator, but the generator synthesizes **continuous instance-specific soft prompts** from the current query-image pair rather than selecting prompts from a fixed pool. Its relevant dimensions are described as **task**, **instance**, **modality**, **prototype / semantic**, and **optimization/subspace**. The paper explicitly states that DRAPE is highly relevant to MDPR and is “a strong example of multi-dimensional dynamic prompt routing,” while also noting that its own emphasis is on **task-specific generators + instance-specific prompts** rather than on the MDPR name itself [2605.10765].

In **DyGFM**, for multi-domain dynamic graph foundation models, the connection is made through **semantic-temporal decoupling**, **divergence-aware cross-domain routing** over source experts, and **divergence-conditioned prompt generation**. The routing dimensions are **semantic divergence**, **temporal divergence**, **target semantic traits**, and **target temporal traits**. The paper states that DyGFM can be viewed as a form of MDPR because prompts are dynamic, conditioned on multiple signals, and routed across source-domain experts based on domain divergence, even though the authors do not frame the method by that name [2605.13540].

In **FineRouter**, prompt routing is transferred from model adaptation to model selection over a pool of frontier LLMs. The system combines **semantic task structure** and **model-preference structure** in graph-based latent task discovery, then combines **task-level prior quality values** with **prompt-specific quality estimates** in a task-aware mixture-of-experts router. The paper explicitly characterizes this as a strong instance of what one could call MDPR, because routing is dynamic at inference time and depends on several interacting evidence sources rather than on a monolithic classifier or a manually defined taxonomy [2603.19415].

Taken together, these works suggest a broader research pattern. In the narrow sense, MDPR denotes the five-dimension, class-semantic routing framework for long-tailed VLM tuning. In a wider, interpretive sense, the term can describe prompt-routing systems in which adaptation depends simultaneously on multiple semantic, structural, or optimization dimensions, and in which routing decisions are stabilized by priors, fusion, or structured regularization. The common denominator is not simply dynamic prompting, but dynamic prompting under explicitly factored conditioning signals.

Source: https://www.emergentmind.com/topics/multi-dimensional-dynamic-prompt-routing-mdpr