---
title: Iterative Label Mapping for Visual Prompting
url: https://www.emergentmind.com/topics/iterative-label-mapping-ilm-vp
type: topic
---

# Iterative Label Mapping for Visual Prompting

Iterative Label Mapping-based Visual Prompting (ILM-VP) is a framework designed to enhance the effectiveness of visual prompting (VP) for transfer learning in vision tasks. Visual prompting reprograms a fixed, pre-trained source model to solve new, downstream tasks by optimizing universal input-space perturbations—termed “visual prompts”—and, crucially, specifying a label mapping between source and target class sets. ILM-VP introduces an iterative, bi-level optimization strategy that alternates prompt refinement with dynamic label remapping, demonstrably improving transfer accuracy compared to static or random mapping approaches, and offering extensibility to vision-language models such as CLIP [2211.11635].

## 1. Problem Setting and Formalization

Given a fixed pre-trained “source” classifier $f: \mathbb{R}^d \to \{1,\dots,|S|\}$ (e.g., ResNet-18 trained on ImageNet) and a “target” dataset $\mathcal{D}_t = \{(x_i, y_i)\}_{i=1}^N$ with $x_i \in \mathbb{R}^{d'}$ and labels $y_i \in \{1,\dots,|T|\}$, the goal is to adapt $f$ to new tasks without fine-tuning. VP seeks a universal perturbation $\delta \in \mathbb{R}^d$ applied to all target images—embedding $x$ into the source input domain as $x'(\delta) = h(x, \delta) \in \mathbb{R}^d$—combined with a label mapping $\pi: S \to T$ pairing source and target classes injectively. For a given $(x_i, y_i)$, the overall prediction is $t = \pi(s)$ where $s = \arg \max_{j} f_j(x'_i(\delta))$. Training optimizes
\[
\min_{\delta} \frac{1}{N} \sum_{i=1}^N \ell\left(\pi\big(f(x_i'(\delta))\big), y_i\right),
\]
where $\ell$ is typically cross-entropy loss.

## 2. Label Mapping: Definitions and Metrics

A label mapping $\pi$ is a one-to-one function from source to target classes. Two key metrics assess mapping quality:

- **Mapping Precision**: The fraction of target classes for which the “correct” source partner is mapped back, given ground-truth alignment $s^* : T \to S$.
  \[
  \mathrm{Precision}(\pi) = \frac{1}{|T|}\sum_{t \in T} \mathbf{1}[\pi(s^*(t)) = t]
  \]
  High precision implies few mismatches.

- **Mapping Explanation**: Average log-probability assigned by the source model to the mapped source label, over all prompted images of each target class.
  \[
  \mathrm{Explanation}(\pi) = 
  \frac{1}{|T|} \sum_{t \in T} \frac{1}{|\mathcal{D}_t|} \sum_{i:y_i = t} \log p_f(s_t^\pi \mid x_i'(\delta)),
  \]
  where $p_f(j \mid x)$ is the softmax output, and $s_t^\pi$ is the unique source class with $\pi(s_t^\pi) = t$.

Both metrics empirically correlate strongly with VP target accuracy. Poor mappings—e.g., aligning “daisy” in Flowers102 to a visually dissimilar or semantically unrelated source class—degrade VP effectiveness.

## 3. The ILM-VP Framework and Bi-Level Optimization

ILM-VP abandons pre-fixed mappings in favor of an alternating optimization paradigm. It alternately updates the visual prompt $\delta$ and re-aligns the mapping $\pi$ over training epochs.

**Bi-level formulation:**
- **Upper-level (prompt learning):**
  \[
  \delta^* = \arg\min_{\delta} \frac{1}{N} \sum_{i=1}^N \ell\left(\pi(f(x_i'(\delta))), y_i\right)
  \]
- **Lower-level (mapping update):**
  \[
  \pi^* = \arg\max_{\pi} \mathrm{Explanation}(\pi) \quad\text{s.t. $\pi$ is a bijection $S \to T$}
  \]

Practically, after $K$ epochs of prompt SGD, the mapping is recomputed—per-target-class—by frequency or average likelihood from prompted images. This iterative process is described in the following pseudocode:

```
Input: D_t, source model f, initial map π⁰, initial prompt δ⁰
for k = 0…K−1 do
  # Upper level: prompt generation
  for SGD steps do
    sample (x, y) ∈ D_t
    x' = h(x, δᵏ)
    s = f(x')
    loss = CrossEntropy(πᵏ(s), y)
    δᵏ ← δᵏ − η ∇_δ loss
  end for
  # Lower level: label mapping
  for each t ∈ T do
    consider all prompted x'_i with y_i = t
    record frequency or avg log-prob for each s
    pick s_t = argmax_s { #images where f(x') = s }
    set πᵏ₊₁(s_t) = t
  end for
end for
Output: final prompt δ^K and mapping π^K
```

This synergy between mapping refinement and prompt optimization yields more effective and interpretable mappings, as the two components reinforce each other: improved $\pi$ enhances prompt learning signal, while a stronger $\delta$ refines class alignment.

## 4. Extension to CLIP and Vision-Language Models

For CLIP, which pairs image and text encoders with a contrastive objective, the iterative mapping procedure incorporates text prompt selection per class. With $m$ candidate text templates $\{T_1,...,T_m\}$ and $|T|$ classes, each label–template pair $(j, t)$ forms a “virtual” source token in $\tilde{S}$ of size $m \cdot |T|$. The approach proceeds as follows:

1. **Prompt Update**: Optimize $\delta$ via SGD to minimize cross-entropy between $y$ and the top-scoring text token according to cosine similarity of image and text features.
2. **Text-Prompt Mapping**: For each target class $t$, select the text template $j_t$ maximizing the average similarity on all samples with $y = t$.
3. **Iteration**: Repeat prompt/image and mapping steps.

This iterative text prompt plus label mapping (TP+LM) strategy achieves substantial improvements over fixed template prompting, as evidenced by reported experimental gains.

## 5. Experimental Results and Comparative Analysis

Empirical validation spans 13 diverse target datasets (including Flowers102, CIFAR-10/100, DTD, Food101, GTSRB, and ABIDE) and multiple source models (ResNet-18, ResNet-50, ResNeXt-101). Evaluation baselines include:

- **RLM-VP**: random one-to-one mapping
- **FLM-VP**: fixed frequency mapping before prompting
- **LP**: linear probe on source features
- **FF**: full fine-tuning
- **VP+TP**: CLIP with fixed text prompt

Selected results (ResNet-18 → Flowers102):

| Method     | Accuracy (%) |
|------------|--------------|
| RLM-VP     | ~11.0        |
| FLM-VP     | ~20.0        |
| ILM-VP     | ~27.9        |
| LP         | ~88.0        |
| FF         | ~97.1        |

For CLIP-based VP on Flowers102:
- VP+TP (single prompt): 70.0%
- VP+TP+LM (iterative text + label map): 83.7%

ILM-VP achieves consistent improvements (3–8 percentage point average gain) over RLM-VP and FLM-VP across all source/target pairs while preserving a parameter-efficient footprint—the prompt $\delta$ alone is learned, with no update to backbone network parameters. On CLIP, iterative mapping similarly produces gains of 5–15 percentage points over fixed strategies.

## 6. Practical Considerations and Future Directions

Key insights from the analysis and experimentation include:

- High-quality label mapping is critical for effective VP; poor mapping can significantly impede transfer performance.
- Static or pre-prompt mappings (even frequency-based) are consistently inferior to iterative, dynamic alignment.
- Joint, bi-level optimization of prompt perturbations and label mapping leads to both greater accuracy and more semantically interpretable correspondences between domains.
- The ILM-VP paradigm is naturally extensible to vision-language architectures; treating each (template, label) tuple as a virtual source enables per-class template selection and further boosts transfer.
- Practical recommendations include initializing the mapping randomly or by frequency, alternating between prompt and mapping updates (10–50 prompt SGD steps per mapping re-estimation), and monitoring not only the primary loss but also the mapping precision and explanation metrics.

*Potential extensions* noted include partial reprogramming of intermediate model layers (“multilayer prompts”), graph-based or semantically-informed matching criteria, and adversarial-robust mapping tailored to noisy target domains [2211.11635].

## 7. Context and Significance

ILM-VP addresses a previously neglected consideration in visual prompting: the dynamic relationship between label mapping and prompt effectiveness. By formally quantifying mapping quality, introducing the explanation metric, and demonstrating empirical gains across models and tasks, ILM-VP establishes label mapping as a central component of visual reprogramming. The framework closes much of the performance gap to full fine-tuning with orders-of-magnitude lower adaptation cost and reveals promising directions for efficient transfer learning, especially in settings requiring parameter-efficient adaptation of large vision models [2211.11635].

Source: https://www.emergentmind.com/topics/iterative-label-mapping-ilm-vp