Adaptive-label ICM: Dynamic Label Adaptation

• Adaptive-label ICM is a framework that modifies label representations in in-context examples to enhance semantic density and boost classification accuracy within context length limits.
• Techniques like Label Distribution Enhancement and Visual Descriptions Enhancement enrich information content, reduce label noise, and improve few-shot performance in vision-language models.
• The approach extends to weak supervision and image coding by optimizing label mixing and data augmentation, achieving efficient resource utilization and task-specific compression.

Adaptive-label ICM refers to a family of adaptive techniques for In-Context Classification and related computational frameworks, primarily leveraging the manipulation or dynamic adaptation of label representations to improve classification or task-specific performance under resource, supervision, or context constraints. This includes (but is not limited to) strategies for knowledge-dense prompting in few-shot in-context classification with vision-LLMs (VLMs), and advances in context- or cluster-adaptive synthetic label generation for data augmentation in weakly-supervised learning. Adaptive-label ICM methods target both increasing semantic density per in-context example (ICE) and tailoring label mixing for label noise control or task-optimized model compression.

1. Adaptive-label In-Context Classification: Problem Setting and Formalization

In the context of vision-LLMs supporting in-context learning (ICL), adaptive-label ICM entails modifying how class labels are incorporated within in-context examples presented to the model. Let a VLM with maximal context length $L_\mathrm{max}$ be given. The in-context classification task is defined as follows (Chen et al., 2023):

• The context set $D = \{ (I_1, L_1), \dots, (I_M, L_M) \}$, with $I_i$ an image and $L_i$ a ground-truth label (text).
• For test image $\tilde{I}$, select $n$ in-context examples (ICEs) $\{(I'_1, L'_1), \dots, (I'_n, L'_n)\} \subset D$ and construct the sequence

$$S = [ (I'_1, \text{``A photo of a } \{L'_1\}.''); \dots; (I'_n, \text{``A photo of a } \{L'_n\}.''); \tilde{I}, \text{``This is a photo of a''}~[\text{mask}] ].$$

• The VLM generates the token sequence to fill $[\text{mask}]$, with per-class log-probability scores defined as

$$s(c, \tilde{I}) = \frac{1}{\ell(c)} \sum_{t=1}^{\ell(c)} \log P(w_t = \text{token}_t(c) \mid S, w_{1:t-1})$$

over candidate class names $D = \{ (I_1, L_1), \dots, (I_M, L_M) \}$0. The predicted class is $D = \{ (I_1, L_1), \dots, (I_M, L_M) \}$1.

Adaptive-label ICM replaces each single-label $D = \{ (I_1, L_1), \dots, (I_M, L_M) \}$2 with a richer label representation—either a soft distribution over labels or an expanded visual description—subject to the constraint $D = \{ (I_1, L_1), \dots, (I_M, L_M) \}$3, aiming to maximize accuracy while minimizing $D = \{ (I_1, L_1), \dots, (I_M, L_M) \}$4.

2. Label Space Manipulation Strategies

2.1 Label Distribution Enhancement (LDE)

LDE increases label expressiveness by representing each ICE label as a distribution over the label set $D = \{ (I_1, L_1), \dots, (I_M, L_M) \}$5 (Chen et al., 2023). The strategy computes cross-modal similarities via a frozen CLIP encoder:

• Compute image and text embeddings: $D = \{ (I_1, L_1), \dots, (I_M, L_M) \}$6 and $D = \{ (I_1, L_1), \dots, (I_M, L_M) \}$7 for all $D = \{ (I_1, L_1), \dots, (I_M, L_M) \}$8.
• Similarity: $D = \{ (I_1, L_1), \dots, (I_M, L_M) \}$9.
• Softmax label distribution: $I_i$0.

LDE provides three prompt variants:

• Equidistributed Label (EL): Select top-$I_i$1 labels by similarity, assign equal weights.
• Distributed Label (DL): Use full softmax distribution.
• Descriptive Distribution (DD): Verbosely format $I_i$2 as a prompt, e.g., ``Labels: sparrow(0.72), robin(0.15), finch(0.13).''

No model optimization is performed; LDE alters ICE presentation text only.

2.2 Visual Descriptions Enhancement (VDE)

VDE targets fine-grained or rare labels lacking informative textual associations (Chen et al., 2023):

• For each $I_i$3, retrieve $I_i$4 prototypical images via CLIP similarity.
• Query the VLM with ``Q: What are the useful visual features for distinguishing a {L_m}?'' and feed the $I_i$5-image context.
• Decode a bullet-list $I_i$6 of concise, discriminative visual features.
• Substitute a $I_i$7'' bya $I_i$8 distinguished by $I_i$9'' in each ICE.

Both LDE and VDE strategies increase the information content (knowledge density) per ICE, thus improving shot efficiency.

3. Adaptive-label ICM Algorithmic Workflow

The generic workflow is as follows (Chen et al., 2023):

1. Selection: Retrieve top-$L_i$0 ICEs via CLIP similarity to test image ($L_i$1) or randomly.
2. Label Adaptation: For each $L_i$2 in ICEs:
   • If using LDE, compute distribution and format label prompt as per chosen variant.
   • If VDE is enabled, append descriptive features for $L_i$3.
   • Construct prompt entry: $L_i$4{label_text}$L_i$5
3. Prompt Assembly: Create in-context sequence $L_i$6 with $L_i$7 ICEs plus the masked test image.
4. VLM Inference: Generate tokens to fill $L_i$8 and compute $L_i$9 for each $\tilde{I}$0.
5. Prediction: Output $\tilde{I}$1.

This process directly manipulates ICE label content to increase semantic density under a sequence length budget.

4. Empirical Performance and Knowledge-Density Analysis

Key experimental findings are summarized in the following tables (all with Open-Flamingo-3B, ViT-L/14 backbone, BF16, no fine-tuning) (Chen et al., 2023):

Dataset	SL 2-shot	CLIP	LDE(DD)	VDE	ENS (LDE+VDE)
ImageNet	73.90%	75.54%	75.83%	75.00%	76.21% (+0.67)
CUB-200	63.07%	56.90%	75.83%	69.05%	69.05% (+12.15)
Stanford Dogs	66.38%	69.09%	66.69%	64.31%	67.76% (–1.33)
Stanford Cars	79.55%	77.19%	77.73%	80.57%	80.11% (+2.92)

Dataset	SL 1-shot	LDE(DD)	VDE	ENS
ImageNet	68.50%	73.63%	73.65%	75.08%
CUB-200	48.86%	66.62%	67.97%	69.05%
Stanford Dogs	61.40%	65.55%	63.87%	62.96%
Stanford Cars	74.19%	75.64%	79.14%	78.88%

LDE and VDE drive substantial 1- and 2-shot performance gains, often matching or surpassing 4-shot single-label (SL) baselines.

Significance:

• Single-label ICEs encode only a class token; LDE compacts a probability vector over multiple classes, compressing several ICEs' worth of information.
• VDE injects explicit visual features, reducing class ambiguity, especially in fine-grained settings.
• From an information-theoretic perspective, enhancing the ICE raises its mutual information with the queried class, thus fewer ICEs attain the same disambiguation.

5. Adaptive-label Mixup: Adaptive ICM in Weak Supervision

Adaptive-label ICM mechanisms extend to other modalities and learning settings, notably mixup-based data augmentation for complementary-label learning (CLL), where only negative (“does not belong to class $\tilde{I}$2”) labels are available (Mai et al., 22 Sep 2025). Intra-Cluster Mixup (ICM) restricts mixup to pairs within unsupervised feature clusters, substantially reducing noisy synthetic labels that violate CLL constraints.

Further adaptivity is realized by adjusting label mixing coefficients ($\tilde{I}$3) or label composition functions using cluster-level statistics:

• Cluster purity–adaptive mixing: Set $\tilde{I}$4 where $\tilde{I}$5 is majority-class proportion.
• Distance weighting (“MICM”): Combine label components inversely proportional to distance in feature space.
• Adaptive $\tilde{I}$6: Lower $\tilde{I}$7 if synthetic-label noise ratio increases.
• Cluster-size scaling: Scale $\tilde{I}$8 by cluster size to favor more “hard” mixing in small clusters.

This yields a context-aware, dynamically adaptive-label ICM procedure that trades off between encouraging interpolation and controlling synthetic-label noise. When implemented, this approach consistently enhances accuracy under both balanced and imbalanced settings, exceeding naive mixup and standard CLL losses by 5–30 percentage points (Mai et al., 22 Sep 2025).

6. Related Directions: Adaptive-label Approaches in Image Coding for Machines

In image coding for machines (ICM), adaptive-label ideas are embodied in frameworks like Prompt-ICM, where task-driven prompts (compression prompts and task-adaptive prompts) steer both the feature compression process and downstream analysis (Feng et al., 2023). Compression prompts—importance maps—allocate more bits to semantically salient regions, while task-adaptive prompts modulate backbone inference for specific tasks (classification, detection, segmentation). The overall optimization objective balances rate and task-driven distortion:

$\tilde{I}$9

Task-adaptive prompt learning achieves task specialization with orders of magnitude fewer parameters than full fine-tuning.

Across three vision tasks and at equivalent bitrates, Prompt-ICM outperforms classical codecs (VVC, HEVC) and strong learned codecs by 5–15 percentage points in task accuracy, and reduces required bitrate by 30–50% for the same accuracy (Feng et al., 2023). Thus, adaptive-label and prompt-driven approaches enable efficient resource allocation across supervised and data-efficient learning regimes.

7. Limitations and Future Directions

Adaptive-label ICM approaches, especially in the context of large VLMs and weak supervision, are presently limited by their dependence on frozen encoders (e.g., CLIP), the fidelity of generated descriptions, and the available sequence context for ICEs. While empirical findings robustly support gains in shot efficiency and information utilization, there is ongoing research to formalize the optimal label-density/sequence-length trade-off, to further automate label-space enrichment, and to improve model generalization to out-of-distribution classes (Chen et al., 2023, Mai et al., 22 Sep 2025).

A plausible implication is that adaptive-label ICM will serve as a general paradigm for knowledge-efficient in-context learning, augmentation, and resource-aware compression, particularly in multi-task and data-sparse regimes.