AdaBoosting Text Prompts for Vision-Language Models

Abstract: The classification accuracy of pretrained Vision-LLMs (VLMs) relies on the quality of the text prompts. Handcrafted templates and LLM-generated descriptions not only make predictions more interpretable, but also enable reuse of the same prompts across heterogeneous VLMs. Recent works construct task-adapted text prompts with a small number of labeled images. However, existing few-shot text prompting methods do not explicitly focus on misclassified examples during prompt construction, leading to only marginal improvements even as more shots become available. To fully exploit few-shot supervision, we propose Text Prompt Boosting (TPB), an AdaBoost-inspired framework that treats each text-prompt-based classifier as a weak learner and sequentially aggregates them into a strong ensemble by explicitly targeting hard, misclassified examples. Extensive experiments show that TPB preserves task-intrinsic, model-agnostic cues in text space, enabling robust cross-model transfer. Across eleven classification benchmarks, TPB improves accuracy on the source model and preserves shot-driven gains when transferred to larger, more capable VLMs, where existing methods struggle to sustain such improvements.

• The paper introduces Text Prompt Boosting (TPB), which uses SAMME.R boosting and greedy prompt composition to build ensembles that focus on misclassified examples rather than optimizing only aggregate accuracy.
• TPB improves average CLIP accuracy from 63.8% at one shot to 70.3% at 16 shots, compared with ProAPO’s 62.7% and 64.4%, demonstrating stronger few-shot scalability.
• The method transfers prompts across heterogeneous VLMs by re-embedding natural-language prompts, retaining roughly 2.4–2.5 percentage points of shot-driven gains while avoiding model-specific soft-prompt training.

Overview

"AdaBoosting Text Prompts for Vision-LLMs" (2607.00684) addresses a specific weakness in few-shot text-prompt adaptation of Vision-LLMs (VLMs): existing methods optimize a single aggregate validation metric, which saturates once easy samples are correctly classified, so accuracy improves only marginally as more labeled shots become available. The authors propose Text Prompt Boosting (TPB), an AdaBoost-inspired framework that treats each class-wise text prompt collection as a weak classifier and sequentially aggregates them into an ensemble that explicitly targets misclassified examples. The central empirical claims are shot scalability on the source model and preservation of shot-driven gains when prompts are transferred by re-embedding to larger, heterogeneous target VLMs.

Motivation and positioning

The paper situates itself within hard-prompt optimization for CLIP-style models. Zero-shot description-based methods such as DCLIP and CuPL rely solely on LLM priors and lack grounding in the target visual distribution. Few-shot methods such as LLMbo and ProAPO add visual grounding but optimize overall validation accuracy; because this metric is dominated by easily recognizable samples, gains plateau with additional shots. Soft prompting methods like CoOp achieve higher source-model accuracy at larger shot counts but are tied to a model-specific representation space and cannot be transferred directly. TPB follows the hard-prompt paradigm to retain interpretability and cross-model reusability while introducing boosting to exploit additional supervision.

The connection to prior AdaBoost applications is drawn explicitly: unlike Viola-Jones-style detectors or BoosTexter, where weak learners are hand-designed features or lexical predicates, here the weak learner is a full VLM classifier induced by a prompt collection.

Method

TPB operates over $M$ boosting rounds using SAMME.R, the multi-class real-valued AdaBoost variant. Three components define the framework:

Large prompt pools. For each class, the pool combines 80 CLIP hand-crafted templates, LLM-generated descriptions from DCLIP, CuPL, GPT4Vis, and related works, plus concatenations of templates and descriptions.

Greedy Prompt Composition (GPC). Exhaustive joint search over per-class prompt banks is infeasible, so GPC proceeds in two stages. Stage 1 selects the single shared template minimizing weighted classification error under current sample weights. Stage 2 greedily appends prompts class-by-class, choosing the candidate yielding the largest reduction in weighted error until a full pass produces no update. A notable design choice is prompt duplication: since class scores are unweighted averages over bank prompts, appending $d$ copies of a prompt approximates continuous weighting without learnable parameters. The supplementary material derives closed-form thresholds for "upside" and "downside" prediction flips, showing that the change in weighted error is a step function of $d$, so the optimal duplication factor can be found exactly over a finite threshold set. A $\Delta\varepsilon/d$ normalization penalizes large duplications to preserve bank diversity.

Boosting-loop augmentation. With augmentation factor $a$, each training image is replicated via random resized crop and horizontal flip before each round's weighted-error evaluation. This is essential: without augmentation, the ensemble memorizes the few-shot set within a few rounds, training accuracy saturates near 100%, and subsequent rounds receive no informative errors — test accuracy stagnates. With $a=4$, transformed views keep producing fresh mistakes, sustaining improvement across all 50 rounds.

Inference cost is comparable to single-prompt baselines since ensemble text embeddings are precomputed; each image requires one forward pass.

Shot scalability results

On eleven benchmarks (ImageNet-1K, Caltech101, Pets, Cars, Flowers102, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, UCF101) with OpenAI CLIP RN50, TPB achieves 63.8% average top-1 accuracy at 1 shot versus ProAPO's 62.7%, and 70.3% at 16 shots versus ProAPO's 64.4% — a 6.5 pp gain from 1 to 16 shots where ProAPO gains only 1.7 pp. On ViT-B/32, TPB reaches 72.9% average at 16 shots. CoOp remains higher at 16 shots on the source model (73.2% RN50), but the paper argues this advantage does not survive transfer.

A qualitative analysis on a beagle/basset hound task illustrates the mechanism: the first weak classifier captures dominant cues ("large floppy ears," "low stature") but fails on an occluded basset hound lying down; reweighting forces later rounds to select alternative cues (facial wrinkles, snout shape), which a static single-objective method would never resolve.

Cross-model transfer

Prompts optimized on OpenAI CLIP ViT-B/32 transfer by simple re-embedding to five ViT-L-scale targets (OpenCLIP, SigLIP2, DFN, EVA-02-CLIP, CLIPA-v2) and four ViT-H-scale targets (OpenCLIP, DFN, CLIPA-v2, MetaCLIP2). At 16 shots, TPB lifts average accuracy from 76.76% (zero-shot) to 82.07% on ViT-L targets and from 78.75% to 84.24% on ViT-H targets, exceeding ProAPO by 2.06 pp and 1.85 pp respectively. Critically for the paper's thesis, TPB retains shot-driven gains after transfer (+2.41 pp from 1→16 shots on ViT-L, +2.51 pp on ViT-H), whereas ProAPO gains only 0.42 pp and 0.32 pp. Supplementary results extend this trend to ViT-G and ViT-E targets, and a large-to-small experiment (ViT-L/14 → EVA-02 B/16) shows +6.98 pp retained gain, providing initial evidence of bidirectional transferability.

Against continuous baselines transferred via emulated fine-tuning (EFT), TPB holds a margin while requiring only a single forward pass rather than three concurrent models. However, the target-wise breakdown shows CoOp+EFT and PromptSRC+EFT occasionally exceed TPB at 16 shots on individual targets; their behavior is highly target-dependent, with severe degradation on DFN targets, which the authors attribute to brittleness of logit-space emulation across architecture or recipe changes.

Failure analysis of gradient-optimized prompts. PEZ, which optimizes discrete context tokens by gradient projection, transfers catastrophically: adapted on RN50 it improves source accuracy from 55.98% to 61.09%, but drops ViT-L/14 accuracy from 71.07% to 53.02% — below zero-shot. Inspection reveals prompts like "hey darby pls violets kissestc accept liza any,, ♡♡ :-) life behaved pug": syntactically text, semantically noise. The paper draws a pointed conclusion: transferability does not follow merely from being text; prompts must remain close to fluent, human-readable natural language, or they act as model-specific codes.

Ablations

Several ablations isolate the contributions:

Factor	Finding
Augmentation factor	Monotone gains up to $a=4$; largest in low-shot regimes
Prompt pool diversity	CuPL+DCLIP beats CuPL alone despite DCLIP being individually weaker; full pool best
Number of rounds $M$	Saturates around $M=30$–50; $M=1$ is slightly below ProAPO at 1 shot
Duplication	Boosting alone already outperforms baselines; duplication adds ~0.1–0.5 pp
Fixed exposure ($d$0)	More boosting, less augmentation favors transfer; more augmentation favors source accuracy

The fixed-exposure ablation is the most informative: it demonstrates that transfer robustness stems from iterative reweighting rather than augmented data exposure per se, though it also reveals a tension — configurations maximizing source accuracy degrade transfer accuracy.

Limitations and open questions

The paper concedes three limitations directly. First, construction cost becomes non-negligible on large-scale datasets where the candidate space grows substantially; ImageNet required reducing $d$1 from 50 to 30. Second, AdaBoost-style reweighting may overemphasize noisy or atypical few-shot samples, a risk inherent to any hard-example-focused method under tiny supervision. Third, TPB selects from a fixed prompt pool, limiting flexibility for fine-grained or ambiguous classes not well covered by existing LLM descriptions. An additional open question raised implicitly by the ablations is whether the source-versus-transfer trade-off under fixed exposure can be resolved adaptively rather than by a fixed $d$2 allocation.

Conclusion

TPB reframes few-shot text prompting as an ensemble-learning problem, applying SAMME.R boosting over discrete natural-language prompt collections selected by greedy weighted-error minimization. Its strongest empirical results are the 6.5 pp shot-scalability gain on the source model and the retention of shot-driven improvements (roughly 2.4–2.5 pp) after cross-model transfer, where ProAPO saturates below 0.5 pp. The PEZ failure analysis adds a substantive observation about what makes text prompts transferable. The method's dependence on a static prompt pool, its construction cost at scale, and its sensitivity to noisy few-shot labels remain unresolved constraints on its applicability.