C-GAP: Class-Aware and Online Prompting Improves Vision-Language Models on Imbalanced Classes

Published 10 Jul 2026 in cs.CV | (2607.09008v1)

Abstract: Safety-critical perception systems must reliably detect rare object classes within small label spaces, a setting that long-tailed detection methods, designed for hundreds of classes with dense annotation, fundamentally do not address. Open-vocabulary detectors offer a promising alternative, as they use natural language queries at inference time, making prompt quality a first-class lever for detection performance. We exploit this property to address class imbalance: rather than retraining models or collecting additional annotations, we ask whether iteratively refining the language prompts, fed to frozen detectors, can improve minority class detection. We introduce C-GAP Caption-Guided Augmentation and Prompting), a detector-agnostic, annotation-free framework that operates in two phases. First, we establish a composite caption baseline combining per-image scene descriptions with class-quantity context, which we show outperforms scene-description only or class-quantity-only prompts across multiple open-vocabulary architectures and benchmarks. Second, an LLM iteratively refines each image's caption individually, with trials triaged into accept, tentative, or regenerate buckets based on minority-class [email protected] against a dynamic threshold derived from the composite baseline. Refinement terminates early once sufficient [email protected] gain is achieved. No detector weights are updated at any stage. Our experiments shows that C-GAP improves minority-class average precision up to 53% over the baselines. On COCO, C-GAP improves minority-class [email protected] by ~81% relative over the composite baseline (17.69 -> 32.09). Experiments confirm that composite captions provide the critical foundation for effective refinement: using scene-description-only or class-quantity-only prompts as the refinement starting point yields diminishing returns, supporting both stages of C-GAP as necessary contributions.

Abstract PDF Upgrade to Chat

Authors (3)

Summary

The paper introduces C-GAP, a training-free prompt refinement method that significantly improves minority-class AP in vision-language models.
It employs a two-phase approach combining scene description and class quantity to generate composite captions refined iteratively using LLM feedback.
C-GAP achieves up to 81% relative improvement for minority classes on datasets like COCO, Cityscapes, and Chula Vista without altering model weights.

C-GAP: Class-Aware Online Prompting for Minority-Class Vision-LLM Adaptation

Introduction

Open-vocabulary detectors (OVDs) have established a paradigm shift in object detection by using natural language querying at inference, decoupling detectors from rigidly fixed categorical heads. A critical challenge, especially in safety-critical perception deployments such as urban traffic monitoring, arises from severe class imbalance—majority classes (e.g., cars) dominate, while minority classes (e.g., cyclists, pedestrians) are both rare and crucial for reliable detection. Standard long-tail learning methods (re-sampling, loss reweighting) cannot leverage unlabeled rare-class data, and collecting additional minority annotations is infeasible in practical scenarios.

The paper "C-GAP: Class-Aware and Online Prompting Improves Vision-LLMs on Imbalanced Classes" (2607.09008) proposes C-GAP, a detector-agnostic, training-free, annotation-free framework that leverages iterative, LLM-driven caption refinement to drive substantial gains for minority-class detection—using feedback from the frozen detector itself, and without updating any model parameters. C-GAP achieves up to 81% relative improvement in minority-class AP@0.5 on COCO and recovers non-zero recall from zero in extremely imbalanced settings (e.g., Chula Vista).

Figure 1: Chula Vista intersection: cars dominate ( ${\sim}80\%$ ) while minority classes such as cyclists constitute only $\sim8\%$ of instances, illustrating the real-world challenge of imbalanced detection.

Problem Formulation and Framework

C-GAP addresses the minority-class bottleneck in OVDs by fully exploiting natural language prompting as a control axis at inference. The setup assumes a frozen detector $f_\theta$ and leverages two complementary prompt modalities for each image:

Scene Description (SD): Global context summarizing overall scene.
Class Quantity (CQ): Explicit enumeration of object class counts from the fixed class list.

These are concatenated into per-image composite captions (CC), which serve as the initialization for prompt refinement. Empirical analysis reveals that neither SD nor CQ, when used in isolation, consistently drives robust minority recall across backbones or datasets—thus, their combination is essential.

C-GAP then iteratively refines per-image captions using LLMs, where feedback is determined by the aggregate minority-class [email protected] produced by the frozen detector under the current prompt set.

Figure 2: Overview of the C-GAP framework: Phase I constructs per-image composite captions (Scene Description + Class Quantity); Phase II iteratively refines these captions, assigning trials to triage buckets based on measured [email protected], with early stopping and selective regeneration.

Iterative refinement is governed by a three-bucket triage with adaptive thresholds relative to the starting composite-caption [email protected] baseline:

Regenerate (B1): Candidate captions with AP below a low threshold are discarded, and corrective feedback (e.g., encourage minority-class surfacing) is emphasized.
Tentative (B2): Captions within a margin are retained and further refined.
Keep (B3): Captions achieving AP above the high threshold may terminate refinement early.

At no point are model weights modified; all gains are attributed exclusively to prompt content.

Experimental Setup

Experiments span three evaluation datasets:

COCO (MS-COCO 2017 val) [lin2014coco]: Minority = bus
Cityscapes [cordts2016cityscapes]: Minority = truck
Chula Vista (Intersection Surveillance): Minority = bike (an extreme class-imbalance regime)

Evaluations leverage four state-of-the-art OVDs with frozen weights:

Grounding DINO [liu2023groundingdino]
OmDet-Turbo [zhao2024omdetturbo]
OWLv2 [minderer2023owlv2]
YOLO-World [cheng2024yoloworld]

The main metric is minority-class [email protected], with mAP@0.5 serving to monitor for possible degradation in majority-class performance. No detector is fine-tuned and no additional annotations are used. Caption generation in both phases uses instruction-tuned vision-LLMs.

Figure 3: Class-instance distributions across the three datasets, highlighting the extreme imbalances that motivate C-GAP's approach.

Results

Composite Caption Type Analysis

Static prompt selection is brittle: for a fixed OVD, minority-class AP can range from zero to high values solely due to prompt structure. Across 36 model-dataset-caption type configurations, no static caption type (SD, CQ, or CC) consistently dominates for minority AP. The sensitivity is architecture- and domain-specific, often requiring prohibitively exhaustive prompt search.

C-GAP consistently outperforms the composite-caption (CC) baselines in 10 of 12 backbone-dataset pairs. For example:

COCO/YOLO-World: $17.69 \to 32.09$ (+81% relative) for bus
Chula Vista/Grounding DINO: $0.00 \to 10.29$ (non-zero recall achieved purely through prompting)
Cityscapes/OmDet-Turbo: $21.15 \to 22.10$ for truck

Figure 4: Paired CC→C-GAP minority-class [email protected] for each backbone/dataset pair; upward slopes indicate consistent improvement from caption refinement.

Gains are concentrated in cases where initial CC prompts yield zero or low minority recall, affirming that prompt content—not detector capacity—is the bottleneck in the low-data regime.

Qualitative and Prompt Analysis

Refined per-image captions explicitly surface the minority class only where appropriate. For example, on Chula Vista with a cyclist minority, refined prompts increase Bike detections without introducing hallucination or majority-class mislabeling.

Figure 5: An example of a C-GAP-refined caption that surfaces the minority class, resulting in improved detection (in this case, Bike).

Qualitative comparisons further demonstrate that C-GAP recovers missed minority-class objects using identical detector weights—a direct success of prompt engineering rather than model retraining.

Figure 6: Qualitative detection comparison: C-GAP (bottom) recovers missed minority-class (Bike) instances, unlike the CC baseline (top).

Minority AP/Overall mAP Tradeoff

In most configurations, overall [email protected] is unchanged or only minimally decreased despite substantial gains in minority-class [email protected]. This demonstrates that prompt refinement can sharpen minority sensitivity without wholesale sacrifice of majority performance.

Figure 7: Minority [email protected] gain vs. [email protected] change; most gains occur without significant mAP degradation, confirming an effective tradeoff.

Ablation and Generator–Backbone Analysis

Two-Phase Structure: Ablations show that composite caption initialization is crucial; CC→C-GAP consistently outperforms SD or CQ initialization across trial budgets, highlighting the necessity of aligning both context and class presence before refinement.
Generator Sensitivity: Final outcomes depend on both the LLM used for refinement and the OVD’s prompt processing mechanisms. For instance, YOLO-World exhibits increased keep rates when paired with moondream (concise captions), whereas multi-modal fusion models benefit from more descriptive LLMs.

Theoretical and Practical Implications

C-GAP operationalizes the insight that prompt design, evaluated in a closed loop with the inference-time frozen OVD, is an efficient mechanism for adapting to real-world, annotation-sparse, imbalanced-class scenarios. Most notably, it provides:

Annotation-Free, Training-Free Adaptation: All improvements are realized without any new labels, detector parameter updates, or fine-tuning cycles.
Model-Agnostic Prompt Engineering: The method generalizes across disparate OVD architectures, illustrating that LLM-driven refinement interacts nontrivially with both model internals and domain-specific class distributions.
Diagnostic Value: Triage bucket distributions provide actionable, label-free diagnostics: persistent B1 signals generator–backbone mismatch; B2 saturation signals initialization near optimal for the detector; B3-rich regimes indicate effective feedback/utilization.

In practical deployments, especially in smart-city scenarios or any setting with shifting instance statistics (edge devices, traffic intersections, etc.), C-GAP enables continuous improvement in rare-class detection without annotation or training overhead.

Future Directions

Advancements include dynamic margin calibration for the triage mechanism—potentially leveraging backbone- or domain-specific AP variance during deployment—and extension of the feedback signal to multi-class minority objectives. Integrating C-GAP into an online, streaming feedback loop is poised to deliver annotation-free ongoing adaptation in production safety-critical systems.

Conclusion

C-GAP embodies a robust, systematic approach to addressing minority-class underperformance in OVDs by leveraging the full capacity of language-prompted, feedback-driven refinement. The separation of model parameters, prompt content, and label-free evaluation offers both practical utility and new theoretical insight into the deployment-time adaptation of vision-language architectures.