ConceptBank: Robust Calibration for OVS

• ConceptBank is a calibration framework for open-vocabulary segmentation that constructs dataset-specific calibrated concept embeddings to address data and concept drift.
• It employs prototype estimation, representative mining, and LLM-guided concept fusion to refine prompt embeddings without requiring parameter updates.
• Empirical evaluations demonstrate substantial mIoU gains on natural scenes and remote sensing datasets, enhancing both robustness and efficiency.

ConceptBank is a parameter-free calibration framework designed to improve the robustness and efficiency of open-vocabulary segmentation (OVS) with promptable foundation models, most notably SAM3. By systematically constructing a dataset-specific collection of calibrated concept embeddings, ConceptBank enables effective adaptation to both data drift (changes in input distribution) and concept drift (shifts in label semantics) in new domains, thereby addressing key vulnerabilities of static prompt-based segmenters (Pei et al., 6 Feb 2026).

1. Motivation: Managing Data and Concept Drift in OVS

OVS models like SAM3 segment images by matching dense visual features against fixed prompt embeddings generated from class names. This paradigm is highly sensitive to two forms of distributional shift:

• Data drift ($P_S(X) \neq P_D(X)$): Target images exhibit spectral, geometric, or textural statistics distinct from the pre-training set. As a result, region embeddings $\phi_V(x)$ may be misaligned with source text prompts $\mathbf e_c^{S} = \phi_T(t_c)$, yielding inaccurate or incomplete masks.
• Concept drift ($P_S(Y|X) \neq P_D(Y|X)$): The meaning of a class name or its visual manifestation may differ between domains (e.g., "mouse" as animal vs. peripheral), rendering static prompt embeddings non-representative of the target task.

ConceptBank addresses both issues by forgoing static source prompts in favor of a compact, dataset-calibrated set of query embeddings $\{\mathbf e_c^*\}_{c\in\mathcal C}$, derived directly from target-domain supervision. This process involves: (i) anchoring concept representations to actual target-domain visual features (prototype estimation), (ii) filtering out outlier exemplars likely to be artifacts of drift (representative mining), and (iii) rectifying semantic mismatch via fusion of multiple LLM-generated candidate prompts (concept fusion).

2. Formal Framework and Mathematical Definitions

Let $\mathcal D_{\mathrm{sup}} = \{(x_i, y_i)\}_{i=1}^N$ denote a labeled support set in the target domain, with $y_i^c$ as the binary mask for class $c$ in $x_i$.

2.1 Prototype Estimation

For each mask-crop $(v, y_i^c)$, extract dense feature maps $\phi_V(x)$0 at location $\phi_V(x)$1 and aggregate using a mask-pooled embedding:

$\phi_V(x)$2

Aggregate across $\phi_V(x)$3 mask-crops for class $\phi_V(x)$4 to obtain the class prototype:

$\phi_V(x)$5

2.2 Representative Support Mining

To mitigate the influence of outlier crops (e.g., occluded, background-dominated), select only the $\phi_V(x)$6 mask-pooled embeddings closest to the prototype (by cosine similarity):

$\phi_V(x)$7

In practice, $\phi_V(x)$8.

2.3 Concept Fusion

Expand each class name via LLM (e.g., GPT) to $\phi_V(x)$9 candidate prompts $\mathbf e_c^{S} = \phi_T(t_c)$0, yielding their embeddings $\mathbf e_c^{S} = \phi_T(t_c)$1.

Each candidate is scored by mean Dice coefficient over $\mathbf e_c^{S} = \phi_T(t_c)$2:

$\mathbf e_c^{S} = \phi_T(t_c)$3

Fuse the top $\mathbf e_c^{S} = \phi_T(t_c)$4 candidates via temperature-softmax:

$\mathbf e_c^{S} = \phi_T(t_c)$5

Typically, $\mathbf e_c^{S} = \phi_T(t_c)$6 and $\mathbf e_c^{S} = \phi_T(t_c)$7.

3. Algorithmic Workflow and Pseudocode

ConceptBank construction consists of three main stages: prototype estimation, representative support mining, and concept fusion. The implementation is parameter-free with respect to SAM3 and requires no gradient-based updates.

Algorithm Outline:

$\{\mathbf e_c^*\}_{c\in\mathcal C}$9

Inference: Store concept bank $\mathbf e_c^{S} = \phi_T(t_c)$8 as a $\mathbf e_c^{S} = \phi_T(t_c)$9 matrix; at test time, for a new image $P_S(Y|X) \neq P_D(Y|X)$0, compute all class masks via

$P_S(Y|X) \neq P_D(Y|X)$1

in a single forward pass.

4. Experimental Evaluation and Impact

4.1 Domains and Benchmarks

ConceptBank is evaluated on diverse OVS datasets that pose both data and concept drift:

• Natural-scene OVS: Pascal VOC21, Pascal Context60, COCO-Object, VOC20, PC59, COCO-Stuff, Cityscapes, ADE20K.
• Remote-sensing OVS: LoveDA, Potsdam, Vaihingen, iSAID.

These benchmarks introduce visual and semantic shifts, including overhead perspectives and varying class definitions.

4.2 Quantitative Results

Scenario	Reference (SAM3, %)	ConceptBank (%)	Gain (%)
Natural Scenes	57.5	67.1	+9.6
PC60	46.1	56.5	+10.4
Cityscapes	49.5	62.3	+12.8
Remote Sensing	39.1	52.1	+13.0
LoveDA	35.6	49.4	+13.8
iSAID	17.7	35.4	+17.7

Over eight natural scene and four remote-sensing datasets, ConceptBank consistently yields a substantial increase in mean IoU (mIoU), outperforming vanilla SAM3 and prompt expansion baselines by up to $P_S(Y|X) \neq P_D(Y|X)$2 on challenging settings.

4.3 Ablation Analysis

Component-wise contributions to overall gain (natural scenes / remote sensing):

• Prompt expansion alone: $P_S(Y|X) \neq P_D(Y|X)$3 / $P_S(Y|X) \neq P_D(Y|X)$4
• • Prototype anchoring: $P_S(Y|X) \neq P_D(Y|X)$5 / $P_S(Y|X) \neq P_D(Y|X)$6
• • Representative mining: $P_S(Y|X) \neq P_D(Y|X)$7 / $P_S(Y|X) \neq P_D(Y|X)$8
• • Concept fusion (full pipeline): $P_S(Y|X) \neq P_D(Y|X)$9 / $\{\mathbf e_c^*\}_{c\in\mathcal C}$0

This establishes the necessity of all three stages—each improves drift robustness incrementally.

5. Practical Integration and Efficiency

ConceptBank is designed for seamless integration with frozen foundation models such as SAM3:

• Offline: Execute build_concept_bank(...) once per target dataset to generate $\{\mathbf e_c^*\}_{c\in\mathcal C}$1.
• Online: Replace standard SAM3 text embedding inference with precomputed $\{\mathbf e_c^*\}_{c\in\mathcal C}$2 entries for all segmentation queries.
• Hyperparameters: Defaults—$\{\mathbf e_c^*\}_{c\in\mathcal C}$3 (robust in $\{\mathbf e_c^*\}_{c\in\mathcal C}$4), $\{\mathbf e_c^*\}_{c\in\mathcal C}$5 (number of prompt variants), $\{\mathbf e_c^*\}_{c\in\mathcal C}$6, $\{\mathbf e_c^*\}_{c\in\mathcal C}$7.
• Prompt Expansion: Supply the official dataset label definitions to the LLM; restrict alterations to synonyms or concise attribute phrases.
• Efficiency: No gradients or model parameter updates are required. Inference is $\{\mathbf e_c^*\}_{c\in\mathcal C}$8 faster than single-prompt SAM3 and considerably faster than multi-prompt ensembling.

A plausible implication is that the parameter-free nature and efficiency enable rapid, robust adaptation to novel domains without retraining or extensive prompt engineering.

6. Significance and Limitations

ConceptBank establishes a new baseline for distribution-robust OVS. Its three-stage pipeline—prototype anchoring, representative mining, and LLM-guided concept fusion—directly addresses two principal modes of drift, restoring alignment between prompts and visual evidence in-the-wild scenarios. All empirical results are obtained with no changes to the base SAM3 model weights.

A potential limitation is the reliance on sufficient support set annotation and the quality of LLM-generated prompt expansions. The method’s performance depends on how well the support data and prompt variants span the shifted distributions. However, empirical results suggest robustness to support size and prompt configuration within broad default settings, across diverse domain shifts (Pei et al., 6 Feb 2026).