---
title: 'ConceptBank: Robust Calibration for OVS'
url: https://www.emergentmind.com/topics/conceptbank
type: topic
---

# ConceptBank: Robust Calibration for OVS

ConceptBank is a parameter-free calibration framework designed to improve the robustness and efficiency of open-vocabulary segmentation (OVS) with promptable foundation models, most notably SAM3. By systematically constructing a dataset-specific collection of calibrated concept embeddings, ConceptBank enables effective adaptation to both data drift (changes in input distribution) and concept drift (shifts in label semantics) in new domains, thereby addressing key vulnerabilities of static prompt-based segmenters [2602.06333].

## 1. Motivation: Managing Data and Concept Drift in OVS

OVS models like SAM3 segment images by matching dense visual features against fixed prompt embeddings generated from class names. This paradigm is highly sensitive to two forms of distributional shift:

- **Data drift** ($P_S(X) \neq P_D(X)$): Target images exhibit spectral, geometric, or textural statistics distinct from the pre-training set. As a result, region embeddings $\phi_V(x)$ may be misaligned with source text prompts $\mathbf e_c^{S} = \phi_T(t_c)$, yielding inaccurate or incomplete masks.
- **Concept drift** ($P_S(Y|X) \neq P_D(Y|X)$): The meaning of a class name or its visual manifestation may differ between domains (e.g., "mouse" as animal vs. peripheral), rendering static prompt embeddings non-representative of the target task.

ConceptBank addresses both issues by forgoing static source prompts in favor of a compact, dataset-calibrated set of query embeddings $\{\mathbf e_c^*\}_{c\in\mathcal C}$, derived directly from target-domain supervision. This process involves: (i) anchoring concept representations to actual target-domain visual features (prototype estimation), (ii) filtering out outlier exemplars likely to be artifacts of drift (representative mining), and (iii) rectifying semantic mismatch via fusion of multiple LLM-generated candidate prompts (concept fusion).

## 2. Formal Framework and Mathematical Definitions

Let $\mathcal D_{\mathrm{sup}} = \{(x_i, y_i)\}_{i=1}^N$ denote a labeled support set in the target domain, with $y_i^c$ as the binary mask for class $c$ in $x_i$.

### 2.1 Prototype Estimation

For each mask-crop $(v, y_i^c)$, extract dense feature maps $\psi(u;v) \in \mathbb{R}^d$ at location $u$ and aggregate using a mask-pooled embedding:

$$
\mathbf z(v, y_i^c) = \mathrm{Norm}\Bigl( \frac{\sum_{u} y_i^c(u)\, \psi(u; v)} {\sum_{u} y_i^c(u) + \epsilon} \Bigr)
\tag{1}
$$

Aggregate across $N_c$ mask-crops for class $c$ to obtain the class prototype:

$$
\mathbf p_c = \mathrm{Norm}\Bigl( \frac{1}{N_c} \sum_{i,v} \mathbf z(v, y_i^c) \Bigr)
\tag{2}
$$

### 2.2 Representative Support Mining

To mitigate the influence of outlier crops (e.g., occluded, background-dominated), select only the $K$ mask-pooled embeddings closest to the prototype (by cosine similarity):

$$
\mathcal R_c = \mathrm{Top}_K\!\Bigl( \{(v, y)\},\; (v, y) \mapsto \cos(\mathbf z(v, y), \mathbf p_c) \Bigr)
\tag{3}
$$

In practice, $K = 10$.

### 2.3 Concept Fusion

Expand each class name via LLM (e.g., GPT) to $M$ candidate prompts $\mathcal T_c = \{t_{c,1},...,t_{c,M}\}$, yielding their embeddings $\mathbf E_c = \{\mathbf e_{c,m} = \phi_T(t_{c,m})\}_{m=1}^M$.

Each candidate is scored by mean Dice coefficient over $\mathcal R_c$:

$$
s_{c,m} = \frac{1}{|\mathcal R_c|} \sum_{(v, y) \in \mathcal R_c} \mathrm{Dice}\bigl( f_{\Phi}(v, \mathbf e_{c,m}),\, y \bigr)
\tag{4}
$$

Fuse the top $J$ candidates via temperature-softmax:

$$
\mathbf e_c^* = \sum_{j \in \mathrm{Top}_J(s_{c,\cdot})} w_{c,j}\, \mathrm{Norm}(\mathbf e_{c,j}),\quad
w_{c,j} = \frac{\exp(s_{c,j}/\tau)}{\sum_{k}\exp(s_{c,k}/\tau)}
\tag{5}
$$

Typically, $J=M$ and $\tau=1.0$.

## 3. Algorithmic Workflow and Pseudocode

ConceptBank construction consists of three main stages: prototype estimation, representative support mining, and concept fusion. The implementation is parameter-free with respect to SAM3 and requires no gradient-based updates.

**Algorithm Outline:**
```python
def build_concept_bank(D_sup, C, Phi, K=10, M=10, J=None, tau=1.0):
    # Stage I: Prototype Estimation
    Z = {c: [] for c in C}
    for x, masks in D_sup:
        for c in C:
            if c in masks:
                for (v, y_crop) in extract_crops(x, masks[c]):
                    z = mask_pool(Phi, v, y_crop)  # Eq.(1)
                    Z[c].append(z)
    P = {c: normalize(mean(Z[c])) for c in C}  # Eq.(2)

    # Stage II: Representative Mining
    R = {}
    for c in C:
        sims = [cos(z, P[c]) for z in Z[c]]
        idx = topk_indices(sims, k=min(K, len(sims)))
        R[c] = [Z[c][i] for i in idx]

    # Stage III: Concept Fusion
    B = {}
    for c in C:
        T_c = expand_prompts_LLM(c, M)
        E_c = [Phi.phi_T(t) for t in T_c]
        S = []
        for e in E_c:
            dice_vals = []
            for z in R[c]:
                y_hat = Phi.f_Phi(v, e)
                dice_vals.append(dice_coeff(y_hat, y))
            S.append(mean(dice_vals))
        Jc = topk_indices(S, k=J or M)
        ws = softmax([S[j]/tau for j in Jc])
        e_star = sum(w * normalize(E_c[j]) for w,j in zip(ws, Jc))
        B[c] = e_star
    return B
```

**Inference:** Store concept bank $\mathcal B = \{\mathbf e_c^*\}$ as a $|\mathcal C| \times d$ matrix; at test time, for a new image $x$, compute all class masks via
$$
\hat y^c = f_{\Phi}(x, \mathbf e_c^*)
$$
in a single forward pass.

## 4. Experimental Evaluation and Impact

### 4.1 Domains and Benchmarks

ConceptBank is evaluated on diverse OVS datasets that pose both data and concept drift:

- **Natural-scene OVS:** Pascal VOC21, Pascal Context60, COCO-Object, VOC20, PC59, COCO-Stuff, Cityscapes, ADE20K.
- **Remote-sensing OVS:** LoveDA, Potsdam, Vaihingen, iSAID.

These benchmarks introduce visual and semantic shifts, including overhead perspectives and varying class definitions.

### 4.2 Quantitative Results

| Scenario        | Reference (SAM3, %) | ConceptBank (%) | Gain (%) |
|-----------------|--------------------|-----------------|----------|
| Natural Scenes  | 57.5               | 67.1            | +9.6     |
| PC60            | 46.1               | 56.5            | +10.4    |
| Cityscapes      | 49.5               | 62.3            | +12.8    |
| Remote Sensing  | 39.1               | 52.1            | +13.0    |
| LoveDA          | 35.6               | 49.4            | +13.8    |
| iSAID           | 17.7               | 35.4            | +17.7    |

Over eight natural scene and four remote-sensing datasets, ConceptBank consistently yields a substantial increase in mean IoU (mIoU), outperforming vanilla SAM3 and prompt expansion baselines by up to $+17.7\%$ on challenging settings.

### 4.3 Ablation Analysis

Component-wise contributions to overall gain (natural scenes / remote sensing):

- Prompt expansion alone: $+0.5\%$ / $+3.1\%$
- + Prototype anchoring: $+2.6\%$ / $+6.4\%$
- + Representative mining: $+5.3\%$ / $+9.1\%$
- + Concept fusion (full pipeline): $+7.2\%$ / $+13.0\%$

This establishes the necessity of all three stages—each improves drift robustness incrementally.

## 5. Practical Integration and Efficiency

ConceptBank is designed for seamless integration with frozen foundation models such as SAM3:

- **Offline:** Execute `build_concept_bank(...)` once per target dataset to generate $\mathcal B$.
- **Online:** Replace standard SAM3 text embedding inference with precomputed $\mathbf e_c^*$ entries for all segmentation queries.
- **Hyperparameters:** Defaults—$K=10$ (robust in $[5,30]$), $M \approx 4{-}10$ (number of prompt variants), $J = M$, $\tau=1.0$.
- **Prompt Expansion:** Supply the official dataset label definitions to the LLM; restrict alterations to synonyms or concise attribute phrases.
- **Efficiency:** No gradients or model parameter updates are required. Inference is $1.25\times$ faster than single-prompt SAM3 and considerably faster than multi-prompt ensembling.

A plausible implication is that the parameter-free nature and efficiency enable rapid, robust adaptation to novel domains without retraining or extensive prompt engineering.

## 6. Significance and Limitations

ConceptBank establishes a new baseline for distribution-robust OVS. Its three-stage pipeline—prototype anchoring, representative mining, and LLM-guided concept fusion—directly addresses two principal modes of drift, restoring alignment between prompts and visual evidence in-the-wild scenarios. All empirical results are obtained with no changes to the base SAM3 model weights.

A potential limitation is the reliance on sufficient support set annotation and the quality of LLM-generated prompt expansions. The method’s performance depends on how well the support data and prompt variants span the shifted distributions. However, empirical results suggest robustness to support size and prompt configuration within broad default settings, across diverse domain shifts [2602.06333].

Source: https://www.emergentmind.com/topics/conceptbank