---
title: Personalized OV Semantic Segmentation
url: https://www.emergentmind.com/topics/personalized-open-vocabulary-semantic-segmentation
type: topic
---

# Personalized OV Semantic Segmentation

Searching arXiv for recent papers on personalized open-vocabulary semantic segmentation and closely related open-vocabulary segmentation methods.
{"queries":[{"query":"\"Personalized OVSS: Understanding Personal Concept in Open-Vocabulary Semantic Segmentation\"","max_results":5},{"query":"\"Diffusion Models for Open-Vocabulary Segmentation\"","max_results":5},{"query":"\"Training-Free Generalized Few-Shot Segmentation through Open-Vocabulary Semantic Arbitration\"","max_results":5},{"query":"\"Open-Vocabulary Universal Image Segmentation with MaskCLIP\"","max_results":5}]}
Personalized open-vocabulary semantic segmentation extends open-vocabulary semantic segmentation from category-level text grounding to user-specific visual concepts. In the formulation introduced in "Personalized OVSS: Understanding Personal Concept in Open-Vocabulary Semantic Segmentation" [2507.11030], the task is defined by a small set of support image–mask pairs that specify a personal concept such as “my mug,” together with a query image and an open vocabulary of class names. The objective is twofold: to segment the personal concept accurately in the query image and to preserve the original open-vocabulary performance on the pre-existing vocabulary. The central difficulty is that standard OVSS can segment regions from arbitrarily given text descriptions, even for classes unseen during training, but it fails to understand personal texts such as “my mug cup” when multiple visually similar instances are present [2507.11030].

## 1. Task formulation and problem setting

The personalized OVSS formulation uses \(K\) support image–mask pairs \(\{(I_s^i,M_s^i)\}_{i=1}^K\) to define a user’s personal concept \(P\), a query image \(I_q\), and an open vocabulary \(\mathcal V=\{t_1,\dots,t_V\}\). The baseline OVSS backbone provides mask embeddings \(Z_{\rm open}\in\mathbb R^{N\times D}\) and mask proposals \(M_{\rm open}\in\mathbb R^{h\times w\times N}\). The open-vocabulary text side is represented by fixed CLIP-text embeddings \(T_{\rm open}\in\mathbb R^{V\times D}\), while personalization adds a learnable embedding \(T_{\rm per}\in\mathbb R^{1\times D}\) corresponding to a personal token such as “\(<\)my-mug\(>\)” [2507.11030].

After concatenating the open-vocabulary and personal text embeddings,
\[
T=[T_{\rm open};\,T_{\rm per}]\in\mathbb R^{(V+1)\times D},
\]
the model computes text–mask similarity
\[
S \;=\; T\,Z^{T}
\]
and predicts per-pixel class probabilities as
\[
P_{\rm pred} \;=\; \mathrm{softmax}\bigl(M\times S^T\bigr)\quad\in\mathbb R^{h\times w\times (V+1)}.
\]
The task therefore augments an existing OVSS model with a new personal class while retaining the original semantic label space [2507.11030].

A key point is that the personal concept is not merely a new category name. The support masks specify the user’s intended visual target, and the task is explicitly motivated by cases such as recognizing “my mug cup” among “multiple mug cups.” This suggests that personalized OVSS is a support-grounded extension of open-vocabulary segmentation rather than a simple prompt-engineering variant.

## 2. Relation to open-vocabulary and training-free segmentation

Personalized OVSS sits within a broader family of methods that treat segmentation as a function of free-form text or support-time conditioning rather than closed-set retraining. "Diffusion Models for Open-Vocabulary Segmentation" defines open-vocabulary segmentation as the task of segmenting anything that can be named in an image and proposes OVDiff, which synthesizes support image sets for arbitrary textual categories using Stable Diffusion v1.5, extracts total cross-attention attribution maps, builds foreground and background prototypes, and performs zero-shot segmentation by cosine nearest-prototype assignment in a frozen feature space [2306.09316]. Its support-set synthesis is class-specific and data-free, but it targets arbitrary textual categories rather than personal concepts.

"Open-Vocabulary Universal Image Segmentation with MaskCLIP" addresses semantic, instance, and panoptic segmentation for arbitrary categories of text-based descriptions at inference time. MaskCLIP uses a class-agnostic mask-proposal network followed by a MaskCLIP Visual Encoder that interleaves mask tokens with a frozen ViT-based CLIP image encoder, producing refined masks and mask features that are classified by similarity to CLIP text embeddings [2208.08984]. The paper notes that users can pass new category names such as “toy rabbit” versus “real rabbit,” “empty bottle” versus “filled bottle,” and even face identities like “Obama” versus “Biden,” without retraining. A plausible implication is that prior OVSS systems can support rich text-conditioned category distinctions while still lacking the support-grounded disambiguation required by personal concepts.

"Training-Free Generalized Few-Shot Segmentation through Open-Vocabulary Semantic Arbitration" studies a different but adjacent regime: generalized few-shot semantic segmentation solved through inference-time coordination of frozen priors. Open-V combines SAM3 Promptable Concept Segmentation with a \(K\)-shot CLIP support centroid and per-pixel semantic arbitration, introducing no trainable components and supporting arbitrary semantic categories at inference time [2606.09474]. It is explicitly described as suitable for use-cases where new categories must be added on the fly, including home inventory and wildlife species. This line of work shows that support information can be injected at inference time, but personalized OVSS focuses on adapting an OVSS model so that a personal token refers to a specific user-defined concept while preserving the existing open-vocabulary label set [2507.11030].

## 3. Prompt tuning for a personal token

The personalization method in [2507.11030] follows a standard CoOp-style prompt-tuning design. The personal text embedding is initialized as a single learnable token from
\[
T_{\rm per}\biglarrow \mathrm{E}_{\rm t}(\text{“a photo of my [class]”}),
\]
and then concatenated with the fixed open-vocabulary text embeddings:
\[
T \;=\;\bigl[T_{\rm open};\,T_{\rm per}\bigr]\;\in\;\mathbb R^{(V+1)\times D}.
\]
This augmented text matrix is passed through the unchanged OVSS text encoder and downstream segmentation head [2507.11030].

Under this formulation, per-pixel class probabilities are computed from text–mask similarities:
\[
S \;=\;T\,Z^T,
\qquad
P_{\rm pred}(x,y,i)\;=\;\frac{\exp\bigl(S_{i,j(x,y)}\bigr)}{\sum_{i'}\exp\bigl(S_{i',j(x,y)}\bigr)},
\]
where \(j(x,y)\) is the mask-proposal index covering pixel \((x,y)\). The trainable parameters are intentionally limited to the personal text prompt and the negative-mask heads described below, making the method a plug-in adaptation rather than a full OVSS retraining scheme [2507.11030].

The motivation for prompt tuning is that the existing CLIP-aligned language interface already provides a natural insertion point for a personalized concept. However, the paper reports that prompt tuning on only positive examples tends to overfit and produces high false-positive activation on unrelated instances of the same base class. That observation drives the two main architectural additions: negative mask proposal and visual embedding injection [2507.11030].

## 4. Negative mask proposal and visual embedding injection

The negative mask proposal mechanism is introduced to model “everything except the personal concept.” In the embedding branch, the method learns
\[
Z_{\rm neg} \;=\; W_Z\,Z_{\rm open},
\quad
W_Z\in\mathbb R^{1\times N},
\]
and forms the augmented mask-embedding set
\[
Z=[Z_{\rm open};\,Z_{\rm neg}]\in\mathbb R^{(N+1)\times D}.
\]
In parallel, the mask branch constructs
\[
M_{\rm neg} \;=\; W_{M}\ast M_{\rm open}
\quad\text{and}\quad
M=[M_{\rm open};\,M_{\rm neg}]\in\mathbb R^{h\times w\times (N+1)}.
\]
The negative-mask head is supervised by a binary-cross-entropy term against the complement of the ground-truth personal mask,
\[
\mathcal L_{\rm neg}^{(M)}
=\mathrm{BCE}\bigl(M_{\rm neg},\,1 - M_{\rm gt}\bigr),
\]
and by a second objective that pushes the negative-mask embedding to match all non-personal words equally:
\[
\mathcal L_{\rm neg}^{(Z)}
\;=\;
-\sum_{i=1,i\neq k}^{V}
\frac1{V-1}\,\log\!\bigl(S[i,j_{\rm neg}]\bigr).
\]
The stated purpose is to reduce false predictions that arise when the personal prompt over-activates on semantically related but non-personal objects [2507.11030].

Visual embedding injection augments the personal text token with masked visual evidence extracted by a frozen CLIP-image encoder \(E_v\). The masked support feature is
\[
F_{\rm per}
=
\frac{1}{\sum M'_{\rm gt}}
\sum_{x,y}\bigl(E_v(I_s)\bigr)_{x,y}\,\cdot\,M'_{\rm gt}(x,y)
\quad\in\;\mathbb R^D,
\]
where \(M'_{\rm gt}\) is the ground-truth mask resized to the CLIP feature resolution. The personal token is then replaced by
\[
T_{\rm per}^{\rm vis}
= \alpha\,F_{\rm per} \;+\;(1-\alpha)\,T_{\rm per}.
\]
This injects a small contribution of masked visual features into each dimension of the learned prompt token [2507.11030].

The two mechanisms have complementary roles in the ablations reported for CUB\(^\text{per}\). Prompt tuning alone improves recall but lowers precision; adding the negative mask raises precision and improves \( \mathrm{IoU}^{\rm per} \); visual injection alone improves recall on hard cases but can reduce \( \mathrm{IoU}^{\rm per} \). The full configuration combines the gains most effectively [2507.11030].

## 5. Objective, optimization, and empirical evaluation

The overall training objective combines the inherited segmentation losses of the underlying OVSS model with the new personalization-specific terms. The paper states that \(\mathcal L_{\rm dice}\), \(\mathcal L_{\rm bce}\), and \(\mathcal L_{\rm cls}\) are the per-mask Dice, BCE, and classification losses inherited from SAN/ODISE, and adds \(\mathcal L_{\rm neg}^{(M)}\), \(\mathcal L_{\rm neg}^{(Z)}\), and a small \(L_2\) regularizer on \(T_{\rm per}\) [2507.11030]. In practice, \(\lambda_1,\lambda_2,\lambda_3\) are set to the same values as the underlying OVSS model, \(\lambda_{\rm neg}^Z=0.1\), \(\lambda_{\rm neg}^M=10\mbox{--}500\) depending on dataset, \(\alpha=0.01\mbox{--}0.1\), and \(\lambda_{\rm reg}=1\mathrm{e}{-4}\).

Evaluation is performed on three newly established benchmarks. FSS\(^\text{per}\) uses 30 fine-grained classes from FSS-1000, with test sets containing equal images with and without the personal concept. CUB\(^\text{per}\) covers all 200 bird species from CUB-200. ADE\(^\text{per}\) uses 30 selected ADE-20K classes combined with open-vocabulary and personal masks. Personalization uses \(K=1,3,5\) support pairs, and the metrics are \( \mathrm{IoU}^{\rm per} \) for the personal mask and mIoU over all open-vocabulary classes plus the new “my-concept” class [2507.11030].

| Dataset | Method | \( \mathrm{IoU}^{\rm per} \) / mIoU |
|---|---|---|
| FSS\(^\text{per}\) | SAN (base) | 41.08 / 55.68 |
| FSS\(^\text{per}\) | +Ours | 53.56 / 56.32 |
| CUB\(^\text{per}\) | SAN (base) | 68.25 / 77.32 |
| CUB\(^\text{per}\) | +Ours | 76.90 / 77.83 |
| ADE\(^\text{per}\) | SAN (base) | 6.88 / 17.20 |
| ADE\(^\text{per}\) | +Ours | 22.67 / 17.22 |

On CUB\(^\text{per}\) with \(K=5\) and the SAN backbone, the ablation sequence is especially informative. The base model reports mIoU \(77.32\), \( \mathrm{IoU}^{\rm per} \) \(68.25\), precision \(92.25\), and recall \(72.95\). Adding text-prompt tuning only yields mIoU \(77.89\), \( \mathrm{IoU}^{\rm per} \) \(69.70\), precision \(74.75\), and recall \(91.04\). Adding prompt tuning and negative mask yields mIoU \(77.89\), \( \mathrm{IoU}^{\rm per} \) \(73.71\), precision \(80.07\), and recall \(90.17\). Adding prompt tuning and visual injection yields mIoU \(77.65\), \( \mathrm{IoU}^{\rm per} \) \(65.94\), precision \(70.06\), and recall \(91.58\). The full method reaches mIoU \(78.29\), \( \mathrm{IoU}^{\rm per} \) \(76.80\), precision \(84.51\), and recall \(89.07\) [2507.11030].

These results are summarized in the paper as retaining original OVSS mIoU while boosting personal IoU by \(+8\mbox{--}16\) points. Qualitative examples are reported to show that the method can locate “my bird” among a flock and avoid false positives better than prompt-only tuning [2507.11030].

## 6. Strengths, limitations, and adjacent personalization paradigms

The method is described as a plug-in adapter that adds only \(\sim 0.4\) M parameters, requires a few annotated masks at personalization time, and incurs marginal inference overhead of \(< 5\) ms on a GPU [2507.11030]. Its main empirical strength is that it improves personal-concept segmentation while maintaining essentially unchanged open-vocabulary mIoU. The reported component-wise behavior is also specific: the negative-mask head cuts false positives, reflected in higher precision, while visual injection recovers recall on hard cases [2507.11030].

The limitations are equally explicit. The method requires a few annotated masks during personalization, and hyperparameters such as \(\lambda_{\rm neg}^M\) and \(\alpha\) need light tuning per dataset [2507.11030]. More generally, this indicates that personalized OVSS is not purely a text-only problem; it relies on support supervision to define what the personal concept means for a particular user.

Adjacent work highlights alternative forms of personalization and adaptation. Open-V shows that a user can supply any novel class examples at inference by computing a CLIP support centroid \(v_c\) from \(K\) masked exemplars, adding the new class name to the text prompt list, and rerunning the PCS-plus-arbitration pipeline, with no retraining [2606.09474]. OVDiff shows that one can synthesize a class-specific segmenter directly from text prompts using diffusion-generated support sets, CutLER proposals, and frozen feature extractors, again with zero additional training [2306.09316]. MaskCLIP, meanwhile, can accept online custom categories at inference and uses CLIP-aligned text strings to segment arbitrary categories in semantic, instance, and panoptic settings [2208.08984]. 

Taken together, these results suggest a stratification within open-vocabulary segmentation. One regime adds arbitrary semantic categories at inference time through frozen priors or synthesized supports; another, represented by personalized OVSS, uses a few support masks to redefine a text token as a user-specific visual concept while preserving the original vocabulary [2507.11030]. The latter addresses a narrower but practically important ambiguity: not “what category is present,” but “which one is mine.”

Source: https://www.emergentmind.com/topics/personalized-open-vocabulary-semantic-segmentation