---
title: State-Enhanced Semantic Prototypes (SESP)
url: https://www.emergentmind.com/topics/state-enhanced-semantic-prototypes-sesp
type: topic
---

# State-Enhanced Semantic Prototypes (SESP)

State-Enhanced Semantic Prototypes (SESP) constitute a prototype construction paradigm specifically designed to address limitations of static semantic prototypes in weakly supervised open-vocabulary object detection (WS-OVOD). SESP explicitly injects state information into semantic representations, enhancing cross-modal alignment between visual region features and their corresponding textual prototypes. This approach enables detectors to capture intra-class visual variability driven by different object states—such as poses or activities—by aggregating state-aware textual descriptions into rich, discriminative prototypes that improve generalization to novel and weakly labeled object categories [2511.18012].

## 1. Motivation: Limitations of Static Semantic Prototypes

Conventional open-vocabulary object detectors employ text embeddings of category names (e.g., “cat,” “dog”) as semantic prototypes to align with visual region features. Such static prototypes are sensitive to intra-class variation, as they cannot accommodate significant state-driven visual differences (e.g., a cat at rest versus mid-leap). Previous prompt-enrichment methods generate generic, state-agnostic descriptions (“a small furry feline”), failing to capture nuanced pose, activity, or state-induced appearance shifts. The SESP framework advances beyond state-agnostic prototypes by incorporating explicit state information, creating prototypes that are naturally more discriminative and better aligned with the diversity of real-world object appearances [2511.18012].

## 2. Construction of State-Aware Textual Descriptions

SESP leverages a large language model (GPT-4) deployed via API to generate concise, visually descriptive sentences for each class in its prevalent states. For a given class $C$, a two-part prompting strategy is utilized:

- Query for common states or forms:  
  "What are the common states or forms of $C$? For each common state of $C$, provide a one-sentence description of its visual appearance."

- Query for a generic appearance description:  
  "What does $C$ generally look like? Provide a one-sentence generic description of its appearance."

The model responds with $K$ state phrases (e.g., “a sleeping cat,” “a running dog”) and a broad generic description (e.g., “a small furry animal often found in homes”). These responses provide explicit state information and general category cues as textual inputs to the prototype formulation process [2511.18012].

## 3. Prototype Formulation and Aggregation

For each class $c$, the $K$ generated state descriptions $S_c = \{s_1, \ldots, s_K\}$ and the generic description $des_c$ are embedded via a frozen pretrained text encoder, $f_t(\cdot)$, specifically the CLIP (ViT-B/32) text encoder:

- $t_{c,s_k} = f_t(s_k)$ for $k = 1, \ldots, K$
- $t_{c,des} = f_t(des_c)$

Each vector may be $\ell_2$-normalized for unit length: $\hat{t}_{c,s_k} = t_{c,s_k} / \| t_{c,s_k} \|_2$.

The embeddings are aggregated using mean pooling, combining both state-specific and generic information:

$$
P_c = \frac{1}{K+1} \left( t_{c,des} + \sum_{k=1}^K t_{c,s_k} \right)
$$

Here, $P_c$ is a class-specific, state-enhanced prototype in the CLIP latent space, encoding both general and state-conditioned cues for improved discriminability [2511.18012].

## 4. Integration into Training and Classification Loss

SESP prototypes are incorporated into the classification objective for open-vocabulary detection models. For region proposal $i$ with visual feature $f_i$ and ground-truth class $y_i$, classification employs a cosine-based head with temperature $\tau$:

$$
\mathcal{L}_{SESP} = -\sum_i \log 
\frac{\exp(f_i^\top P_{y_i} / \tau)}
{\sum_{c \in \mathcal{C}} \exp(f_i^\top P_c / \tau)}
$$

- $f_i \in \mathbb{R}^D$: vision feature for proposal $i$
- $P_c \in \mathbb{R}^D$: SESP of class $c$
- $y_i$: ground-truth class label for proposal $i$
- $\mathcal{C}$: category set
- $\tau$: softmax temperature (default $\sim$0.01 from CLIP)

This loss function simultaneously governs fully-supervised and weakly supervised branches, enforcing alignment between region features and state-aware prototypes throughout the training regime [2511.18012].

## 5. Implementation Details

Key components of the SESP pipeline include:

| Component                | Implementation                     | Note                                           |
|--------------------------|------------------------------------|------------------------------------------------|
| Text Encoder             | CLIP (ViT-B/32), frozen            | Ensures consistency and compatibility          |
| LLM for Description      | GPT-4 via API                      | Offline generation, prompted per class         |
| Number of States $K$     | 5 or 7                             | Ablation finds optimal trade-off at $K=7$      |
| Aggregation              | Mean pooling                       | Simple, stable, and effective                  |
| Temperature $\tau$       | $\sim$0.01                         | Adopted from CLIP definition                   |
| Storage                  | Prototypes cached offline           | One $P_c$ per class in vocabulary              |

Prompt responses for all classes are generated and cached once per training session, ensuring efficient prototype lookup and stability across runs. No fine-tuning of the text encoder is performed during model training [2511.18012].

## 6. Empirical Evaluation and Performance Impact

The inclusion of SESP in the Detic baseline on the OV-COCO WS-OVOD benchmark results in an increase of +1.8 APⁿ₅₀ on novel classes, demonstrating the practical benefit of modeling intra-class state variation. When $K=7$, the performance peaks; values of $K > 7$ yield marginal decreases attributable to noisy or redundant state descriptions. Ablation studies show that alternative aggregation methods (median, two-stage mean, similarity-weighted mean) lead to only minor variation ($\pm$0.2 APⁿ₅₀), indicating robustness of mean pooling.

In zero-shot evaluation on Objects365, SESP improves AP_r (rare classes) from 12.4 to 13.7 over the Detic baseline. Combined with Scene-Augmented Pseudo Prototypes (SAPP), the approach achieves a cumulative +3.5 APⁿ₅₀ improvement, establishing the importance of integrating both state-aware and context-aware semantics for robust weakly supervised open-vocabulary detection [2511.18012].

Source: https://www.emergentmind.com/topics/state-enhanced-semantic-prototypes-sesp