Ov-GSR: Open-vocabulary Grounded Situation Recognition

• The paper introduces Ov-GSR, a framework that parses images into verb-centric frames with open-set recognition of actions, roles, and entities.
• It employs modular explainers that integrate language models, CLIP, and Grounding DINO to enhance verb disambiguation and role grounding.
• Experimental evaluations on SWiG and Ov-SWiG benchmarks show improved zero-shot and generalized recognition for seen, rare, and unseen situations.

Searching arXiv for the cited Ov-GSR and related GSR papers to ground the article in the latest relevant literature. arXiv search query: (Liu et al., 2023) Open Scene Understanding Grounded Situation Recognition Segment Anything Open-vocabulary Grounded Situation Recognition (Ov-GSR) is a generalized form of grounded situation recognition in which an image is parsed into a structured situation representation that includes a salient activity, verb-specific semantic roles, entity labels, and spatial grounding, while extending beyond conventional closed-set training assumptions. Across recent formulations, the task is framed either as zero-shot grounded situation recognition without task-specific training, or as generalized recognition over both base and unseen situations at test time. In both cases, Ov-GSR departs from standard GSR by treating verb understanding, role grounding, and noun recognition as open-set or unseen-composition problems rather than as prediction over a fixed training vocabulary alone (Lei et al., 2024, Cai et al., 19 Jul 2025).

1. Formal task definition and representational structure

Grounded situation recognition represents an image through a verb-centric frame. One formulation defines the output for an image $I$ as

$\mathcal{F}_v = \{y_v, \mathcal{FR}_v\},$

where $y_v \in \mathcal{V}$ is the predicted verb and

$$\mathcal{FR}_v = \{(r, y_n, \mathbf{b}) \mid r \in \mathcal{R}_v\}$$

is the set of filled semantic roles for that verb, with role $r$, noun $y_n \in \mathcal{N}$, and bounding box $\mathbf{b} \in \mathcal{B}$. A closely related formulation writes a situation as

$$s = \{v, \mathcal{F}_v\}, \qquad \mathbf{f}_r = (r, n_r, c_r),$$

where $c_r \in \mathbb{R}^4$ denotes bounding box coordinates for the entity filling role $r$ (Lei et al., 2024, Cai et al., 19 Jul 2025).

These formulations make clear that Ov-GSR is not merely an action classifier. It is a structured prediction problem with three coupled subproblems: verb recognition, semantic role grounding, and noun recognition. The coupling is central: role sets are induced by verb templates, noun predictions are conditioned on role and scene context, and grounding must align role semantics with spatial evidence. This suggests that Ov-GSR inherits the ontological structure of FrameNet-style situation recognition while pushing toward open-set generalization in both semantics and grounding (Lei et al., 2024).

2. Open-vocabulary regimes and their relation to conventional GSR

Conventional GSR is typically trained in a closed-set regime in which the activity and role/entity vocabulary at test time is assumed to match training. Ov-GSR modifies that assumption. In the zero-shot formulation, the model must solve the full structured prediction problem without training on GSR annotations. In the generalized formulation proposed later, the model is trained on base situations

$\mathcal{F}_v = \{y_v, \mathcal{FR}_v\},$0

but is evaluated over

$\mathcal{F}_v = \{y_v, \mathcal{FR}_v\},$1

so inference may include unseen situations $\mathcal{F}_v = \{y_v, \mathcal{FR}_v\},$2 in addition to seen ones (Lei et al., 2024, Cai et al., 19 Jul 2025).

Two technical motivations recur in the literature. First, closed-set GSR degrades on unseen situations because it cannot generalize to activities or role combinations not seen in training. Second, it is biased toward frequent situations and under-recognizes rare cases. A separate but related diagnosis in the zero-shot setting identifies three weaknesses of direct class prompting: ambiguous verbs, rigid grounding templates, and context-free noun classification. Taken together, these analyses position Ov-GSR as an attempt to move beyond label matching toward compositional, language-mediated, and scene-aware reasoning over structured situations (Lei et al., 2024, Cai et al., 19 Jul 2025).

A useful boundary case is the distinction between genuinely open-vocabulary semantics and merely large-vocabulary or open-scene behavior. A system can operate over many verbs, roles, and entities or use class-agnostic segmentation, yet still remain semantically closed-set if its verb and noun predictions are restricted to a predefined ontology. This distinction becomes important when comparing Ov-GSR with adjacent work such as assistive open-scene understanding (Liu et al., 2023).

3. Zero-shot Ov-GSR via language explanation

The paper "Seeing Beyond Classes: Zero-Shot Grounded Situation Recognition via Language Explainer" introduces LEX, a zero-shot/Ov-GSR pipeline built from off-the-shelf components and an auxiliary LLM that generates explanation-rich prompts for all three stages of GSR. The baseline uses CLIP for verb and noun similarity, Grounding DINO for role localization, class-based prompts such as “a photo of [VERB CLASS]” or “a photo of [NOUN CLASS],” and verb-centric templates for grounding. LEX replaces these sparse prompts with generated natural-language explanations through three modules: a verb explainer, a grounding explainer, and a noun explainer (Lei et al., 2024).

The verb explainer generates general verb-centric descriptions for each verb class,

$\mathcal{F}_v = \{y_v, \mathcal{FR}_v\},$3

and combines them with the original class prompt using a training-free weighting scheme based on discriminability. The weighting is computed from the entropy of class-normalized similarities between generated descriptions and scene descriptions, so lower-entropy descriptions receive larger weights. Verb scoring then interpolates between class-name similarity and weighted explanation similarity: $\mathcal{F}_v = \{y_v, \mathcal{FR}_v\},$4 The explicit purpose is to disambiguate nearby verbs such as shopping and buying by injecting richer semantic cues than a bare class label (Lei et al., 2024).

The grounding explainer rephrases the verb-centric template while preserving role words,

$\mathcal{F}_v = \{y_v, \mathcal{FR}_v\},$5

and submits each rephrased prompt to Grounding DINO: $\mathcal{F}_v = \{y_v, \mathcal{FR}_v\},$6 For each role, the system retains the highest-confidence box among the candidate prompts. This directly addresses sensitivity of grounding models to awkward or rigid template wording. The noun explainer then generates scene-specific noun descriptions conditioned on the verb template and role,

$\mathcal{F}_v = \{y_v, \mathcal{FR}_v\},$7

performs noun filtering with LLM-derived commonsense to discard implausible categories, and scores nouns using a prompt mixture analogous to verb scoring, with uniform weights

$\mathcal{F}_v = \{y_v, \mathcal{FR}_v\},$8

A final refinement step fills the template with predicted nouns and selects the globally most image-consistent sentence by maximizing image–text similarity (Lei et al., 2024).

The architecture is intentionally plug-and-play. CLIP is used for verb and noun similarity scoring, Grounding DINO for semantic role localization, and GPT-3.5-turbo for explanation generation. No retraining of CLIP or Grounding DINO is required. This suggests that one operative interpretation of zero-shot Ov-GSR is not the invention of a new end-to-end architecture, but the reparameterization of structured visual reasoning through richer language interfaces (Lei et al., 2024).

4. Distilling foundation-model knowledge into an Ov-GSR student

The paper "From Semantics, Scene to Instance-awareness: Distilling Foundation Model for Open-vocabulary Situation Recognition" introduces Multimodal Interactive Prompt Distillation (MIPD), which formulates Ov-GSR as recognition over seen, rare, and unseen situations and transfers the zero-shot generalization ability of a large multimodal teacher into a smaller GSR student suitable for deployment. The teacher is a frozen MLLM, instantiated as InstructBLIP in experiments, and the student is a lightweight CLIP ViT-L/14-based model with task heads. The training objective combines teacher-guided distillation with supervised situation prediction: $\mathcal{F}_v = \{y_v, \mathcal{FR}_v\},$9 with final loss

$y_v \in \mathcal{V}$0

The objective explicitly targets generalization beyond the base situation set while retaining standard GSR supervision (Cai et al., 19 Jul 2025).

A central component is the LLM-based Judgmental Rationales Generator (JRG). It produces positive and negative glimpse and gaze rationales,

$y_v \in \mathcal{V}$1

where glimpse denotes coarse scene-level understanding and gaze denotes finer instance-level focus. The procedure generates pseudo rationales with an MLLM, scores them with an LLM judge, refines those below a threshold, and constructs negatives by semantically similar but incorrect modifications to activity, entities, or attributes. The paper mentions a typical judge-score cutoff such as $y_v \in \mathcal{V}$2. Negative rationales are then incorporated through

$y_v \in \mathcal{V}$3

so that the student is trained not only toward positive semantics but also against deceptively similar errors (Cai et al., 19 Jul 2025).

MIPD additionally introduces two learnable prompts: a scene-aware prompt $y_v \in \mathcal{V}$4, attached to the edges of the visual feature map to capture holistic context, and an instance-perception prompt $y_v \in \mathcal{V}$5, built from instance coordinates to focus on regional entity understanding. These prompts are aligned with positive rationales through Negative-Guided Multimodal Prompting Alignment (NMPA): $y_v \in \mathcal{V}$6

$y_v \in \mathcal{V}$7

The distillation loss

$y_v \in \mathcal{V}$8

matches student features to the teacher’s aligned multimodal representations. Rationales are not used at inference time; only the lightweight student predicts the activity, role/entity labels, and bounding boxes (Cai et al., 19 Jul 2025).

This framework shifts Ov-GSR from prompt engineering alone to teacher–student transfer of structured multimodal knowledge. The semantic content transferred includes contextual rationales, holistic scene cues, instance-level perception, and contrastive information from negatives. A plausible implication is that MIPD treats open-vocabulary generalization not as direct zero-shot inference by a foundation model, but as compression of foundation-model reasoning into a deployable structured predictor (Cai et al., 19 Jul 2025).

5. Datasets, metrics, and reported empirical behavior

The primary benchmark across these studies is SWiG, the grounded extension of imSitu. One report describes SWiG with train/val/test splits of 75K / 25K / 25K, 278,336 added bounding boxes, and totals of 504 verbs, 190 roles, and 11,538 entities. Another describes SWiG as containing 25,200 test images, 504 verb categories, 190 semantic role categories, and 9,929 noun entity categories, with about 69.3% of semantic roles having bounding-box annotations, 1–6 roles per image with average 3.55, and three annotated frames from three annotators. The open-vocabulary distillation work evaluates on a refined Ov-SWiG split with seen, rare, and unseen settings (Liu et al., 2023, Lei et al., 2024, Cai et al., 19 Jul 2025).

Standard evaluation uses five situation-recognition metrics. In one naming convention these are verb, value, val-all, grnd, and grnd-all; in another they are verb, value, value-all, grounded-value, and grounded-value-all. Results are typically reported under Top-1-Verb, Top-5-Verb, and Ground-Truth-Verb settings, with downstream metrics counted as incorrect if the verb is wrong in Top-1 or Top-5 evaluation. Grounding accuracy requires IoU $y_v \in \mathcal{V}$9, and one paper states the equivalent criterion conceptually as noun-correct plus box IoU $$\mathcal{FR}_v = \{(r, y_n, \mathbf{b}) \mid r \in \mathcal{R}_v\}$$0 (Liu et al., 2023, Lei et al., 2024).

Paper	Benchmark setting	Representative reported result
LEX (Lei et al., 2024)	SWiG, Top-1-Verb	verb 32.41, value 9.37, val-all 1.61, grnd 7.26, grnd-all 0.98
MIPD (Cai et al., 19 Jul 2025)	Ov-SWiG test, Top-1-all	verb 41.96, value 34.11, val-all 21.56, grnd 22.86, grnd-all 7.57
OpenSU (Liu et al., 2023)	SWiG test, Top-1 Verb	verb 50.10, value 41.20, value-all 26.56, grounded-value 34.27, grounded-value-all 15.70

LEX substantially outperforms class-prompt baselines on SWiG. Under Top-1-Verb, CLS reports verb 30.18, value 4.70, val-all 0.16, grnd 3.09, and grnd-all 0.07, whereas LEX reports 32.41, 9.37, 1.61, 7.26, and 0.98. Under Ground-Truth-Verb, CLS reports value 13.51, val-all 0.42, grnd 9.05, and grnd-all 0.14, while LEX reports 29.92, 4.68, 23.57, and 3.08. Ablations attribute gains to all three explainers, with noun filtering especially important, and improvements remain consistent across CLIP backbones including ViT-B/16, ViT-B/32, ViT-L/14, and ViT-L/14@336px (Lei et al., 2024).

MIPD reports the best performance on Ov-SWiG across seen, rare, and unseen settings. On the dev set, Top-1-all scores are verb 41.87, value 34.29, val-all 22.02, grnd 23.29, and grnd-all 7.85; rare scores are verb 29.10, value 23.58, grnd 14.65; unseen scores are verb 7.80, value 4.73, grnd 3.97. On the test set, Top-1-all scores are verb 41.96, value 34.11, val-all 21.56, grnd 22.86, and grnd-all 7.57; rare scores are verb 28.30, value 22.37, grnd 13.59; unseen scores are verb 7.40, value 4.08, grnd 3.53. The paper also reports HICO-DET open-vocabulary HOI detection results of 17.84 unseen, 25.45 seen, and 23.96 full (Cai et al., 19 Jul 2025).

These results should be interpreted by protocol rather than as a single ranking. OpenSU reports higher closed-set SWiG numbers, but its own analysis emphasizes that it is not a true open-vocabulary GSR system in the modern sense; LEX evaluates zero-shot GSR without task-specific training; MIPD evaluates a generalized seen/rare/unseen Ov-GSR setting. This suggests that empirical comparison in this area depends as much on the openness assumption as on raw metric values (Liu et al., 2023, Lei et al., 2024, Cai et al., 19 Jul 2025).

6. Boundary cases, misconceptions, and relation to open-scene understanding

A recurring misconception is to equate any large-vocabulary or foundation-model-augmented GSR pipeline with Ov-GSR. The paper "Open Scene Understanding: Grounded Situation Recognition Meets Segment Anything for Helping People with Visual Impairments" is explicitly better understood as an assistive, scene-understanding extension of GSR rather than a true open-vocabulary grounded situation recognition system. Its OpenSU pipeline predicts verbs, roles, entity nouns, and bounding boxes with a GSR model, then passes those boxes as prompts to SAM to obtain dense instance masks. The semantic stage remains a fixed-label GSR problem over predefined verbs, roles, and entity classes derived from SWiG, even though SAM itself is class-agnostic and promptable (Liu et al., 2023).

Technically, OpenSU modifies CoFormer by replacing its CNN backbone with a Swin Transformer-Tiny backbone pretrained on ImageNet-1k, retaining a multi-decoder encoder–decoder design with Glance Transformer, Gaze-S1, and Gaze-S2, and replacing ReLU with GELU throughout the GSR decoders and classifiers: $$\mathcal{FR}_v = \{(r, y_n, \mathbf{b}) \mid r \in \mathcal{R}_v\}$$1

$$\mathcal{FR}_v = \{(r, y_n, \mathbf{b}) \mid r \in \mathcal{R}_v\}$$2

It uses the same losses as CoFormer—verb classification, noun classification, box existence, and box regression—and is trained end-to-end on SWiG with AdamW, batch size 4, 40 epochs, weight decay $$\mathcal{FR}_v = \{(r, y_n, \mathbf{b}) \mid r \in \mathcal{R}_v\}$$3, $$\mathcal{FR}_v = \{(r, y_n, \mathbf{b}) \mid r \in \mathcal{R}_v\}$$4, $$\mathcal{FR}_v = \{(r, y_n, \mathbf{b}) \mid r \in \mathcal{R}_v\}$$5, backbone learning rate $$\mathcal{FR}_v = \{(r, y_n, \mathbf{b}) \mid r \in \mathcal{R}_v\}$$6, decoder learning rate $$\mathcal{FR}_v = \{(r, y_n, \mathbf{b}) \mid r \in \mathcal{R}_v\}$$7, and a $$\mathcal{FR}_v = \{(r, y_n, \mathbf{b}) \mid r \in \mathcal{R}_v\}$$8 learning-rate decay after epoch 30. Runtime comparisons report 74.04 s and 1310.66M parameters for Grounding DINO + SAM + BLIP, 1.34 s and 738.09M parameters for OpenSU (SAM), and 0.70 s and 107.13M parameters for OpenSU (MobileSAM) (Liu et al., 2023).

The importance of this boundary case is conceptual. OpenSU advances grounding quality by replacing coarse boxes with pixel-wise masks and supports user-directed assistive interaction through region-of-interest selection methods such as fingertip pointing, head pose, laser pointer, or IMU-based direction cues. Yet its own description states that there is no mechanism to ground arbitrary user-specified or unseen entity categories via language prompts, no text-prompted open-vocabulary role/entity grounding benchmark, and no way for SAM to recover if upstream semantic prediction fails. By contrast, LEX targets zero-shot language-mediated structured prediction, and MIPD targets seen/rare/unseen generalization through distillation. The result is a clearer taxonomy: open-scene grounded situation segmentation, zero-shot Ov-GSR, and generalized unseen-situation Ov-GSR are related but non-identical problem settings (Liu et al., 2023, Lei et al., 2024, Cai et al., 19 Jul 2025).