ScenarioCLIP Model Overview

• ScenarioCLIP is a compositional visual-language model that extends classic CLIP by incorporating separate streams for global, object, and relation-level scene understanding.
• It employs multi-level contrastive and knowledge-distillation losses to align visual and textual embeddings across various hierarchical representations.
• Empirical results show robust improvements in zero-shot retrieval, scene-graph construction, and object detection, underscoring its effectiveness in structured visual analysis.

ScenarioCLIP refers to a family of compositional visual-LLMs that extend the CLIP architecture to structured scene understanding, particularly focusing on multi-object, multi-action, and explicit relation modeling for fine-grained image analysis. Designed to overcome limitations of standard CLIP models, which typically handle single-label or global image-text correspondence, ScenarioCLIP incorporates separate streams for global, object-level, and relation-level representations, supervised via multi-level contrastive and knowledge-distillation objectives. The model is pretrained on a large-scale Action-Genome dataset constructed with automated and semi-automated text-image relation curation, supporting robust zero-shot and fine-tuning performance on downstream tasks such as cross-modal retrieval, situation recognition, predicate classification, and scene-graph construction (Sinha et al., 25 Nov 2025, Roy et al., 2023).

1. Architecture and Model Design

ScenarioCLIP implements a “multi-stream” CLIP extension comprising parallel encoders for three distinct sources of compositional visual and textual information:

• Global Scene Stream: Encodes the entire image and a corresponding compositional caption using visual and textual encoders initialized from CLIP ViT-B/32 weights, each outputting a 512-dimensional embedding.
• Object Stream: Processes individual object crops (ROIs) and object names, with each object instance encoded separately and mapped to a 512-dimensional embedding via the same backbone.
• Relation Stream: Each relation consists of a triplet (object₁, predicate, object₂) that is visually grounded using focused region masks. These regions are produced by detecting bounding boxes with GroundingDINO, segmenting with SAM, and combining the two object masks using RBF weighting centered at the object centroids.

Each visual stream has a corresponding text stream for syntactically aligned modeling—captions for global, canonical object names for objects, and canonicalized relation triplets for relations. Fusion is handled without additional transformer layers; instead, hierarchical and cross-grained information are aligned via multi-level contrastive learning and an exponential moving average (EMA) knowledge-distillation module, facilitating soft alignment of global and local embeddings via Kullback-Leibler divergence (Sinha et al., 25 Nov 2025).

2. Pretraining Objectives and Supervision

ScenarioCLIP’s learning is orchestrated through two principal loss components:

• Contrastive Alignment Loss ($\mathcal{L}_{CA}$): Extending classic symmetric CLIP contrastive learning, ScenarioCLIP applies this loss independently for global, object, and relation levels. For a batch of $B$ scenes, the similarity is

$\text{sim}(x, y) = \frac{x \cdot y}{\tau}$

where $\tau$ is a learnable temperature. $\mathcal{L}_{CA}$ sums the standard CLIP loss over global, object ($n_O$ instances), and relation ($n_R$ triplets) streams.

• Knowledge-Distillation Loss ($\mathcal{L}_{KD}$): An EMA “teacher” network provides averaged embeddings over the training trajectory. $\mathcal{L}_{KD}$ enforces consistency
  • from the global teacher to local student (visual),
  • and from the local teacher to global student (text),
  • using KL divergence between $\ell_2$-normalized embedding vectors.

The full pretraining objective is:

$$\mathcal{L}_{\text{total}} = \lambda_{CA} \mathcal{L}_{CA} + \lambda_{KD} \mathcal{L}_{KD}$$

with tunable weights. A plausible implication is that this dual-headed loss aligns representations at both high-level scenario and fine-grained entity levels, facilitating compositional generalization to unseen scenes and relations (Sinha et al., 25 Nov 2025).

3. Action-Genome Dataset Construction

ScenarioCLIP relies on a large-scale, custom-constructed dataset. The Action-Genome dataset is obtained using a three-stage procedure:

1. Vision-LLM Annotation: Images are annotated by Ovis-Gemma 9B with global action captions, object lists, and positive relation triplets. Hard negatives are generated by manipulating triplet structure and substituting predicates with antonyms.
2. Object Grounding: Objects are grounded in images via GroundingDINO, producing bounding boxes.
3. Relation Region Generation: Segmentation masks (SAM) for each object are blended with an RBF weighting centered at the two objects to obtain focused relation regions, masking out unrelated pixels and blurring backgrounds.

The vocabulary is normalized by stripping adjectives and colors (NLTK POS), collapsing singular/plural forms, embedding with BERT, clustering via HDBSCAN, and assigning canonical classes. The resulting dataset contains 615,805 images, 740 action classes, 4,812 object classes, and 225,609 relation classes (Sinha et al., 25 Nov 2025).

4. Downstream Applications and Fine-Tuning

ScenarioCLIP supports a suite of scenario-based visual understanding tasks:

• Zero-Shot Retrieval: Using cosine similarity at each embedding level for global, object, and relation retrieval.
• Linear-Probe Classification: Freezes encoders; attaches shallow classification heads for global actions, objects, or relations, trained with cross-entropy.
• Object Detection: Uses the global encoder for RPN backbone and object encoder for the RoI feature head within Faster R-CNN; supports feature pyramid networks and copy-paste data augmentation.
• Predicate and Scene-Graph Classification: Predicate classification is performed given ground-truth boxes by encoding region unions and similarity ranking; scene-graph classification predicts object categories and proceeds similarly.
• Relation Localization: A convolutional decoder maps relation-level patch features to binary masks optimized with $\ell_2$ loss (Sinha et al., 25 Nov 2025).

5. Empirical Performance

On the Action-Genome test split, ScenarioCLIP achieves substantial gains over PyramidCLIP and global CLIP baselines:

Task	PyramidCLIP	ScenarioCLIP w/o KD	ScenarioCLIP
Zero-Shot Action Top-1	54.95	57.52	57.63
Zero-Shot Object Top-1	37.67	51.26	51.86
Zero-Shot Rel. Top-1	16.43	19.45	19.56
Linear-Probe Action Top-1	75.36	79.46	79.19
Linear-Probe Object Top-1	79.74	84.25	84.30
Linear-Probe Rel. Top-1	40.78	44.75	44.71

Predicate classification and scene-graph tasks benefit as well: Predicate R@1 increases from 30.33 to 34.84, and scene-graph R@1 from 25.64 to 31.34. Object detection AP matches or slightly exceeds PyramidCLIP. Relation localization with a specialized encoder achieves lower MAE compared to PyramidCLIP in the unfrozen setting. This suggests that explicit region-level modeling and hierarchical alignment provide measurable improvements in compositional scene analysis (Sinha et al., 25 Nov 2025).

6. Ablation and Analysis

Ablation studies indicate that the knowledge-distillation component ($\mathcal{L}_{KD}$) primarily sharpens the retrieval embedding space, with removal leading to a ~0.6% drop in zero-shot object Top-1 accuracy and a 0.2 point reduction in box AP$_{50}$. Alternative distillation weight schedules yield only marginal gains. Region-level contrastive learning and hard negative mining boost relation R@1 by ~3 points. Qualitative analyses—t-SNE visualization, Grad-CAM saliency—demonstrate tighter clustering and sharper localization for ScenarioCLIP compared to PyramidCLIP. Typical failure modes such as rare, small or occluded object mislocalization and long-tailed predicates remain challenging (Sinha et al., 25 Nov 2025).

7. Relationship to Situation Recognition and ClipSitu Models

ScenarioCLIP subsumes earlier CLIP-based situation recognition approaches such as the ClipSitu models (“ClipSitu MLP” and “ClipSitu XTF”). ClipSitu reframes situation recognition in a classical FrameNet-style setup, where images are assigned a verb (activity) and a set of semantic role–noun pairs, using frozen CLIP features as image and text tokens (Roy et al., 2023):

• ClipSitu MLP: Concatenates pooled CLIP image features, verb embeddings, and role embeddings, passing them through deep MLP blocks for noun prediction.
• ClipSitu XTF: Employs a one-stage cross-attention transformer where verb-role queries attend to CLIP patch embeddings (ViT patches), supporting semantic role labelling and localization.

These architectures obtain state-of-the-art results on imSitu, e.g., ClipSitu XTF with ViT-L14@336px backbone achieves Top-1 value accuracy of 47.23% (+14.1pp over CoFormer), Top-1 all-roles-correct of 29.73%, and verb prediction of 58.19%. A plausible implication is that incorporating CLIP as a universal embedding space, with role-specific conditioning and cross-modal fusion, is highly effective for conditional structured prediction tasks (Roy et al., 2023).

8. Implementation and Resources

ScenarioCLIP is implemented in PyTorch with HuggingFace Transformers (clip-vit-base-patch32 backbone) and optional Lightning integration. Pretraining is performed on 8×A100 GPUs (40GB) over 12 epochs (~48h wall clock). Optimizer settings use AdamW (base lr $2 \times 10^{-5}$, cosine schedule), EMA decay of 0.9995, and separate low-lr schedules for temperature parameters. Downstream task scripts and pretrained models are available at https://github.com/scenario-clip/ScenarioCLIP. Action-Genome splits and evaluation tools are included (Sinha et al., 25 Nov 2025).

---

ScenarioCLIP introduces explicit multi-level visual-text alignment and hierarchical knowledge distillation, leveraging a curated large-scale compositional dataset to achieve superior compositional scene understanding, situating it as a key advancement in the extension of CLIP models for fine-grained scenario analysis and structured visual understanding.