- The paper introduces the SWIM framework that enhances video object understanding by aligning textual object nouns with precise spatial visual features.
- It employs explicit supervision using the NL-Refer dataset and cross-modal attention regularization, achieving significant improvements on benchmarks like VideoRefer-Q.
- Results demonstrate robustness and scalability, enabling practical, human-centric video analysis while motivating further multimodal alignment research.
See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understanding
The SWIM (See What I Mean) framework addresses a core deficiency in multimodal LLMs (MLLMs): their inability to reliably conduct fine-grained object understanding in video solely from natural language prompts. While modern MLLMs such as Qwen2.5-VL leverage large-scale pretraining to achieve general scene comprehension, these systems exhibit a pronounced semantic misalignment—object nouns in user inputs frequently result in diffuse and imprecise attention patterns, in contrast to the more localized activations produced by attribute words. This systemic discordance, rooted in the disparate roles of attributes (low-level, spatially-anchored features) and object nouns (high-level, context-dependent representations), limits the practical referential transparency of MLLMs for object-centric video understanding.
Dataset Construction: NL-Refer
SWIM relies on explicit training supervision to bridge the vision-language alignment gap for object-level grounding. To enable this, the authors introduce NL-Refer, an enriched dataset derived from VideoRefer, where every video frame is paired with a human message and a precise object-level mask. Crucially, the data refinement pipeline employs a GPT-4o-based system to synthesize referring expressions that replace generic placeholders with detailed yet concise descriptors of each target object, and marks the core object noun using special tokens. By doing so, each textual reference in the training set is deterministically matched to a concrete spatial region, facilitating direct cross-attention analysis and targeted supervision.
Methodology: Cross-modal Attention Regularization
SWIM applies an auxiliary attention supervision mechanism during fine-tuning:
- Tagging and Alignment: Each natural language prompt in NL-Refer is tokenized, with the core object noun assigned an explicit tag. The model extracts cross-attention maps from the object noun token to all visual (patch-level) tokens.
- Spatial Consistency: The extracted attention maps are upsampled (if necessary) and directly aligned with the ground-truth segmentation masks via pixel-wise binary cross-entropy loss. By aggregating attention across multiple sparsely-spaced decoder layers and employing mean fusion, SWIM provides a robust consistency signal, guiding the model toward stable, layer-invariant noun-to-region grounding.
- No Architectural Overhead: Critically, the supervision is applied solely during training, with no modifications required to the MLLM architecture or visual encoders, and no visual prompts (masks, points, or boxes) needed at inference.
Empirical Results
Quantitative and qualitative analyses underscore the efficacy of SWIM's textual prompt-based approach:
- Fine-Grained VideoRefer Benchmark: On region-specific QA and description benchmarks, SWIM exhibits significant performance gains. For example, SWIM achieves 78.3% accuracy on VideoRefer-Q (a +6.4% improvement over VideoRefer-7B), and an average score of 3.78 on VideoRefer-D, outperforming both visual-prompt-based baselines and prior state-of-the-art MLLMs. Subject correspondence and attribute description scores, in particular, evidence clear improvement in spatial text-visual alignment.
- General Video-Language Reasoning: While optimized for fine-grained tasks, SWIM remains competitive on MVBench, VideoMME, and ActivityNet-QA, indicating no statistical trade-off in broader temporal or compositional reasoning.
- Ablation Studies:
- Supervising across 6 evenly distributed decoder layers yields optimal stability and accuracy for attention alignment.
- Mean attention fusion across layers is superior to additive or multiplicative fusion for preserving salient cross-modal activations.
- Binary cross-entropy as the loss function provides the most robust attention sharpness and reference region specificity.
- Attention Localization and Robustness: SWIM achieves higher GamePoint@P and fine-grained alignment scores (AP/AUC/NSS/Precision) compared to Qwen2.5-VL, and demonstrates resilience to synonym-based perturbations in noun references, affirming the robustness of its grounding mechanism.
Theoretical and Practical Implications
SWIM presents compelling evidence that explicit, object-noun-targeted supervision during training resolves a fundamental misalignment in MLLMs, obviating the need for inference-time visual prompts. This paradigm enables more human-centric, natural language-driven object localization in video, aligning model behavior with typical user intent and interaction patterns. Practically, the reduction in architectural and data input complexity enhances deployability in real-world, prompt-driven applications.
On a theoretical level, SWIM motivates a reevaluation of the canonical pretraining biases in MLLMs by exposing the asymmetry between spatially grounded (attribute) and abstract (noun) semantic tokens. The findings suggest that fine-tuning strategies exploiting explicit, token-level linking between modalities can compensate for dataset biases and poorly structured high-level representations produced during large-scale pretraining.
Future Directions
The demonstrated scalability of SWIM with increasing mask-annotated data volume, and its non-plateauing performance trajectory, strongly suggest that additional gains could be realized through larger, more diverse, and more finely annotated datasets. The strategy of aligning internal cross-attention mechanisms to explicit ground-truth regions also points to broader utility for other neural architectures, tasks such as visual-language segmentation, instruction following, and complex multi-object chain-of-thought reasoning, where precise cross-modal referentiality is paramount.
Further research may explore adaptive supervision schedules, joint adaptation across noun and attribute tokens, and extension to compositional prompts involving multiple or occluded referents. SWIM's robust token-anchored masking could also be leveraged for interpretable model diagnostics in safety-critical or multi-agent settings.
Conclusion
SWIM constitutes a methodologically sound advance for aligning textual and visual object representations in MLLMs, achieving superior fine-grained object understanding in video without reliance on visual prompts or network modifications. The construction of NL-Refer and explicit attention regularization fundamentally improve spatial reference grounding for object nouns, validated by both comprehensive metrics and qualitative inspection. The approach not only improves model fidelity to real-world user input but also opens pathways for further research in fine-grained multimodal alignment and scalable object-centric supervision (2605.18018).
