- The paper introduces a novel framework combining Semantic-consistent Spatial Attention Alignment (SSA) and Spatial Attention Refinement (SAR) to mitigate synonym-induced grounding inconsistency.
- The paper achieves state-of-the-art results on OV-COCO and OV-LVIS benchmarks, significantly reducing performance drops under synonym substitution.
- The paper builds and validates a synonym-enriched visual corpus (SEViC) that leverages multiple synonyms and definitions to enhance robust training across diverse categories.
SynCLIP: Synonym-Coherent Pretraining for Open-Vocabulary Dense Perception
Introduction
The proliferation of open-vocabulary dense perception (OVDP) tasks requires models to interpret and localize arbitrary concepts with textual guidance, transcending fixed label sets. However, CLIP-based approaches—while excelling at global vision-language alignment—exhibit synonym-induced grounding inconsistency, whereby semantically equivalent expressions (e.g., “dog” vs. “canine”) yield divergent spatial attention distributions. This limitation degrades model robustness and downstream performance in tasks such as open-vocabulary detection (OVD) and open-vocabulary segmentation (OVS). SynCLIP introduces a synonym-coherent framework that enforces semantic and spatial alignment across synonyms, aiming for reliable dense grounding irrespective of label phrasing (2607.11008).
SynCLIP Framework
SynCLIP is a pretraining paradigm built atop frozen CLIP architectures, incorporating two principal modules to achieve synonym-robust grounding: Semantic-consistent Spatial Attention alignment (SSA) and Spatial Attention Refinement (SAR).
Semantic-consistent Spatial Attention Alignment (SSA)
SSA addresses synonym-induced inconsistency by minimizing discrepancies between attention maps produced for original labels and their synonymous/definition-based variants. This is operationalized through a semantic alignment loss that exploits a dual visual encoder (student/teacher) architecture. For every image, attention maps are calculated between dense image features and both original and semantically enriched expressions. The loss penalizes spatial misalignment, compelling the model toward lexical invariance in grounding.
Spatial Attention Refinement (SAR)
While SSA enforces aggregate consistency, SAR further enhances spatial precision. Semantic token selection identifies top attention-weighted tokens within the enriched expressions, harnessing context from pretrained Vision Foundation Models (VFMs, e.g., DINOv2) for these local regions. Cosine similarity matrices are computed between selected tokens and the full dense image grid, and aggregated via simple averaging to yield a context-aware, spatially refined attention map. A final aggregation fuses semantic and spatial attention, regulated by tunable coefficients, and the process is supervised via an optimized alignment loss.
Synonym-Enriched Visual Corpus (SEViC)
Effective synonym-consistent pretraining requires rich lexical diversity. SynCLIP introduces SEViC: a corpus that augments each COCO/LVIS category with multiple synonyms and textual definitions, generated by prompting LLMs (e.g., DeepSeek) and filtered for semantic precision using a secondary LLM validator. The corpus encompasses 1,234 categories and 11,558 validated expressions over 118,287 images, supporting robust supervision across lexical variants.
Experimental Results
Evaluations on OV-COCO and OV-LVIS benchmarks demonstrate that SynCLIP surpasses all prior CLIP-based methods. Notably, on OV-COCO, SynCLIP achieves 43.6 AP₅₀ on novel classes with ViT-B/16 and 49.8 with ViT-L/14 backbones—substantially exceeding baseline F-ViT+DeCLIP and CLIPSelf methods. On OV-LVIS (which features a more challenging, long-tailed label distribution), SynCLIP maintains state-of-the-art or comparable results (27.8 and 37.2 mAPmask with ViT-B/16 and ViT-L/14), underscoring its generalization beyond frequent, well-posed categories.
Robustness to Synonym Substitution
A key claim is that SynCLIP reduces synonym-induced grounding variation. Under synonym substitution—where category queries are replaced with validated synonyms—the drop in AP₅₀ for novel classes is -4.4 (SynCLIP), versus -8.8 (CLIPSelf) and -9.5 (DeCLIP), more than halving grounding inconsistency relative to prior SOTA. Qualitative attention visualizations further confirm that SynCLIP preserves consistent localization under synonym changes.
Ablation Analyses
Ablation studies disentangle the contribution of each module. SSA alone yields marginal gains, with primary improvements resulting from SAR’s precision-centric design. Their combination produces the highest AP₅₀. Additional gains are observed when enriching the reference expressions with both synonyms and definitions, indicating their complementary value for training alignment. SAR’s semantic token hyperparameter exhibits a sweet spot (k≈7), balancing informativeness and noise.
Efficiency Considerations
Pretraining with SynCLIP incurs moderate overhead (training time per epoch: 32.5min for ViT-B/16 vs. 19.3min for DeCLIP), but inference-time parameters and FLOPs are unchanged. This efficiency ensures practicality for real-world dense perception deployments.
Implications and Future Directions
SynCLIP’s synonym-coherent approach establishes a new paradigm for robust open-vocabulary dense perception by explicitly modeling semantic invariance at the attention level. The framework’s design allows extension beyond object detection and segmentation to any spatial vision-language task where lexical rephrasings may occur (e.g., robotics, in-the-wild multimedia search). SEViC sets a precedent as both a resource and a methodology for leveraging LLMs to enrich vision-language training corpora.
Theoretically, this work underscores the limitations of conventional contrastive vision-language pretraining when faced with fine-grained linguistic variability. It motivates further research in compositional and context-aware vision-language pretraining, potentially integrating paraphrase-sensitive objectives or dynamic grounding modules. Practically, synonym robustness is likely to become a critical factor in the deployment of OVDP systems in safety-critical or zero-shot environments.
Conclusion
SynCLIP identifies and addresses the synonym-induced grounding inconsistency inherent in CLIP-based open-vocabulary dense perception. By aligning spatial attention across semantically equivalent expressions and refining grounding via linguistic and visual context integration, SynCLIP delivers both state-of-the-art performance and pronounced synonym robustness. Its modular, corpus-enriched pretraining advances the goal of reliable and interpretable dense perception in linguistically diverse real-world applications (2607.11008).