Scene Change Detection with Vision-Language Representation Learning

Abstract: Scene change detection (SCD) is crucial for urban monitoring and navigation but remains challenging in real-world environments due to lighting variations, seasonal shifts, viewpoint differences, and complex urban layouts. Existing methods rely primarily on low-level visual features, limiting their ability to accurately identify changed objects amid the visual complexity of urban scenes. In this paper, we propose LangSCD, a vision-language framework for scene change detection that overcomes this single-modal limitation by incorporating semantic reasoning through language. Our approach introduces a modular language component that leverages vision-LLMs (VLMs) to generate textual descriptions of scene changes, which are fused with visual features through a cross-modal feature enhancer. We further introduce a geometric-semantic matching module that refines the predicted masks by enforcing semantic consistency and spatial completeness. Existing real-world scene change detection benchmarks provide only binary change annotations, which are insufficient for downstream applications requiring fine-grained understanding of scene dynamics. To address this limitation, we introduce NYC-CD, a large-scale dataset of 8,122 real-world image pairs collected in New York City with multiclass change annotations generated through a semi-automatic pipeline. Extensive experiments across multiple street-view benchmarks demonstrate that our language and matching modules consistently improve existing change-detection architectures, achieving state-of-the-art performance and highlighting the value of integrating linguistic reasoning with visual representations for robust scene change detection.

• The paper introduces LangSCD, a language-guided scene change detection framework that fuses visual and linguistic cues to overcome limitations in traditional methods.
• It employs a plug-and-play cross-modal feature enhancer and geometric-semantic mask refinement to accurately identify true structural changes versus transient distractions.
• Experimental results on the NYC-CD dataset demonstrate substantial improvements with F1 rising from 0.13 to 0.70 and IoU from 0.17 to 0.58, validating its efficacy.

Scene Change Detection with Vision-Language Representation Learning: LangSCD

Introduction

This work addresses the intrinsic limitations of single-modal scene change detection (SCD) in unconstrained urban environments by introducing LangSCD, a vision–language framework that infuses object-level semantic reasoning into change detection pipelines. Scene change detection is foundational for long-term visual place recognition (VPR), urban monitoring, and dataset maintenance, especially given the volatility of appearance, seasonal variations, and viewpoint shifts in real-world street-view settings. Traditional SCD approaches, whether CNN or transformer-based, operate at a pixel or region level by detecting low-level visual discrepancies, resulting in high rates of false positives due to illumination, reflections, and parallax, and failing to model semantically meaningful urban changes.

LangSCD overcomes these obstacles by introducing a language-centric pipeline. The framework integrates vision-LLMs (VLMs), such as GPT-4o, to generate natural language change descriptions, which are then fused into the image representation via a cross-modal feature enhancer. The predictions are further refined using a geometric-semantic matching module, enforcing both object-level completeness and semantic congruence. This approach enables more robust differentiation between true structural modifications, appearance variations (notably vegetation), and viewpoint-induced artifacts. The methodology is validated on the new NYC-CD dataset, which is the first large-scale, multi-class urban street-view SCD benchmark, annotated using a scalable, semi-automatic vision-language-driven procedure.

Figure 1: Overview of the LangSCD pipeline, illustrating language-guided visual representation enhancement and geometric-semantic mask refinement.

Methodology

Cross-Modal Feature Enhancement

The core of LangSCD’s architecture is a plug-and-play cross-modal feature enhancer. Given two temporally disjoint street-view images ($T_0$, $T_1$), a VLM generates free-form textual descriptions about objects and appearance changes (e.g., “green trees”, “new billboard”). These texts are tokenized and encoded (using BERT), producing semantic vectors. In parallel, visual tokens are extracted (e.g., DINOv2 features). A series of transformer-based layers perform bidirectional cross-attention: image tokens attend to language tokens, aligning visual regions with semantic cues, while textual representations are refined using visual context. Enhanced image features incorporate linguistically-informed priors, which suppress visually distractive non-structural changes (e.g., shadows, mirror artifacts).

Geometric–Semantic Mask Refinement

Object-level change detection requires segmentation masks that are both semantically relevant and spatially consistent. The geometric matching module leverages SAM2 tracking to generate class-agnostic, geometrically inconsistent object proposals across the temporal sequence. Masks from $T_1$ that do not correspond to $T_0$ are selected based on overlap thresholds ($\alpha_t$ for geometric, $\alpha_g$ for semantic consistency). The semantic matching module utilizes open-vocabulary models (e.g., Grounded SAM) to segment language-referred regions, filtering out predictions not supported by semantic guidance. The union of these modules yields object-aware, complete change masks that align with human-understandable change categories: new/missing objects, vegetation appearance shifts, and viewpoint-occluded entities.

Figure 2: Qualitative results of the geometric–semantic matching module, showing denoising and spatial completion.

Annotation Pipeline and Dataset

A significant contribution is the NYC-CD dataset, which systematically addresses the deficiencies of prior benchmarks, namely the absence of explicit appearance and viewpoint change labeling and lack of object-level mask quality. The annotation procedure is a semi-automatic pipeline:

• Vision–Language Guided Captioning: For each image pair, GPT-4o is prompted to enumerate new/missing objects and describe appearance changes, being explicitly guided to ignore illumination and transient distractors.
• Open-vocabulary Segmentation: Grounded SAM then segments visual regions corresponding to captioned entities.
• Geometric Consistency Check: SAM2 tracking identifies spatially inconsistent objects.
• Matching and Manual Curation: Mask overlaps between semantic and geometric proposals are computed; only consistent regions are retained. Human annotators then remove spurious masks, ensuring high-quality supervision.

The resulting dataset contains 8,122 image pairs (split across train/val/test), each indexed for three change types: structural/object, appearance (vegetation), and viewpoint-induced. This multi-class granularity directly supports the development and evaluation of SCD systems that can distinguish between genuine semantic updates and geometric occlusions.

Figure 3: Semi-automatic annotation pipeline integrating GPT captions, open-vocabulary segmentation, geometric tracking, and manual refinement.

Figure 4: Examples from NYC-CD, showcasing multi-class annotations: red (new objects), green (appearance/vegetation changes), blue (viewpoint-induced occlusions).

Experimental Results

Quantitative Evaluation

LangSCD modules can be integrated as plug-ins into major SCD architectures (C-3PO, RSCD, GeSCF), and evaluated on both standard (VL-CMU-CD, PSCD) and the new NYC-CD benchmark. Across all tested datasets, LangSCD yields substantial improvements in both F1 score and IoU over unimodal and naive VLM baselines. On the challenging NYC-CD set, adding LangSCD to RSCD increases F1 from 0.13 to 0.70 and IoU from 0.17 to 0.58 (i.e., +57% and +41% relative gains). On ChangeVPR, VL-CMU-CD, and PSCD, consistent improvements (between +4–46% in F1, +2–41% in IoU) are seen, underscoring the universality of language-guided enhancement for urban SCD.

Figure 5: F1 and IoU improvements attained by the language module across multiple urban and remote sensing datasets, demonstrating cross-domain generalization.

A detailed ablation demonstrates the independence of the method from any particular VLM used for caption generation or annotation (e.g., InternVL, QwenVL, GPT-4o), and the complementary effects of text enhancement and geometric–semantic matching. Multi-class evaluations further evidence the ability of LangSCD to resolve especially ambiguous categories (e.g., viewpoint-induced changes), yielding more balanced macro-F1/mIoU.

Qualitative Analysis

LangSCD produces spatially complete, object-aligned masks that strongly agree with ground-truth annotations. It excels at suppressing false positives due to environmental distractors, supports detection of subtle urban dynamics (billboards, plant cycles), and correctly partitions geometric occlusions from physical scene changes—whereas prior SCD models output fragmented or binary-valued masks regardless of the change typology.

Figure 6: Visualizations comparing LangSCD to state-of-the-art baselines for multi-class change detection.

Implications and Future Directions

LangSCD demonstrates that incorporation of language priors into SCD pipelines enables semantically structured detection in complex, unconstrained visual domains. Practically, this means improved urban VPR, more precise map updating, and better anomaly filtering for mobile robotics and autonomous navigation. Theoretically, the results highlight the limitations of purely visual models in the presence of cross-domain appearance drift and underscore the necessity of semantic grounding for robust perception.

The modularity of the LangSCD pipeline supports future extension to unified foundation models for change detection at scale. Open challenges include real-time inference efficiency, further reduction of annotation noise and bias, and extending language–vision fusion to handle larger viewpoint discrepancies and video input.

Conclusion

LangSCD establishes a new framework for scene change detection, tightly integrating language-based semantic reasoning with geometric consistency modules to outperform uni-modal models in realistic, high-variance urban environments. The NYC-CD dataset fills a critical gap for multiclass, object-level urban change detection. The consistent and substantial empirical gains—enabling robust distinction between structural, appearance-based, and viewpoint changes—signal a new direction for SCD research towards vision–language unified architectures and more comprehensive annotation standards.