Online Scene Change Detection

• Online scene change detection is a method that identifies, localizes, and segments dynamic changes in real-world environments without relying on perfectly paired reference data.
• It employs advanced techniques like reference retrieval, geometric alignment, and semantic aggregation to overcome challenges such as misalignment, occlusions, and environmental variability.
• Real-time SCD systems integrate efficient feature extraction, pose estimation, and multi-view fusion to support applications in robotic mapping, AR localization, and maintenance surveillance.

Online scene change detection (SCD) is the problem of identifying, localizing, and segmenting changes to a real-world environment from streaming sensor or image observations, without reliance on perfectly aligned or temporally synchronized reference data. Unlike conventional SCD—which assumes a perfectly paired and spatially aligned reference image for every query—online SCD must accommodate severe viewpoint misalignment, unconstrained revisit trajectories, environmental variations, and the absence of explicit query-reference pairings. This task is central to robotic mapping, lifelong SLAM, maintenance surveillance, and dynamic mapping for AR/AR localization, with methods evolving toward label-free, pose-agnostic, and real-time architectures that leverage multi-view geometric priors and self-supervised visual cues.

1. Problem Formulation and Task Characteristics

The canonical SCD setup assumes a triplet $(r, q, y)$, where the reference image $r$ and query image $q$ are perfectly aligned and the objective is to predict a pixel-wise binary (or multi-class) change map $\hat{y}\approx y$ (Cho et al., 13 Jun 2025). Online SCD, however, is formulated over streams of images, where the query image (or sequence) must be compared against a potentially large, unpaired and heterogeneous database of prior observations: $$q \in \mathcal{I}_q,\quad \mathcal{I}_r = \{r^{(1)}, r^{(2)}, \dots, r^{(M)}\}$$ The output is a binary or multi-class change mask $\hat{y}: \Omega \to \{0,1\}$ over the image pixel domain $\Omega$, or, for multi-class SCD, a segmentation over $C$ change types (e.g., new, missing, replaced, rotated) (Park et al., 2021).

Key challenges in this task include non-trivial viewpoint variations, incomplete field-of-view (FOV) overlap, frequent occlusions/dis-occlusions, and environmental perturbations such as lighting or airborne particulates. No privileged access to ground-truth camera poses is assumed, and pairing between current and prior views is typically solved via visual place recognition, geometric correspondence, or fast pose estimation (Cho et al., 13 Jun 2025, Galappaththige et al., 15 Nov 2025, Liu et al., 14 Sep 2025).

2. Algorithmic Frameworks for Online SCD

Online SCD architectures are broadly organized into the following stages, with specific instantiations in recent literature.

2.1 Reference Retrieval and Pairing

Given the impracticality of exhaustive search over large-scale uncurated databases, reference frames relevant to the current query are selected using lightweight descriptors, such as those from Visual Place Recognition (VPR) models. This is commonly implemented as top-$K$ nearest neighbor retrieval in descriptor space: $$\mathcal{R} = \operatorname{TopK}\{\mathrm{sim}(d(q), d(r))\,|\,r \in \mathcal{I}_r\}$$ Fast approximate nearest-neighbor techniques (e.g., Faiss) are employed for large $M$, with $K$ typically limited (e.g., $1 \leq K \leq 5$) for computational tractability (Cho et al., 13 Jun 2025).

For SLAM-aware datasets or city-scale mapping, closest-pose selection may be used, where pairs are selected based on proximity in translation ($\leq 1\,\mathrm{m}$) and attitude ($\leq 0.2\,\mathrm{rad}$) (Wilf et al., 2022). Subsequent spatial alignment is refined via geometric model fitting (homography, PnP+RANSAC) or patch-level correlation.

2.2 Feature Representation and Alignment

Robust dense features are extracted using frozen vision transformers (DINOv2, SAM), deep convolutional backbones (DeepLab, ResNet), or hand-crafted local descriptors (D2Net). Features are $\ell_2$-normalized channel-wise to facilitate cosine or Euclidean similarity computation (Cho et al., 13 Jun 2025, Guo et al., 2018). For multi-view consistency, features from multiple retrieved references are pseudo-aligned to the query via sliding-window patch similarity or 3D geometric warping (Cho et al., 13 Jun 2025, Liu et al., 14 Sep 2025, Sachdeva et al., 2023).

Geometric priors—obtained via geometric foundation models (GFMs)—provide explicit camera intrinsics, poses, and per-pixel depth, enabling precise cross-view warping. Pixels are reprojected according to estimated camera transformations and depths, with occlusions detected through forward-backward depth consistency (Liu et al., 14 Sep 2025): $$M_{\rm occ}^1(p_1)= \begin{cases} 1, &D_2^1(p_1)-D_2\bigl(p_2^1\bigr)>\tau\ 0, &\text{otherwise} \end{cases}$$

2.3 Change Metric and Semantic Aggregation

Change detection is framed as patch-wise or pixel-wise metric learning. Siamese networks with contrastive or thresholded contrastive loss (TCL) directly measure the divergence between aligned features, tolerating small misalignments or illumination changes ($\tau>0$) (Guo et al., 2018). In multi-reference settings, semantic aggregation is performed using multi-head attention (MHA) on concatenated dense feature maps. Hierarchical spatial alignment/fusion at multiple scales ($n\in\{1,2,4\}$) enables robustness to viewpoint and FOV mismatch (Cho et al., 13 Jun 2025).

For frameworks leveraging 3D reconstruction, changes are detected via fusion of photometric, feature-level, and structural cues between synthesized reference views (rendered from 3D Gaussian splatting) and incoming frames (Galappaththige et al., 15 Nov 2025). A self-supervised multi-view fusion loss is optimized to integrate these cues while suppressing degenerate ("all-changed") solutions.

2.4 Object- and Region-Level Change Reasoning

Open-set and class-agnostic detection is often realized by aggregating mask proposals or region candidates. Zero-shot segmentation (e.g., from SAM), mask propagation via tracking (DEVA/XMem), and bounding box keypoint decoding (CenterNet head) all contribute to improved segmentation of novel or rare changes (Cho et al., 17 Jun 2024, Sachdeva et al., 2023). Change-type classification (missing, new, replaced, rotated) is tackled in multi-class SCD using cross-entropy loss over appropriately labeled pixels (Park et al., 2021).

2.5 Online and Real-Time Considerations

Pipeline modules are optimized for low-latency streaming. Critical optimizations include:

• Pre-computation and indexing of reference descriptors for sub-10 ms KNN retrieval (Cho et al., 13 Jun 2025)
• Limiting spatial alignment to coarse grids except when detailed reasoning is required
• Batched GPU inference and half-precision acceleration for feature extraction and depth estimation (Liu et al., 14 Sep 2025, Galappaththige et al., 15 Nov 2025)
• Region proposals and feature aggregation tuned to minimize redundant processing (e.g., using only overlapping FOVs)
• Efficient per-frame pose estimation (EPnP+RANSAC: ∼16 ms/frame) and change-guided 3D model update (few seconds for scene-scale updates) (Galappaththige et al., 15 Nov 2025)

Pipelined or incremental updates are common, with mask propagation or feature warps recursively performed between adjacent video frames or keyframes.

3. Datasets, Evaluation Protocols, and Metrics

Online SCD research is underpinned by several rigorously designed benchmarks:

• ChangeSim: Photorealistic industrial warehouse environments, multi-class annotations (missing/new/replaced/rotated), strong environmental variations (dust, low-light), sequence-based access (Park et al., 2021, Cho et al., 13 Jun 2025, Cho et al., 17 Jun 2024, Liu et al., 14 Sep 2025)
• VL-CMU-CD and PSCD: Urban and panoramic datasets recast for unaligned SCD, evaluating F1/mIoU across downsampling-induced reference sparsity (Cho et al., 13 Jun 2025, Liu et al., 14 Sep 2025)
• PACLSD: Real indoor/outdoor scenes, used to assess robustness to geometry and appearance shifts (Liu et al., 14 Sep 2025, Galappaththige et al., 15 Nov 2025)
• RC-3D and CL-Splats: Smaller, real-world and tabletop datasets for bounding-box performance and 3DGS-based update (Sachdeva et al., 2023, Galappaththige et al., 15 Nov 2025)

Evaluation is reported in F1-score (per-pixel binary or multi-class), mIoU (per-class segmentation), AP (region proposals), PSNR/SSIM/LPIPS (for updated scene representations), and inference speed (FPS or ms/frame).

Sample quantitative results:

Method / Dataset	F1 (ChangeSim)	F1 (PACLSD)	FPS
ECD (Ours, (Cho et al., 13 Jun 2025))	0.4815	—	5–10
GeoSCD (Liu et al., 14 Sep 2025)	0.531	0.318	2–3
CYWS-3D (Sachdeva et al., 2023)	0.68 (AP)	—	4–10
Ours (online, (Galappaththige et al., 15 Nov 2025))	0.638	0.638	11.2

Performance degrades gracefully with increasing reference sparsity or environmental noise, and frameworks utilizing geometric priors, multi-view fusion, or semantic aggregation substantially outperform baselines using naive pairwise differencing or single-reference matching.

4. Integration of Geometric and Semantic Cues

Explicit geometric priors enhance discriminative power under severe misalignment and occlusion:

• Per-pixel 3D reprojection constrains comparison to overlapping, non-occluded regions (Liu et al., 14 Sep 2025)
• Depth-based occlusion priors, with adaptive consistency thresholds, mask out areas where cross-view physical correspondence breaks down
• 3D scene representations (Gaussian splatting, 3D point clouds) support direct alignment and change masking in the world coordinate frame, enabling efficient region-level scene updates (Galappaththige et al., 15 Nov 2025, Wilf et al., 2022)
• Semantic proposals (from VFM backbones) guide region merges and object-level change interpretation

This geometric–semantic integration replaces (or complements) brute-force dense attention, significantly reducing spurious detections from ambiguous illumination, shadow, or viewpoint-induced appearance shifts.

5. Limitations, Open Problems, and Directions

Despite rapid advances, online SCD systems face several challenges:

• Extreme view-dependent effects (specularity, transparency) remain difficult for both feature-based and geometric approaches (Galappaththige et al., 15 Nov 2025)
• Sparse revisit coverage or poor-quality depth/pose estimation can undermine both visual and geometric matching (Liu et al., 14 Sep 2025, Galappaththige et al., 15 Nov 2025)
• Multi-class change segmentation (distinguishing between change types) is considerably harder than binary detection, with accuracy dropping by >30 mIoU points in benchmarks (Park et al., 2021)
• End-to-end differentiable models that can jointly learn reference-selection/pairing and change segmentation in streaming settings are not yet fully realized (Park et al., 2021)

Emerging strategies include:

• Hierarchical fusion of temporal, geometric, and semantic priors
• Explicit uncertainty modeling around alignment, depth, and feature matching
• Integration with online SLAM for loop-closure and persistent mapping
• Generalization to dynamic environments with moving agents and objects

6. Application Domains and System Integration

Online SCD is critical for:

• Autonomous robotic mapping in changing industrial, urban, or natural environments (Park et al., 2021)
• Map maintenance for localization systems (visual place recognition, crowd-sourced VPS) (Wilf et al., 2022)
• Continuous AR/VR scene updating and reality modeling (Galappaththige et al., 15 Nov 2025)
• Automated maintenance, safety monitoring, or inventory tracking

Practical adoption is underpinned by pipelines that minimize compute/storage overhead, tolerate pose uncertainty, and provide semantic explanations or object-level summaries of detected changes.

7. Representative Methodological Summary

Main Step	Example Implementation(s)	Reference(s)
Reference Retrieval/Pairing	VPR (BoQ), PnP+RANSAC, pose-proximal selection	(Cho et al., 13 Jun 2025, Galappaththige et al., 15 Nov 2025, Wilf et al., 2022)
Feature Extraction	DINOv2, SAM, DeepLabV2, D2Net, XFeat	(Cho et al., 13 Jun 2025, Liu et al., 14 Sep 2025, Galappaththige et al., 15 Nov 2025)
Alignment/Registration	Patch-wise correlation, geometric warping, 5DOF homography	(Liu et al., 14 Sep 2025, Wilf et al., 2022)
Geometric Priors	3D scene modeling, GFM, depth/pose wrappers	(Liu et al., 14 Sep 2025, Galappaththige et al., 15 Nov 2025)
Change Metric/Aggregation	MHA, TCL, attention fusion, region proposals	(Cho et al., 13 Jun 2025, Guo et al., 2018, Liu et al., 14 Sep 2025)
Online Inference/Latency	Batched retrieval, GPU acceleration, chunked mask propagation	(Galappaththige et al., 15 Nov 2025, Cho et al., 13 Jun 2025, Cho et al., 17 Jun 2024)
Map/Scene Update	3DGS selective update, local cloud merge, propagation via NetVLAD	(Galappaththige et al., 15 Nov 2025, Wilf et al., 2022)

This synthesis reflects current state-of-the-art approaches for online scene change detection, highlighting the essential combination of alignment, geometric reasoning, semantic fusion, and computational efficiency necessary for robust real-time deployment in operational environments.