SceneGlue: Scene-Aware Sparse Matching

• SceneGlue is a scene-aware sparse matcher that integrates parallel attention to simultaneously exchange intra- and inter-image information.
• It employs a Visibility Transformer to estimate keypoint visibility from local match supervision, reducing false positives in non-overlapping regions.
• Empirical evaluations demonstrate that SceneGlue improves image matching, homography estimation, and localization performance compared to traditional methods.

Searching arXiv for papers on SceneGlue and closely related uses of the term. SceneGlue is a scene-aware feature matching framework for local feature matching that introduces scene-level reasoning without scene-level groundtruth annotations. It is designed for cross-view correspondence under partial overlap and large changes in scale, viewpoint, or illumination, where traditional methods are constrained by the inherent local nature of feature descriptors and cannot reliably determine whether a keypoint is even visible in the other view. SceneGlue addresses these limits with a hybridizable matching paradigm that integrates implicit parallel attention and explicit cross-view visibility estimation, while remaining a sparse matcher trained using only local feature matches (Du et al., 15 Apr 2026).

1. Problem formulation and motivation

SceneGlue is grounded in three failure modes of conventional sparse matching. First, constrained local context means that point-level features lack global scene semantics, so ambiguities persist in textureless or repetitive regions. Second, occlusion and visibility produce many keypoints in non-overlapping areas; traditional matchers still try to match them, inflating false positives and harming downstream geometry. Third, sequential attention limits arise because applying self-attention and then cross-attention in sequence restricts simultaneous reasoning across and within images (Du et al., 15 Apr 2026).

The framework therefore treats non-local scene information—especially the common visible region—as central to sparse correspondence. Its core idea is to combine two forms of scene awareness. Implicit scene awareness is provided by parallel attention, in which self- and cross-attention run in parallel to exchange information within each image and across images simultaneously. Explicit scene awareness is provided by a Visibility Transformer, a lightweight transformer that predicts cross-view visibility for each keypoint using only local match supervision derived from geometry. The method is explicitly designed to avoid scene-level masks and other extra supervision, distinguishing it from approaches that require dense features or scene-level annotation (Du et al., 15 Apr 2026).

A common misconception is that SceneGlue is a dense or segmentation-driven matcher because it reasons about scene overlap. In fact, it is a sparse matcher: two images produce descriptor sets $d_s \in \mathbb{R}^{M \times C}$ and $d_t \in \mathbb{R}^{N \times C}$, with keypoint coordinates $p_s \in \mathbb{R}^{M \times 3}$ and $p_t \in \mathbb{R}^{N \times 3}$, where the hidden size is $C$ and the default is $256$. The scene-aware signal is added on top of standard keypoint pipelines rather than replacing them with dense correspondence estimation (Du et al., 15 Apr 2026).

2. Architecture and information flow

SceneGlue begins with informative feature representation. In addition to the canonical SuperPoint $1/8$-resolution map, it samples keypoint features from $1$, $1/2$, $1/4$, and $d_t \in \mathbb{R}^{N \times C}$0 scales and fuses them via lightweight linear layers to form per-keypoint descriptors robust to scale variation. It then applies the Wave Position Encoder (Wave-PE), which treats the descriptor as an amplitude $d_t \in \mathbb{R}^{N \times C}$1 and the position as a phase $d_t \in \mathbb{R}^{N \times C}$2, combining them through Euler’s identity. For keypoint $d_t \in \mathbb{R}^{N \times C}$3:

$d_t \in \mathbb{R}^{N \times C}$4

$d_t \in \mathbb{R}^{N \times C}$5

where $d_t \in \mathbb{R}^{N \times C}$6 is elementwise product and $d_t \in \mathbb{R}^{N \times C}$7, $d_t \in \mathbb{R}^{N \times C}$8, and $d_t \in \mathbb{R}^{N \times C}$9 are two-layer MLPs (Du et al., 15 Apr 2026).

The core encoder stacks $p_s \in \mathbb{R}^{M \times 3}$0 parallel-attention layers, with default $p_s \in \mathbb{R}^{M \times 3}$1. Each layer performs intra-view self-attention on each image’s tokens and cross-view attention between images with a shared attention map for both directions. In standard multi-head notation, for token matrix $p_s \in \mathbb{R}^{M \times 3}$2 and per head,

$p_s \in \mathbb{R}^{M \times 3}$3

Let $p_s \in \mathbb{R}^{M \times 3}$4 and $p_s \in \mathbb{R}^{M \times 3}$5 denote source and target tokens, including the two learnable scene tokens per image. SceneGlue computes

$p_s \in \mathbb{R}^{M \times 3}$6

and shared-weight cross-attention

$p_s \in \mathbb{R}^{M \times 3}$7

Per image, SceneGlue fuses the self- and cross-attended features with a small MLP, then adds residuals and layer norm. This parallel scheme lets intra- and inter-image messages flow simultaneously and avoids computing two independent cross-attentions (Du et al., 15 Apr 2026).

Explicit scene awareness is implemented by the Visibility Transformer. SceneGlue maintains two learnable scene tokens per image of size $p_s \in \mathbb{R}^{M \times 3}$8. Before classification, a spatial MLP mixes the two tokens and a channel MLP mixes channels. The Visibility Transformer then establishes relationships between refined scene tokens and local descriptors:

$p_s \in \mathbb{R}^{M \times 3}$9

$p_t \in \mathbb{R}^{N \times 3}$0

The fused features are projected per keypoint to visibility logits, with final probability

$p_t \in \mathbb{R}^{N \times 3}$1

This produces an explicit visible-versus-invisible prediction for each keypoint (Du et al., 15 Apr 2026).

Matching is performed after the parallel-attention stack. SceneGlue computes the similarity matrix $p_t \in \mathbb{R}^{N \times 3}$2 by inner products,

$p_t \in \mathbb{R}^{N \times 3}$3

and converts $p_t \in \mathbb{R}^{N \times 3}$4 to a doubly-stochastic partial assignment $p_t \in \mathbb{R}^{N \times 3}$5 with dustbins via Sinkhorn optimal transport on the augmented matrix, following SuperGlue. Matches are then read from the resulting assignment matrix, optionally with visibility-based filtering in challenging cases (Du et al., 15 Apr 2026).

3. Supervision, losses, and optimization

A defining property of SceneGlue is that it requires no scene-level annotation. The visibility labels are derived from geometric consistency using only local match supervision: a keypoint is labeled visible if it has a ground-truth correspondence under the known homography on HPatches and Oxford100k or camera pose on MegaDepth, YFCC100M, ScanNet, and InLoc; otherwise it is labeled invisible. This converts supervision on local matches into a proxy of scene-level overlap without additional annotations (Du et al., 15 Apr 2026).

The visibility head is trained with binary cross-entropy,

$p_t \in \mathbb{R}^{N \times 3}$6

The matching head uses the negative log-likelihood on the optimal-transport assignment:

$p_t \in \mathbb{R}^{N \times 3}$7

The total loss is

$p_t \in \mathbb{R}^{N \times 3}$8

with ablation showing the best performance at $p_t \in \mathbb{R}^{N \times 3}$9 (Du et al., 15 Apr 2026).

The training regime is task-specific but standardized. SceneGlue uses AdamW, cosine LR, an initial learning rate of $C$0, and linear warm-up. Homography training runs for $C$1 epochs on Oxford100k with batch size $C$2; outdoor pose training runs for $C$3 epochs on MegaDepth with batch size $C$4. The model uses $C$5 parallel-attention layers with $C$6, a SuperPoint backbone, and lightweight multi-scale fusion. Data preparation includes resizing images per task, sampling $C$7–$C$8 keypoints, and padding with random keypoints when needed for batching. Matching threshold $C$9 is used for output selection (Du et al., 15 Apr 2026).

The evaluation protocol spans image matching and homography on HPatches and R1M, outdoor pose on MegaDepth and YFCC100M, indoor pose on ScanNet and InLoc without retraining, and long-term visual localization on Aachen Day–Night using the HLoc pipeline. This breadth is central to the paper’s claim that scene-aware sparse matching improves not just raw correspondences but downstream geometric estimation across multiple regimes (Du et al., 15 Apr 2026).

4. Empirical performance across tasks

On HPatches image matching, SceneGlue reports overall MMA at $256$0 px of $256$1 and is consistently best or tied across thresholds. Under illumination change, it is notably higher at strict thresholds; at $256$2 px it reports $256$3 versus SuperGlue $256$4 and LightGlue $256$5. Under viewpoint change, it is best at strict thresholds $256$6–$256$7 px$256$8 and competitive at larger thresholds (Du et al., 15 Apr 2026).

For homography estimation on R1M, trained on Oxford100k, SceneGlue reports AUC@10px of $256$9 versus $1/8$0 for SAM and $1/8$1 for SuperGlue. Its Precision/Recall/F1 are $1/8$2, compared with $1/8$3 for SAM and $1/8$4 for SuperGlue. These gains are consistent with the paper’s emphasis on suppressing false positives in non-overlapping regions while retaining high recall (Du et al., 15 Apr 2026).

For outdoor pose estimation, SceneGlue reports MegaDepth AUC at $1/8$5 of $1/8$6, competitive or better than SuperGlue $1/8$7, SAM $1/8$8, and LightGlue $1/8$9. On YFCC100M, the exact AUC is $1$0, compared with $1$1 for SuperGlue and $1$2 for LightGlue; the approximate AUC is $1$3, compared with $1$4 for SuperGlue and $1$5 for LightGlue (Du et al., 15 Apr 2026).

For indoor pose estimation, SceneGlue reports ScanNet AUC $1$6 of $1$7, best at $1$8 and $1$9 and on par at $1/2$0 with LightGlue $1/2$1. On InLoc, pose recall on DUC1 at $1/2$2 is $1/2$3, and on DUC2 it is $1/2$4, described as slightly better or tied with SuperGlue and DiffGlue or competitive among top methods (Du et al., 15 Apr 2026).

For long-term visual localization on Aachen Day–Night, SceneGlue reports v1.0 Day $1/2$5, v1.0 Night $1/2$6, v1.1 Day $1/2$7, and v1.1 Night $1/2$8. The paper describes these as best at several thresholds and at or near SOTA elsewhere. This pattern suggests that explicit visibility estimation is especially useful in the night/day and partial-overlap regimes that dominate localization difficulty (Du et al., 15 Apr 2026).

5. Ablations, interpretability, efficiency, and limitations

The ablation study isolates each major component. Starting from a baseline with sequential attention like SuperGlue, the reported F1 is $1/2$9. Adding Wave-PE yields $1/4$0 $1/4$1; adding parallel attention yields $1/4$2 $1/4$3; adding the multi-scale network yields $1/4$4 $1/4$5; and adding the Visibility Transformer yields the full SceneGlue result of $1/4$6 $1/4$7. The paper also reports that performance increases with $1/4$8 and is best at $1/4$9, and that Wave-PE outperforms MLP-PE at similar parameter counts of approximately $d_t \in \mathbb{R}^{N \times C}$00 versus $d_t \in \mathbb{R}^{N \times C}$01 for the encoder (Du et al., 15 Apr 2026).

Interpretability is a stated property rather than an incidental by-product. SceneGlue produces per-keypoint visibility probabilities that delineate co-visible versus non-overlapping regions, and visualizations show visibility boundaries aligning with actual scene overlap. Occluded and non-overlapping areas are flagged as invisible, guiding the matcher to avoid spurious correspondences. The paper frames this as diagnostic utility: when matching fails, predicted overlap can indicate whether the error arose from limited co-visibility or descriptor confusion (Du et al., 15 Apr 2026).

Its computational profile is also explicitly quantified. With $d_t \in \mathbb{R}^{N \times C}$02 keypoints on an RTX 2060 SUPER, SceneGlue reports $d_t \in \mathbb{R}^{N \times C}$03 parameters, $d_t \in \mathbb{R}^{N \times C}$04 FLOPs, and $d_t \in \mathbb{R}^{N \times C}$05 ms end-to-end runtime. The per-layer complexity is

$d_t \in \mathbb{R}^{N \times C}$06

with memory $d_t \in \mathbb{R}^{N \times C}$07 across heads. Because the shared-weight cross-attention computes $d_t \in \mathbb{R}^{N \times C}$08 once and reuses its transpose, it reduces the cross-view term to $d_t \in \mathbb{R}^{N \times C}$09 instead of twice that in standard dual cross-attention (Du et al., 15 Apr 2026).

SceneGlue is positioned as a drop-in matcher. It takes keypoints and descriptors such as SuperPoint, and can replace SuperGlue or LightGlue in SfM/SLAM or HLoc/Aachen pipelines without changes to the rest of the pipeline. Typical settings are $d_t \in \mathbb{R}^{N \times C}$10–$d_t \in \mathbb{R}^{N \times C}$11 keypoints, descriptor dimension $d_t \in \mathbb{R}^{N \times C}$12, $d_t \in \mathbb{R}^{N \times C}$13 layers, $d_t \in \mathbb{R}^{N \times C}$14, and matching threshold $d_t \in \mathbb{R}^{N \times C}$15. For heavy scale changes, the paper recommends keeping multi-scale extraction enabled; for difficult night/day conditions, it recommends thresholding visibility $d_t \in \mathbb{R}^{N \times C}$16 to drop low-confidence keypoints before matching (Du et al., 15 Apr 2026).

The reported limitations are equally specific. Under severe appearance change or very wide baselines, some mismatches persist; the method predicts overlap rather than semantics, so it cannot reason about object category-level cues. Large moving objects can confuse visibility estimation trained for static geometry. Proposed extensions include multi-view training, incorporating semantic priors, larger or multi-scale transformer backbones, hierarchical tokens for scaling to tens of thousands of keypoints, and joint training with adaptive compute in the style of LightGlue (Du et al., 15 Apr 2026).

6. Broader uses of the “SceneGlue” concept

The name “SceneGlue” refers most directly to the scene-aware sparse matcher above, but the broader literature also uses the term descriptively for systems that glue scene structure across modalities, time, or independently reconstructed maps. In Graph-GSReg, the method is described as a “scene gluing” solution for 3D Gaussian Splatting: it aligns two independently optimized 3DGS scenes by constructing a 3D scene graph from a 3DGS and its rendered images, reformulating 3DGS registration as a graph registration problem, and then running Self-Supervised Test-Time Optimization to remove hollows and floaters in the merged scene (Lee et al., 29 Jun 2026).

In 4D Primitive-Mâché, “SceneGlue” refers to the mechanism by which dense geometry observations of moving scene parts, obtained at different timestamps from a casual monocular RGB video, are glued together into temporally consistent object-scale entities. Each entity is represented as a small set of rigid 3D primitives with a single $d_t \in \mathbb{R}^{N \times C}$17 pose per primitive, estimated via dense 2D correspondences between adjacent frames, yielding a persistent 4D reconstruction and object permanence through motion extrapolation (Mazur et al., 18 Dec 2025).

In Universal Scene Graph Generation, USG functions as SceneGlue by unifying heterogeneous modalities into a coherent, modality-invariant yet modality-preserving scene representation. USG-Par operationalizes this via shared mask decoding, a learnable object associator to bridge modality gaps, relation proposal construction, a transformer-based relation decoder, and text-centric contrastive training to mitigate domain imbalances across image, video, 3D, and text inputs (Wu et al., 19 Mar 2025).

This suggests that “SceneGlue” has developed two related meanings in current arXiv usage. One is the proper name of a specific scene-aware sparse feature matcher. The other is a broader design pattern in which scene-level structure is used to glue together local correspondences, object primitives, or cross-modal entities. The common thread is explicit reasoning about global context—co-visible regions in image matching, object-level structure in 3DGS registration, temporal grouping in persistent 4D reconstruction, and cross-modal association in universal scene graphs—even though the underlying representations and objectives differ substantially (Du et al., 15 Apr 2026).