Scene Structure Encoder (SSE) in 3D SLAM

Updated 7 July 2026

Scene Structure Encoder (SSE) is a module that transforms raw RGB-D data into latent representations capturing hierarchical geometry and contextual scene structure.
It integrates DINO-based feature extraction, geometric MLPs, and two-stage attention to fuse color and depth information for downstream tasks like NeRF and 3DGS SLAM.
Empirical results demonstrate that SSE improves reconstruction accuracy and robustness, offering explicit scene organization that benefits various scene understanding pipelines.

Scene Structure Encoder (SSE) denotes a class of modules that transform raw scene observations into structure-aware latent representations for downstream inference. In "DINO-SLAM: DINO-informed RGB-D SLAM for Neural Implicit and Explicit Representations," the term is explicit: SSE maps an RGB image and depth map, $(c,d)$ , to raw DINO features $f_d$ and Enhanced DINO (EDINO) features $f_{ed}$ through $\Phi_{\text{SSE}}:(c,d)\mapsto(f_d,f_{ed})$ , with the stated purpose of capturing hierarchical scene elements and structural relationships for NeRF- and 3DGS-based SLAM (Gong et al., 25 Jul 2025). Closely related architectures realize analogous functions by segmenting photo streams into scenes, encoding scene-level label priors, learning spectral structure maps, building transmitter-conditioned 3D memories, or organizing dynamic scenes as neural scene graphs (Wang et al., 2019, Xu et al., 2020, Shin et al., 2023, Qiao et al., 3 Jun 2026, Deng et al., 2023).

1. Conceptual scope and representational forms

A recurring motivation for SSE-like modules is the insufficiency of generic feature encoders when downstream tasks depend on explicit scene organization. In DINO-SLAM, raw DINO features are described as 2D and image-centric, single-scale and non-factorized, and not directly aligned with 3D SLAM representations such as signed distance fields and Gaussian primitives. In Vision-and-Language Navigation, ImageNet-pre-trained encoders are described as mismatched to scene-centric indoor panoramas and to navigation requirements such as relative pose, traversability, and layout. In point-cloud segmentation, simple concatenation of global features is described as inadequate because it does not act as an explicit scene-level constraint on point-wise labels (Gong et al., 25 Jul 2025, Kuo et al., 2022, Xu et al., 2020).

The literature exhibits several distinct but related representational forms.

System	Encoded structure	Downstream use
DINO-SLAM SSE (Gong et al., 25 Jul 2025)	Geometry-aware EDINO features from RGB-D	NeRF and 3DGS SLAM
Hierarchical photo-scene encoder (Wang et al., 2019)	Photo-to-scene segmentation and scene vectors	Album storytelling
SEA pre-training (Kuo et al., 2022)	3D pose, traversability, and instance-aware view features	VLN agents
SceneEncoder (Xu et al., 2020)	Scene-level class-presence descriptor	Point-cloud semantic segmentation
SSGNet (Shin et al., 2023)	Eigenvector-like structural maps	Depth upsampling, denoising, related low-level tasks
WiSER (Qiao et al., 3 Jun 2026)	Transmitter-conditioned sparse 3D scene memory	Radiomap and CIR prediction
MSG-BART (Liu et al., 2023)	Global and local spatio-temporal scene graph representations	Video-grounded dialogue
ProSGNeRF (Deng et al., 2023)	Dynamic neural scene graphs with object latents and transforms	Urban novel view synthesis

This suggests a broad design space. Some SSEs are factorized multimodal fusion modules, some are explicit segmenters, some are graph encoders, some are global semantic priors, and some are reusable sparse 3D memories. What unifies them is not a single architecture but a common role: scene structure is made explicit enough to supervise, constrain, or query a downstream model.

2. DINO-SLAM and geometry-aware EDINO construction

In DINO-SLAM, SSE is a lightweight fusion network on top of a DINO ViT-S backbone. For each RGB-D frame, the color image $c$ is passed through DINO to obtain $f_d$ , then through an appearance MLP $F_{ed}$ to obtain $f_a$ , while the depth map $d$ is passed through a geometric MLP $F_{eg}$ to obtain $f_d$ 0. Two attention stages refine the representation:

$f_d$ 1

$f_d$ 2

$f_d$ 3

followed by

$f_d$ 4

The paper attributes the hierarchical character of the encoder not to an explicit multi-scale pyramid but to the two-stage attention and to the interaction between local depth cues and global DINO context (Gong et al., 25 Jul 2025).

The downstream coupling is explicit. In the NeRF-based pipeline, tri-planes $f_d$ 5 encode geometry, appearance, and EDINO-related channels. EDINO supervises the encoding tri-plane through

$f_d$ 6

while raw DINO features supervise the rendered feature map through

$f_d$ 7

In the 3DGS-based pipeline, each Gaussian stores a per-Gaussian EDINO feature $f_d$ 8, and the rendered feature map is

$f_d$ 9

with the same $f_{ed}$ 0 objective against per-image DINO features. SSE is trained jointly with the SLAM system rather than in isolation; the DINO backbone is pre-trained, and the SSE MLPs and attention modules are optimized end-to-end through rendering and feature losses, without semantic annotations or pre-defined categories (Gong et al., 25 Jul 2025).

3. Hierarchical, segmental, and graph-based scene factorization

One important lineage of SSEs makes scene structure explicit through segmentation or graph organization. In "Hierarchical Photo-Scene Encoder for Album Storytelling," an album $f_{ed}$ 1 is first encoded photo-by-photo with a CNN and Bi-GRU, then segmented into scenes by a learned boundary detector,

$f_{ed}$ 2

and a scene GRU whose hidden state is reset according to

$f_{ed}$ 3

The resulting scene vectors $f_{ed}$ 4 are concatenated with photo vectors and consumed by a hierarchical attention decoder, while a reconstructor enforces recoverability of album representations from decoder dynamics (Wang et al., 2019).

In "MSG-BART: Multi-granularity Scene Graph-Enhanced Encoder-Decoder LLM for Video-grounded Dialogue Generation," structure is represented as a spatio-temporal scene graph extracted by STTran. The model uses a relational-node graph for global encoding and triplet nodes for local reasoning. Node relevance is first measured by cosine similarity to the question and graph-smoothed by

$f_{ed}$ 5

then updated through question-conditioned graph attention

$f_{ed}$ 6

with updated node embeddings aggregated into a global representation $f_{ed}$ 7. A decoder-side triplet selector similarly extracts a question-relevant local graph representation $f_{ed}$ 8. A multi-pointer network then mixes the graph-enhanced and text-dominant decoding pathways (Liu et al., 2023).

In "ProSGNeRF: Progressive Dynamic Neural Scene Graph with Frequency Modulated Auto-Encoder in Urban Scenes," the structural factorization is a neural scene graph

$f_{ed}$ 9

whose nodes include a world frame, camera nodes, a background field, and dynamic object nodes, and whose edges carry affine transforms

$\Phi_{\text{SSE}}:(c,d)\mapsto(f_d,f_{ed})$ 0

Object samples are transformed into object coordinate space by

$\Phi_{\text{SSE}}:(c,d)\mapsto(f_d,f_{ed})$ 1

and object fields are conditioned on latent shape and appearance codes from a ResNet-34-based auto-encoder. To scale to arbitrarily large scenes, the method allocates overlapping local scene graphs progressively and freezes older graphs when the camera exits their region; the paper notes that NSG on a 30-second clip required more than 1TB memory (Deng et al., 2023).

4. Global priors, auxiliary structural supervision, and spectral bases

Other SSEs do not segment scenes into explicit objects or scenes, but instead inject scene structure as a prior, an auxiliary objective, or a spectral basis. In "SceneEncoder: Scene-Aware Semantic Segmentation of Point Clouds with A Learnable Scene Descriptor," the scene representation is a global descriptor

$\Phi_{\text{SSE}}:(c,d)\mapsto(f_d,f_{ed})$ 2

where $\Phi_{\text{SSE}}:(c,d)\mapsto(f_d,f_{ed})$ 3 denotes the predicted probability that class $\Phi_{\text{SSE}}:(c,d)\mapsto(f_d,f_{ed})$ 4 exists in the scene. Point-level class probabilities are filtered by

$\Phi_{\text{SSE}}:(c,d)\mapsto(f_d,f_{ed})$ 5

and a region similarity loss aligns features of confident points with same-label neighbors through local cosine similarity. The result is a scene-level label prior that gates point-wise predictions (Xu et al., 2020).

In "Structure-Encoding Auxiliary Tasks for Improved Visual Representation in Vision-and-Language Navigation," the encoder remains a ResNet-50, but its representation is reshaped by three simulator-derived auxiliary tasks: 3D jigsaw for relative pose, traversability prediction for navigability, and instance classification for view discrimination. The total loss is

$\Phi_{\text{SSE}}:(c,d)\mapsto(f_d,f_{ed})$ 6

with all three weights set to 1. The resulting frozen features are used as a drop-in replacement for ImageNet features in Speaker-Follower, EnvDrop, and AuxRN, with no architectural changes to those agents (Kuo et al., 2022).

In "Scene Structure Guidance Network: Unfolding Graph Partitioning into Pixel-Wise Feature Learning," the encoded structure is a set of eigenvector-like maps $\Phi_{\text{SSE}}:(c,d)\mapsto(f_d,f_{ed})$ 7 produced by a small encoder-decoder CNN. They are trained with a Laplacian quadratic objective

$\Phi_{\text{SSE}}:(c,d)\mapsto(f_d,f_{ed})$ 8

and a spatial regularizer

$\Phi_{\text{SSE}}:(c,d)\mapsto(f_d,f_{ed})$ 9

combined as

$c$ 0

With softmax across channels, the maps become sparse, mutually competitive structural assignments that are then mixed by a $c$ 1 convolution and injected into downstream networks. The main model has about 56K parameters, and the depth-oriented SSGNet-D about 29K (Shin et al., 2023).

5. Reusable scene memories and query-specific decoders

A more explicit memory-centric formulation appears in "WiSER: A Wireless Scene Encoder for Geometry-Grounded Multi-View Wireless Prediction." The input is a sparse voxel scene

$c$ 2

together with a transmitter location $c$ 3. The encoder produces a transmitter-conditioned sparse 3D scene memory

$c$ 4

where $c$ 5 denotes multi-resolution local scene memories and $c$ 6 a global, transmitter-conditioned scene memory at the coarsest scale. Transmitter conditioning is introduced only at the coarsest scale through Fourier-encoded transmitter coordinates and AdaLN-zero modulation of sparse transformer blocks (Qiao et al., 3 Jun 2026).

The decoders expose a clear query interface. For dense radiomap prediction on a receiver plane, each receiver query is augmented by a ray-corridor selection of top- $c$ 7 local tokens,

$c$ 8

which are concatenated with $c$ 9 and processed by cross-attention and receiver-plane self-attention. For CIR prediction, a DETR-style set decoder uses learnable path queries conditioned on receiver position and outputs unordered tap predictions

$f_d$ 0

trained with Hungarian matching. The shared encoder thus supports heterogeneous outputs—dense path-gain fields and variable-cardinality delay-power tap sets—under a common coordinate frame and a joint objective (Qiao et al., 3 Jun 2026).

This architecture makes the memory role of an SSE particularly explicit. The encoder is run once for a scene-plus-transmitter condition, while the decoders realize different structural queries over the same latent 3D memory.

6. Empirical behavior, ablations, and limitations

Ablations consistently show that the structural component, not merely the presence of an additional feature stream, is responsible for much of the gain. In DINO-SLAM on Replica, the NeRF variant reports Acc/Comp/Comp.rate/Depth L1/RMSE of 2.18/1.75/96.46/0.94/0.63 for the ESLAM baseline without DINO, 2.07/1.67/96.63/0.79/0.58 with raw DINO but without EDINO, and 1.94/1.64/97.06/0.66/0.51 for full SSE. On ScanNet with 3DGS, the ablation RMSE drops from 12.67 without DINO to 11.39 without EDINO and 9.55 with full DINO-SLAM. The complete systems also report average ATE of 1.95 cm for the NeRF variant and 1.36 cm for the 3DGS variant on TUM (Gong et al., 25 Jul 2025).

Comparable patterns appear elsewhere. SEA features improve Test-Unseen success rate by 12 absolute points for Speaker-Follower, 5 for Env-Dropout, and 4 for AuxRN, while Taskonomy probing improves semantic segmentation mAP from 29.40 to 40.27 and lowers normal-estimation RMSE from 0.585 to 0.523. SceneEncoder lifts PointConv on ScanNet from 55.6 mIoU to 62.8 mIoU when combined with region similarity loss. SSGNet reduces Middlebury 2005 $f_d$ 1 depth-upsampling RMSE from 1.043 cm for MMSR to 0.830 cm, and improves Kodak denoising at $f_d$ 2 from 29.27 dB to 29.34 dB PSNR when added to IDR. MSG-BART reaches CIDEr 1.008 on DSTC10-AVSD, while removing GAT, node similarity, triplet similarity, graph encoder, graph decoder, or pointer each lowers performance. WiSER reports radiomap MAE/RMSE/PSNR of 3.834 dB/5.500 dB/26.78 dB, outperforming RF-3DGS at 4.585 dB/6.281 dB/25.62 dB and NeRF2 at 8.238 dB/10.056 dB/21.54 dB, and on CIR prediction reduces peak-power MAE from 11.50 dB for a 3D CNN baseline to 5.89 dB while improving delay MAE from 1.50 ns to 0.61 ns (Kuo et al., 2022, Xu et al., 2020, Shin et al., 2023, Liu et al., 2023, Qiao et al., 3 Jun 2026).

The limitations identified in these papers are equally instructive. The album-storytelling scene encoder uses hard boundaries with a fixed 0.5 threshold and models only a single scene level. SEA relies on a binary traversability label and freezes the encoder during VLN training. SceneEncoder’s global descriptor is coarse, because it predicts only class existence and may suppress true positives when misestimated. SSGNet is lightweight, but the paper describes integration into more complex transformer- or GAN-based systems as non-trivial. WiSER assumes indoor static scenes, a fixed 10 cm voxelization, and a single-band setting, while its material model is simulator-driven (Wang et al., 2019, Kuo et al., 2022, Xu et al., 2020, Shin et al., 2023, Qiao et al., 3 Jun 2026).

Taken together, these systems indicate that SSE is best understood as a functional category rather than a single architecture. It encompasses multimodal fusion modules such as DINO-SLAM, segmental hierarchies such as the photo-scene encoder, graph encoders such as MSG-BART and ProSGNeRF, structural auxiliary encoders such as SEA, semantic scene priors such as SceneEncoder, spectral structure bases such as SSGNet, and reusable sparse 3D memories such as WiSER. A plausible implication is that future SSE designs will continue to differ in representation and supervision, while retaining the same central objective: to make scene structure explicit enough that downstream models can query, constrain, and reuse it.