Unified Scene Representation

Updated 12 May 2026

Unified scene representation is a computational framework that encodes geometry, semantics, physics, and dynamics into a single, compositional model.
It enables seamless integration for tasks including high-fidelity rendering, interactive editing, simulation, and autonomous planning in dynamic environments.
Techniques such as Gaussian-based methods, neural fields, voxel grids, and scene languages address limitations of traditional fragmented approaches.

A unified scene representation refers to a computational or parametric structure that encodes all relevant geometric, semantic, relational, and often physical or dynamic properties of a visual environment—across all entities, modalities, and tasks—within a single, compositional data/model format. The goal is simultaneous support for high-fidelity rendering, scene understanding, interactive editing, and physically plausible simulation, with seamless information flow between components such as geometry, appearance, semantics, physics, and actionability. Unified representations are motivated by the limitations of traditional surface meshes, independent point clouds, and dyadic scene graphs, which each capture only subsets of these requirements and typically lack efficient support for learning, inference, or multi-modal generative tasks.

1. Fundamental Concepts and Motivations

Unified scene representations address the fragmentation in classical computer vision and graphics pipelines, where geometry, semantics, materials, lighting, and dynamic properties are historically managed in disjoint or ad hoc formats (e.g., triangle meshes plus per-object segmentations, scene graphs, or volumetric grids). This separation hinders joint learning, robust prediction, and interactive task planning, particularly in dynamic or multi-modal environments such as robotics, simulation, and autonomous driving.

The motivations for unification are multi-fold:

Joint geometric and semantic reasoning: Accurate manipulation, navigation, or simulation requires a representation that supports perception of both metric geometry and actionable semantics (e.g., part affordances, functional roles) (Ju et al., 18 Dec 2025).
Multi-modality and cross-modal alignment: Modern pipelines require integration of appearance, language, LiDAR, depth, and semantic cues, motivating scene tokenizations that fuse features from diverse sensors and large foundation models (Deng et al., 29 Dec 2025, Ming et al., 22 Nov 2025).
Compositional and editable structures: For interactive tasks such as simulation or digital twins, the representation must support insertion, removal, or transformation of entities with full geometric and relational consistency (Zhong et al., 2023, Xia et al., 7 Oct 2025).
Efficiency and scalability: Memory and computational efficiency demands structures that avoid redundancy, support compact parameterizations, and allow real-time inference or optimization (Ming et al., 22 Nov 2025, Wu et al., 2024).

2. Techniques and Model Structures

Unified representations take diverse forms, often dictated by target applications and modalities. Principal architectural families include:

Gaussian-based representations: Scenes are parameterized as clouds of anisotropic 3D Gaussians, each encoding center, covariance, opacity, view-dependent radiance, and extended with per-primitive semantic and (optionally) motion features. Differentiable rasterization (“Gaussian splatting”) and optimization allow for real-time rendering, fusion of geometric and semantic data, and multi-modal supervision (e.g., color, depth, semantics, LiDAR) (Zhou et al., 2024, Ming et al., 22 Nov 2025, Ren et al., 2024, Deng et al., 29 Dec 2025).
Node-based neural fields: “Scene nodes” encapsulate independent neural radiance fields (NeRFs) with panoptic metadata, enabling per-object editing, compositional rendering, and efficient multi-view, multi-object inference (e.g., bounding box plus local NeRF plus CLIP embedding per node) (Zhong et al., 2023).
Voxelized or hybrid volumetric/parametric grids: Dense or hash-encoded volumetric grids store geometry (e.g., SDF), per-voxel semantic embeddings, materials, and/or illumination basis functions. These are decoded by lightweight MLPs into surface properties, lighting, and semantic labels. Explicit voxelizations facilitate efficient learning-based inverse rendering and joint estimation (Wu et al., 2024, Li et al., 2024, Chu et al., 2024).
Hierarchical or topological scene graphs and complexes: Graph-based structures unify object and part nodes with multiple edge types (spatial, functional, higher-order group relations), augmented by per-node state (e.g., open/closed, on/off), per-edge modality, and temporal links. Topological generalizations (combinatorial complexes) capture irreducible polyadic relations and facilitate higher-order reasoning (Wang et al., 10 Mar 2026, Wu et al., 19 Mar 2025, Ju et al., 18 Dec 2025).
Program- or DSL-based scene languages: Scenes are described as programs comprising compositional entity-building functions (“hierarchical grammars”) plus per-instance embeddings, supporting precise editing, control, and translation to arbitrary downstream renderers (e.g., SDFs, Gaussians, mesh assets) (Zhang et al., 2024).

3. Geometry, Semantics, and Multi-Modality

A central property of unified representations is the explicit and early fusion of geometric, semantic, and multi-modal signals. This is achieved via several mechanisms:

Per-primitive semantic embeddings: Gaussian or node primitives carry low-dimensional semantic embeddings originating from vision-LLMs (CLIP, SEEM, DINOv2, LLaMA-3.1/3.2v), permitting spatially-aligned semantic reasoning and early cross-modal alignment. For example, (Deng et al., 29 Dec 2025) attaches scene-language embeddings $f_i$ to each $G_i$ , with strong results on grounding and planning.
Multi-task learning and feature allocation: Joint optimization against photometric, geometric, semantic, and relational losses ensures that each representation encodes all salient aspects. CUS-GS (Ming et al., 22 Nov 2025) exploits a multimodal memory bank indexed by per-voxel latent queries for explicit feature fusion; UniScene3D (Mao et al., 2 Apr 2026) uses early token-level fusion of colored pointmaps.
Hierarchical semantics and relationships: Scene graphs (MomaGraph (Ju et al., 18 Dec 2025), TopoOR (Wang et al., 10 Mar 2026)) and Scene Language (Zhang et al., 2024) encode entities, their attributes, and both low-order (spatial, functional) and high-order (group) relations, with state representations for dynamic environments.
Modality bridges: Unified pipelines support efficient translation from BEV layouts or occupancy grids to video, LiDAR, and semantic maps, leveraging Gaussian-based rendering as a bridge representation (Li et al., 2024, Deng et al., 29 Dec 2025).

4. Differentiable and Editable Rendering

A core strength of contemporary unified representations lies in their differentiable and compositional rendering capabilities:

Differentiable rasterization and volume rendering: Gaussian splatting (3D-2D projection and compositing) is fully differentiable and supports photometric (color/SSIM), geometric (depth/normal), semantic (cross-entropy on per-pixel labels), and physical (LiDAR alignment, visibility) losses (Zhou et al., 2024, Ren et al., 2024, Ming et al., 22 Nov 2025).
Scene graph and node-based editing: The API in ASSIST (Zhong et al., 2023) allows translation, rotation, duplication, deletion, and cross-scene composition of scene nodes, with multi-view consistency guaranteed by the compositional integrals and per-node local radiance fields.
Multi-modal rendering targets: Many frameworks include volumetric renderers for depth, semantics, and LiDAR, in addition to RGB. For example, UniGaussian (Ren et al., 2024) models pinhole and fisheye camera models via affine Gaussian transforms, allowing unified supervision and transfer across modalities.
Support for physical reasoning: HoloScene (Xia et al., 7 Oct 2025) incorporates object-level geometry, appearance, and physics in a single attributed graph, enabling simulation, editing, and energy-based optimization.

5. Applications and Benchmarks

Unified scene representations have demonstrated substantial empirical benefits across a range of tasks:

Robotic manipulation and planning: MSGField and MomaGraph (Sheng et al., 2024, Ju et al., 18 Dec 2025) enable high-performance instruction following, grasping, and embodied task planning by encoding both geometry and functional state-action affordances, with real-robot validation.
Autonomous driving and simulation: 3D Gaussian-based world models (GaussianDWM (Deng et al., 29 Dec 2025), HERMES (Zhou et al., 24 Jan 2025), UniScene (Li et al., 2024), UniGaussian (Ren et al., 2024)) support joint scene understanding (VQA, grounding, planning, description) and high-fidelity multi-modal generation (video, LiDAR), achieving SOTA on nuScenes, NuInteract, and related datasets.
Interactive editing and simulation: Scene node/neural-field methods (Zhong et al., 2023), HoloScene (Xia et al., 7 Oct 2025), and program-based scenes (Zhang et al., 2024) enable object-wise rearrangement, editing, and compositional generation with photorealistic rendering and physical plausibility.
3D vision-language benchmarks: Uni3DR² (Chu et al., 2024), UniScene3D (Mao et al., 2 Apr 2026), and USG-Par (Wu et al., 19 Mar 2025) enable integration with LLMs for 3D VQA, region description, and cross-modal graph alignment, with superior empirical results.
Efficiency and scalability: CUS-GS (Ming et al., 22 Nov 2025) demonstrates that highly compact (6–20 MB) unified representations can achieve competitive quality and cross-modal performance with real-time inference.

6. Open Challenges and Future Directions

Despite significant advances, several limitations and open areas remain:

Scalability to very large, dynamic, or outdoor scenes: While memory- and compute-efficient unified representations exist (e.g., CUS-GS), scaling to dynamic worlds with complex/rapid motion or very large spatial extents still poses research challenges, particularly in managing token redundancy and temporal consistency (Deng et al., 29 Dec 2025, Ming et al., 22 Nov 2025).
Learning with task-driven adaptation: Many frameworks leverage frozen VLM or LLM-derived embeddings. Learning end-to-end, especially for dynamic scenes, planning, or unsupervised semantics, remains a nontrivial aim (Deng et al., 29 Dec 2025, Ming et al., 22 Nov 2025, Sheng et al., 2024).
Modalities beyond vision and language: Future work includes deep fusion with non-visual inputs (LiDAR, radar, audio), physically-based modeling (friction, mass, restitution), and integrating active perception or control feedback (Xia et al., 7 Oct 2025, Wang et al., 10 Mar 2026).
Unified token interfaces: The standardization of 3D scene “tokens” for both LLM input (understanding) and geometry-guided generation is ongoing; efficient sampling, redundancy, and scaling remain areas for innovation (Deng et al., 29 Dec 2025, Zhou et al., 24 Jan 2025).
Editability and explainability: While modern APIs allow for object manipulation and programmatic editing (Zhong et al., 2023, Zhang et al., 2024), extracting interpretable scene graphs or reasoning traces in complex, multi-modal representations is still under-constrained.

7. Comparison of Principal Approaches

Framework/Paper	Representation	Unified Modalities	Core Application Domains
3D Gaussian Splatting (Zhou et al., 2024, Ming et al., 22 Nov 2025, Ren et al., 2024, Deng et al., 29 Dec 2025)	Anisotropic Gaussian clouds with geometry, appearance, semantics	RGB, depth, semantic, LiDAR, language	Rendering, autonomous driving, multi-modal generation
Scene Node API (ASSIST (Zhong et al., 2023))	Per-object NeRF + semantic/canonical bounds	RGB, CLIP, geometry	3D simulation, interactive editing
Unified Voxelization (UniVoxel (Wu et al., 2024))	Dense/hashing 3D grids: SDF, semantics, materials, illumination	Geometry, materials, lighting	Inverse rendering, photorealistic relighting
Scene Graphs (MomaGraph (Ju et al., 18 Dec 2025), USG (Wu et al., 19 Mar 2025), TopoOR (Wang et al., 10 Mar 2026))	Graphs/hypergraphs: nodes+edges+states	Spatial, functional, group/temporal, modalities	Task-planning, simulation, safety-critical robotics
Program-based Scene Language (Zhang et al., 2024)	Hierarchical program, per-entity embedding	Text, geometry, 3D assets	High-fidelity controllable generation, editing
Fusion (Uni3DR² (Chu et al., 2024), UniScene3D (Mao et al., 2 Apr 2026))	Dense 3D grid + CLIP/SAM token fusion	Geometry, language, semantics	3D VQA, grounding, vision-language tasks

These approaches unify previously segregated scene attributes—geometry, appearance, semantics, physics, and motion—within shared, compositional, differentiably optimized representations. This enables joint learning and inference for tasks including perception, reasoning, synthesis, control, and simulation.