Unified Scene Adapter Architecture

• Unified Scene Adapter is a design principle that preserves scene-level coherence by adaptively unifying analysis, synthesis, and control across heterogeneous tasks.
• It employs modular pipelines—such as scene graphs, panoramic flow-matching, and candidate refinement—to integrate diverse modalities while maintaining consistent semantic frameworks.
• Applications span virtual scene synthesis, panoramic generation, controllable video modeling, and autonomous driving, highlighting its potential for robust scene adaptation.

Searching arXiv for the specified papers and related "Unified Scene Adapter" terminology to ground the article with current references. arXiv search query: "(Pereira et al., 2024) OR (Huang et al., 30 Oct 2025) OR (Yin et al., 18 Jun 2026) OR (Wu et al., 3 Feb 2026) Unified Scene Adapter" Unified Scene Adapter denotes an architectural pattern for preserving scene-level coherence while translating between heterogeneous representations, controls, or task regimes. In one formulation, it is a modular pipeline that converts visual input into a structured semantic description and then into editable 3D virtual synthesis (Pereira et al., 2024). In another, it is an explicit conditioning module inside a video world model, where separate residual branches preserve source-world structure and transfer weather state from a single first frame (Yin et al., 18 Jun 2026). Related work extends the same unifying logic to panoramic generation and intrinsic perception through a lightweight cross-modal adapter (Huang et al., 30 Oct 2025), and to end-to-end autonomous driving through a unified candidate set that preserves fixed priors while adding scene-adaptive refinement (Wu et al., 3 Feb 2026). Taken together, these works suggest a family of systems in which the adapter is not merely an interface layer, but the mechanism that keeps scene semantics, control signals, or planning priors accessible across the full processing chain.

1. Conceptual scope and defining properties

A Unified Scene Adapter is characterized by three recurrent properties. First, it is adaptive: processing modules, conditions, or candidate refinements are chosen or activated according to scene type, output requirements, or interaction complexity. Second, it is coherent: object types, attributes, relations, geometry anchors, or candidate semantics are maintained in a shared substrate rather than recomputed independently at each stage. Third, it is unified: analysis, description, synthesis, control, or selection are treated as stages of one workflow rather than as disjoint applications (Pereira et al., 2024).

This pattern appears in different technical guises. In the visual-scene blueprint, unification is achieved by a knowledge base or ontology, a scene-graph representation, and an overview stage that consumes the same semantic structure used for analysis and editing (Pereira et al., 2024). In OmniX, unification is imposed by treating panoramic generation, intrinsic perception, and completion as a multi-condition flow-matching problem over spatially aligned 2D modalities (Huang et al., 30 Oct 2025). In Holo-World, the adapter factorizes preservation and state transfer into disjoint parameter subspaces, so that camera-consistent structure and weather-dependent appearance do not interfere (Yin et al., 18 Jun 2026). In CdDrive, unification occurs in decision space: the planner keeps original vocabulary trajectories, adds diffusion-refined alternatives, and scores both in a shared module (Wu et al., 3 Feb 2026).

A plausible implication is that “Unified Scene Adapter” is best understood less as a single standardized architecture than as a design principle: one preserves a stable scene scaffold or prior, and one exposes controlled adaptation on top of that scaffold.

2. Architectural blueprint: from visual scene understanding to virtual synthesis

The most explicit system-level blueprint appears in "A transition towards virtual representations of visual scenes" (Pereira et al., 2024). The architecture comprises Algorithm Selector, Data Orchestrator, Visual Scene Analysis, Scene Description, and Scene Synthesis. Its stated flow is:

visual input (images/videos) → Algorithm Selector → Visual Scene Analysis → Scene Description → Scene Synthesis.

The motivation is the limitation of scene understanding that “traditional[ly] involve[d] separate and specialized algorithms for different tasks,” producing “fragmented and disjointed analysis.” The paper frames the goal as a framework capable of “analyz[e] a scene, describ[e] and synthesize it,” while keeping accessible “diverse objects, spatial relationships, contextual cues, and temporal dynamics” throughout processing (Pereira et al., 2024).

The Algorithm Selector is the adaptability layer. It receives user requirements and scene constraints and selects algorithms and parameters according to “the type of scene that is to be analyzed, the type of information that is to be described and the desired output.” In the proposal, this selection is rule-based. The Data Orchestrator is the unifying backbone: it “binds and connects all components in the system” and provides “a common ground” for object types, attributes, and relationships. The proposed implementations are RDF, OWL, and JSON-LD, with knowledge-base population from MS COCO, Open Images, ImageNet, and external ontologies.

The Visual Scene Analysis stage is a multi-stage pipeline including preprocessing, object detection and recognition, semantic segmentation, pose estimation, scene classification, tracking, action recognition, and group behavior analysis. A central requirement is to avoid redundant computation by sharing intermediate results across sub-tasks. The Scene Description module is the semantic bridge: it “enhance[s] the analysis with semantic information” and “provide[s] means to describe the entire scene in a well structured way.” The preferred representation is the scene graph, where nodes represent objects, edges represent relationships, and node attributes encode properties such as color, size, and shape. Ontology-based reasoning is then used to “infer logical consequences” and correct inconsistencies. The resulting graph can be serialized to JSON, XML, or binary formats, and the paper explicitly emphasizes that users may manually modify the scene description without changing the perception pipeline.

The Scene Synthesis stage instantiates 3D models, positions them according to the scene graph, renders them with lighting, textures, and shading, and exports the result in formats such as USD, FBX, OBJ, and COLLADA. The module may employ geometric or voxel-based modeling, template-based model retrieval, rendering methods including ray tracing, rasterization, and global illumination, and generative methods such as GANs or NeRF. The paper introduces no major derived formulas; its formalism is representational rather than mathematical. Even the proof-of-concept graph semantics remain symbolic, including a special “+has” relation for analyzed data.

The proof of concept demonstrates hybrid video generation with virtual avatars. Humans are detected, segmented, and tracked with YOLO-v8; pose is estimated with VIBE; body representation uses SMPL; optional inpainting uses E$^{2}$FGVI; and HMDB51 videos serve as test material. The scene description stores person IDs, pose, and SMPL data in a JSON-serialized scene graph, after which the synthesis stage produces 3D avatars that replicate actions. The reported controllability includes manually altering the representation so that avatars perform rotated or reversed actions. This supports the paper’s central claim that the intermediate representation is not only descriptive, but also an editable adapter between perception and synthesis.

3. Panoramic cross-modal adaptation and graphics-ready 3D scenes

"OmniX: From Unified Panoramic Generation and Perception to Graphics-Ready 3D Scenes" formulates a unified scene adapter around a pre-trained 2D flow-matching generative model, specifically FLUX.1-dev, extended by a lightweight cross-modal adapter (Huang et al., 30 Oct 2025). Rather than train a panoramic foundation model from scratch, it repurposes 2D generative priors for panoramic generation, panoramic intrinsic perception, panoramic completion, and 3D scene construction.

The key mathematical formulation is a multi-condition generalization of flow matching:

$$\hat{\mathbf{z}}_1 = \mathbf{z}_0 + \int_0^1 f_\theta(\mathbf{z}_t, \mathbf{c}^0, \mathbf{c}^1, ..., y, t)\, dt.$$

The conditions $\{\mathbf{c}^i\}$ are spatially aligned with the target latent, so all inputs can be treated as 2D aligned modalities. The supported task families are: panoramic generation and completion using a masked panorama and mask; panoramic perception with RGB $\to X$, where $X$ can be Euclidean distance, normal, albedo, roughness, or metallic; and guided panoramic perception for progressive completion. Optionally, camera rays can be used as additional condition inputs.

The paper evaluates three adapter strategies for incorporating multiple conditions into a DiT-style flow-matching model: Shared-Branch, Shared-Adapter, and Separate-Adapter. The final design is Separate-Adapter, which uses different LoRA modules for different modalities or roles. The paper states that these inputs and outputs share the same 2D positional encoding because they are spatially aligned. This is the adapter’s principal unifying move: heterogeneous panoramic tasks are cast into one conditioning interface while perturbing the pretrained weight distribution only minimally.

OmniX is trained in part on PanoX, a synthetic panorama dataset built in Unreal Engine 5. PanoX contains both indoor and outdoor scenes and provides aligned panoramic RGB, distance maps, world normals, albedo, roughness, and metallic channels, together with text captions extracted by Florence 2. The reported scale is more than 10,000 instances, corresponding to 60,000 panoramic images across modalities, over 8 large-scale 3D scenes consisting of 5 indoor and 3 outdoor scenes. The split is PanoX-Train, PanoX-Val, PanoX-Test with an 8:1:1 split over six scenes, plus PanoX-OutDomain for generalization testing on the remaining two scenes. Additional training and evaluation resources include Structured3D and HDR360-UHD. The implementation uses PyTorch, four Ascend 910B NPUs, batch size = 1, AdamW, learning rate = 1e-4, no LR decay, and panorama resizing to $512 \times 1024$. Text prompts for generation are obtained with BLIP 2.

Two further details are central to the adapter’s practicality. For ERP seam continuity, the paper follows LayerPano3D and uses horizontal blending. For completion and guided perception, it uses occlusion-aware mask sampling derived from panoramic distance maps, random 3D displacement, and ray-intersection-based occlusion estimation. These masks are used during training so that completion is grounded in realistic scene holes.

The payoff is a graphics-ready 3D scene rather than an appearance-only panorama. A panoramic distance map is projected into 3D as mesh vertices, with connectivity inferred from neighboring pixels and relative distances, and remaining modalities—albedo, normal, roughness, metallic—are mapped to the mesh through spherical UV unwrapping. This yields a PBR-ready 3D asset supporting free exploration, PBR-based relighting, and physical simulation. The paper’s argument is that geometry plus intrinsic material maps makes the output suitable for relighting, material editing, physically based rendering, simulation-based interaction, and more faithful reconstruction than pipelines that infer only depth.

4. Residual-space factorization in controllable video world models

"Holo-World: Unified Camera, Object and Weather Control for Video World Model" introduces a Unified Scene Adapter (UniSA) as an explicit architectural module for a harder control regime: first-frame-anchored source-to-state generation from a single image (Yin et al., 18 Jun 2026). The model must follow explicit camera and object controls while either preserving the observed world or transferring it into a target weather state. The paper identifies a tension: the residuals needed for camera-consistent structure, object adherence, and geometry preservation are not the same as those needed for rain, snow, fog, wet surfaces, or overcast lighting.

UniSA is attached to a frozen Wan2.1-T2V-14B video backbone. The backbone, VAE, and text encoder remain fixed; only the adapter is trained. The architecture consists of two parallel branches, a World Adapter and a State Adapter, with distinct parameter subspaces. The control groups are defined as

$$C_{\mathrm{world}} = \{R_{\mathrm{rgb}}, R_{\mathrm{depth}}, R_{\mathrm{normal}}, C_{\mathrm{bbox}}\}, \qquad C_{\mathrm{state}} = \{R_{\mathrm{rgb}}\}.$$

Here, rendered RGB, depth, and normal videos are built from first-frame geometry and the camera trajectory, while $C_{\mathrm{bbox}}$ is an object-control video derived from propagated masks. The same rendered RGB also acts as a source appearance anchor for the state branch. The target-weather video is reserved for supervision only; it is not used as a conditioning input.

The factorization is explicit in the layerwise hints

$$h^w_{\ell} = A^w_{\ell}(C_{\mathrm{world}}, c_{\mathrm{scene}}), \qquad h^s_{\ell} = A^s_{\ell}(C_{\mathrm{state}}, c_{\mathrm{state}}),$$

and in the injected-layer update

$$x_\ell = F_\ell(x_{\ell-1}, c) + \lambda_w h^w_\ell + \lambda_s h^s_\ell .$$

The adapters share the same overall DiT-style structure and output shape but do not share trainable parameters. The injected layers are $$\hat{\mathbf{z}}_1 = \mathbf{z}_0 + \int_0^1 f_\theta(\mathbf{z}_t, \mathbf{c}^0, \mathbf{c}^1, ..., y, t)\, dt.$$0, initialized from corresponding Wan blocks. The world residual is always active with $$\hat{\mathbf{z}}_1 = \mathbf{z}_0 + \int_0^1 f_\theta(\mathbf{z}_t, \mathbf{c}^0, \mathbf{c}^1, ..., y, t)\, dt.$$1, while the state residual is gated by sample type:

$$\hat{\mathbf{z}}_1 = \mathbf{z}_0 + \int_0^1 f_\theta(\mathbf{z}_t, \mathbf{c}^0, \mathbf{c}^1, ..., y, t)\, dt.$$2

This prevents real no-edit samples from forcing the State Adapter to learn identity mappings.

The training objective remains the standard Wan flow-matching loss; UniSA changes only how conditions enter the model. Functionally, rendered background, geometry buffers, and object controls serve as the structural backbone. Depth and normal buffers act as geometry anchors that reduce structure drift and floating texture artifacts, while object bounding-box videos constrain dynamic entities. The weather prompt specifies how the scene should be rendered—fog density, cloudiness, rain, snow, wetness, or other atmospheric cues—without changing which world is being followed. The paper repeatedly emphasizes this separation between scaffold and rendering state.

UniSA is paired with Scene-Weather Decomposed CFG (SW-CFG) because standard CFG still mixes scene and weather guidance at inference time. SW-CFG decomposes guidance into a scene residual and a weather residual, allowing $$\hat{\mathbf{z}}_1 = \mathbf{z}_0 + \int_0^1 f_\theta(\mathbf{z}_t, \mathbf{c}^0, \mathbf{c}^1, ..., y, t)\, dt.$$3 to remain low while $$\hat{\mathbf{z}}_1 = \mathbf{z}_0 + \int_0^1 f_\theta(\mathbf{z}_t, \mathbf{c}^0, \mathbf{c}^1, ..., y, t)\, dt.$$4 is increased to strengthen weather effects. The ablations reported in the paper are aligned with this interpretation: adding G-buffer controls improves background preservation and weather metrics; adding UniSA improves background metrics and VLM weather evaluation, though one ablation stage shows a slight drop in Weather Alignment; and the best balance is obtained when UniSA is combined with SW-CFG. Implementation details include training on 81-frame $$\hat{\mathbf{z}}_1 = \mathbf{z}_0 + \int_0^1 f_\theta(\mathbf{z}_t, \mathbf{c}^0, \mathbf{c}^1, ..., y, t)\, dt.$$5 clips, batch size 1, gradient accumulation 2, learning rate $$\hat{\mathbf{z}}_1 = \mathbf{z}_0 + \int_0^1 f_\theta(\mathbf{z}_t, \mathbf{c}^0, \mathbf{c}^1, ..., y, t)\, dt.$$6, weight decay 0.03, 3,000 steps on 8 H200 GPUs, and inference with 50 denoising steps and default SW-CFG scales $$\hat{\mathbf{z}}_1 = \mathbf{z}_0 + \int_0^1 f_\theta(\mathbf{z}_t, \mathbf{c}^0, \mathbf{c}^1, ..., y, t)\, dt.$$7 and $$\hat{\mathbf{z}}_1 = \mathbf{z}_0 + \int_0^1 f_\theta(\mathbf{z}_t, \mathbf{c}^0, \mathbf{c}^1, ..., y, t)\, dt.$$8.

5. Unified candidate adaptation in end-to-end autonomous driving

"A Unified Candidate Set with Scene-Adaptive Refinement via Diffusion for End-to-End Autonomous Driving" provides a different but structurally analogous interpretation of the adapter idea (Wu et al., 3 Feb 2026). Instead of mediating between modalities, CdDrive mediates between fixed priors and scene-conditioned adaptation. The paper argues that a fixed trajectory vocabulary gives stable coverage in routine driving but may miss optimal trajectories in complex interactions, while scene-adaptive refinement may produce over-correction by perturbing already strong trajectories in simple scenes.

The planner outputs a future trajectory

$$\hat{\mathbf{z}}_1 = \mathbf{z}_0 + \int_0^1 f_\theta(\mathbf{z}_t, \mathbf{c}^0, \mathbf{c}^1, ..., y, t)\, dt.$$9

with position sequence

$\{\mathbf{c}^i\}$0

and a vocabulary built offline from clustered expert demonstrations,

$\{\mathbf{c}^i\}$1

Each anchor $\{\mathbf{c}^i\}$2 is refined by a truncated conditional diffusion process conditioned on scene observation $\{\mathbf{c}^i\}$3. Rather than predict a trajectory from scratch, the model predicts a refinement on top of the anchor,

$\{\mathbf{c}^i\}$4

The final candidate pool is then

$\{\mathbf{c}^i\}$5

This is the core unification step: original vocabulary trajectories and diffusion-refined trajectories coexist in one candidate set.

Before scoring, each candidate is rolled out through a latent world model,

$\{\mathbf{c}^i\}$6

and scored by a shared selection module,

$\{\mathbf{c}^i\}$7

Because both candidate types are scored in the same space, adaptation is optional rather than mandatory. In routine scenes, the selector may retain the original anchor; in highly interactive scenes, it may choose a refined candidate.

A second adapter layer operates at the noise level through HATNA (Horizon-Aware Trajectory Noise Adapter). The paper identifies the problem that i.i.d. Gaussian noise can introduce far-horizon kinks and piecewise-linear artifacts. HATNA modifies the injected noise,

$\{\mathbf{c}^i\}$8

using temporal low-pass smoothing and a horizon-aware scale vector that increases with horizon. The reported interpretation is direct: near-horizon waypoints receive smaller noise for local stability, far-horizon waypoints receive larger noise to reflect greater uncertainty, and smoothing suppresses zig-zag perturbations.

The reported results quantify the value of this unified candidate strategy. On NAVSIM v1, with PDMS

$\{\mathbf{c}^i\}$9

CdDrive with ResNet-34 achieves PDMS: 89.2, NC: 98.7, DAC: 97.5, EP: 82.7, TTC: 95.4, and Comf: 100.0. On NAVSIM v2, with

$\to X$0

CdDrive achieves EPDMS: 86.4. The ablations further support the design: on NAVSIM v1, Vocab-only yields PDMS around 87.4, Diffusion-only yields around 88.4–88.6, and CdDrive (vocab + diffusion) yields 89.2. Without HATNA, PDMS drops to about 89.0; with HATNA, TTC improves to 95.4, Comfort reaches 100.0, and PDMS improves to 89.2. Diffusion-based refinement also outperforms regression-based refinement in both candidate settings, and the reported ADE is 1.05 for diffusion refinement versus 1.46 for regression-based refinement.

6. Applications, limitations, and interpretive boundaries

The application range associated with unified scene adaptation is broad. The 2024 architectural blueprint lists surveillance and security, sports, retail, entertainment, virtual reality, augmented reality, mixed reality, content creation for movies, advertisements, and virtual worlds, dataset augmentation, gamification and serious games, patient treatment and rehabilitation, autonomous driving / UAV adaptation, and privacy-preserving scene storage and visualization (Pereira et al., 2024). OmniX specializes this logic toward graphics-ready 3D scenes suitable for physically based rendering, relighting, and simulation (Huang et al., 30 Oct 2025). Holo-World places it in controllable video generation with explicit camera, object, and weather control from a single first frame (Yin et al., 18 Jun 2026). CdDrive places it in multimodal planning, where unification governs when fixed priors should be preserved and when scene-adaptive candidates should replace them (Wu et al., 3 Feb 2026).

Several interpretive boundaries are equally important. The 2024 blueprint is explicitly a framework proposal and proof of feasibility, not a complete solved system; its ontology and rule system are not fully developed, the algorithm selector remains rule-based, scene description could be much finer-grained, and robust support for highly complex scenes, dynamic scenes, and real-time applications remains future work (Pereira et al., 2024). OmniX suggests that strong pretrained 2D priors can be reused for panoramic tasks, but its unification depends on spatially aligned modalities and a completion strategy that still requires careful seam handling and geometry-derived masking (Huang et al., 30 Oct 2025). Holo-World argues that ordinary single-residual conditioning is insufficient when preservation and weather transfer are entangled; its own ablations indicate that UniSA alone does not fully solve inference-time interference and that SW-CFG is needed for the best preservation-weather trade-off (Yin et al., 18 Jun 2026). CdDrive shows that merely enlarging a fixed vocabulary from 256 to 512 does not substitute for scene-adaptive refinement, but it also makes clear that adaptation can be harmful when applied indiscriminately, hence the emphasis on preserving the original vocabulary to avoid over-correction (Wu et al., 3 Feb 2026).

A common misconception would be to reduce a Unified Scene Adapter to a narrow control head, a post-hoc renderer, or a lightweight wrapper around pre-existing models. The cited works collectively indicate a stronger role. In each case, the adapter determines how scene information is normalized, partitioned, or arbitrated so that later stages do not have to reconstruct the scene model from scratch. This suggests that the distinctive contribution of unified scene adaptation is not any single backbone or modality, but the preservation of a controllable scene scaffold across heterogeneous computations.