Structured Scene Interface (SSI)

• Structured Scene Interface (SSI) is a class of representations that externalizes scene organization for enhanced reasoning and task-specific control.
• SSI frameworks include editable scene graphs, programmatic scene descriptions, and queryable interfaces tailored for image editing, robotics, and language grounding.
• Its explicit structure reduces ambiguity, improves data efficiency, and supports long-horizon reasoning across various perceptual and control tasks.

Searching arXiv for papers on Structured Scene Interface and closely related formulations. Structured Scene Interface (SSI) denotes a class of structured intermediate representations that expose scene organization in a form usable by downstream reasoning, generation, editing, or control. Across recent work, SSI appears both as an explicit term and as a close conceptual fit for related constructs: editable scene graphs for image editing, RGB-only scene abstractions for manipulation, Cypher-accessible 3D scene graphs for robot language grounding, LocalCogMap-based structured scene reasoning for multimodal models, programmatic scene descriptions composed of words and embeddings, and scene abstractions for situated lexical meaning (Phan et al., 15 Jun 2026, Wang et al., 25 Jun 2026, Ray et al., 18 Oct 2025, Zhang et al., 28 Feb 2026, Zhang et al., 2024, Cho et al., 21 May 2026). The literature suggests that SSI is best understood not as a single canonical data structure, but as an interface family whose central property is explicit scene structure.

1. Conceptual scope and definitional boundaries

Several papers use the phrase “Structured Scene Interface” directly, while others instantiate the same abstraction without adopting the exact label. In image editing, SSI is a visual, editable scene graph that replaces brittle free-form prompting with direct manipulation of scene structure (Phan et al., 15 Jun 2026). In robotic manipulation, SSI is a unified, RGB-only intermediate representation that jointly encodes monocular depth features, language-grounded object layouts, and instruction-conditioned 2D motion trajectories (Wang et al., 25 Jun 2026). In robot language grounding over 3D scene graphs, SSI takes the form of a structured database interface in which the LLM issues Cypher queries rather than consuming the entire graph as serialized text (Ray et al., 18 Oct 2025).

Other works define structurally analogous interfaces. SSR does not explicitly use the phrase, but its LocalCogMap, 3D grounding coordinate framework, and MultiQA textualization are described as the closest match to an interface abstraction bridging perception and language reasoning (Zhang et al., 28 Feb 2026). “The Scene Language” represents scenes as a triple of programs, words, and embeddings, explicitly separating structure, semantics, and identity (Zhang et al., 2024). “Scene Abstraction for Lexical Semantics” treats a usage instance as a structured interpretive scene composed of a Contextual Scene and an Expression Profile, thereby extending scene interfaces beyond physical geometry into situated meaning (Cho et al., 21 May 2026).

Work	SSI formulation	Primary function
SSR	LocalCogMap plus MultiQA textualization	Structured spatial reasoning
SceneCraft	Scene graph $G(O,R)$	Interactive image editing
Scene Language	$\Phi(s) := (W, P, Z)$	Scene generation, rendering, editing
SSI-Policy	RGB-only structured scene representation	Few-shot robotic manipulation
RoBoSR	$M_{sg} = (O_t, E_t)$	Embodied reasoning and planning
3DSG interface	Graph database plus Cypher	Grounded QA and PDDL grounding
Scene Abstraction	Contextual Scene plus Expression Profile	Situated lexical semantics

A common misconception is that SSI is synonymous with a conventional scene graph. The recent literature suggests a broader category: some SSIs are graphs, some are executable programs, some are multimodal tensors and trajectory tokens, and some are structured natural-language schemas. What unifies them is the decision to externalize scene structure rather than leave it implicit in an opaque latent state.

2. Core representational patterns

The most common SSI pattern is the object–relation graph. SceneCraft formalizes an image as $G(O,R)$, where $O=\{o_1,o_2,\dots,o_n\}$ is the set of objects and $R=\{r_1,r_2,\dots,r_m\}$ is the set of relations, with each relation represented as a triplet $(o_i,r_k,o_j)$ (Phan et al., 15 Jun 2026). RoBoSR adopts a closely related object-centric graph for robotics, $M_{sg} = (O_t, E_t)$, whose nodes encode objects, attributes, states, affordances, and child parts, while edges encode relations such as on, inside, adjacent, tilted against, and partially occluded (Hu et al., 23 Jun 2026). SSR uses a more localized decomposition: instead of a dense global graph, it represents global layouts as a chain of independent local triplets defined by relative coordinates in a normalized $10 \times 10$ grid (Zhang et al., 28 Feb 2026).

A second pattern is the programmatic or executable interface. The Scene Language defines a scene as $\Phi(s) := (W, P, Z)$, where $\Phi(s) := (W, P, Z)$0 is a set of words or phrases, $\Phi(s) := (W, P, Z)$1 is a program consisting of entity functions indexed by those words, and $\Phi(s) := (W, P, Z)$2 is an ordered list of neural embeddings (Zhang et al., 2024). Its DSL uses macros such as bind, retrieve, call, transform, union, and union-loop, making hierarchy, repetition, and relative pose explicit. This differs from pairwise-relation graph formalisms by treating the scene as something to be computed rather than merely listed.

A third pattern is the multimodal policy interface. SSI-Policy writes the intermediate state as $\Phi(s) := (W, P, Z)$3, followed by $\Phi(s) := (W, P, Z)$4, where the structured representation combines geometry, language-grounded salience, and motion (Wang et al., 25 Jun 2026). The trajectory component is $\Phi(s) := (W, P, Z)$5, with each $\Phi(s) := (W, P, Z)$6. In this formulation, structure appears not as symbolic edges alone but as a coordinated representation of depth feature maps, layout maps, and future pixel-space trajectories.

A fourth pattern is the queryable interface. For large 3D scene graphs, “Structured Interfaces for Automated Reasoning with 3D Scene Graphs” defines a hierarchical graph $\Phi(s) := (W, P, Z)$7 organized by ontology layer and exposes that graph through Cypher queries over a graph database (Ray et al., 18 Oct 2025). The interface is therefore neither a prompt nor a learned latent, but a tool-accessible symbolic world model.

A fifth pattern is the semantic scene schema. Scene Abstraction models lexical meaning through a Contextual Scene composed of Events, Entities, and Setting, together with an Expression Profile composed of Engaged Events, Generalizable Properties, and Evoked Emotions (Cho et al., 21 May 2026). This broadens SSI from physical scene layout to scene-conditioned interpretation.

3. Structured scene reasoning for spatial intelligence

SSR provides one of the clearest formulations of SSI as a bridge between perception and geometric reasoning. It starts from the claim that current multimodal LLMs are weak at spatial intelligence, especially metric geometry, layout consistency, and 3D-aware reasoning, and attributes this to costly modality alignment for 3D inputs and the lack of fine-grained structured scene modeling (Zhang et al., 28 Feb 2026). Its answer is two-level: a feature-level bridge that anchors 3D geometry to pre-aligned 2D visual semantics through cross-modal addition and token interleaving, and a reasoning-level bridge that converts scenes into LocalCogMap triplets and then into MultiQA scaffolds.

The model is a dual-branch MLLM with a 2D branch for appearance and semantic features and a 3D branch using VGGT. The alignment pipeline extracts 3D spatial features from VGGT intermediate layers over $\Phi(s) := (W, P, Z)$8 sampled frames, projects them into the 2D visual embedding space with a lightweight MLP, fuses 3D and 2D features by element-wise addition, and then projects the fused tokens into the LLM embedding space. Instead of concatenating all visual tokens and then all spatial tokens, SSR interleaves them frame-by-frame. The stated rationale is that naive concatenation creates a large positional gap between modalities under M-RoPE, whereas interleaving reduces this gap and improves fine-grained alignment (Zhang et al., 28 Feb 2026).

Its structured scene representation, LocalCogMap, decomposes a scene into independent local triplets consisting of two anchor objects and one target object. The target is localized in a normalized $\Phi(s) := (W, P, Z)$9 grid defined by the anchors. SSR emphasizes that this is a quantitative, discrete local coordinate system rather than a qualitative graph of relations such as left-of or inside-of. To avoid the $M_{sg} = (O_t, E_t)$0 cost of enumerating all triplets, it uses an incremental generation algorithm: it initializes from an object triplet with pairwise distances below a threshold $M_{sg} = (O_t, E_t)$1, then repeatedly adds a new object $M_{sg} = (O_t, E_t)$2 by choosing two already-in-graph objects minimizing $M_{sg} = (O_t, E_t)$3. The resulting graph is textualized as MultiQA, so that each triplet becomes a reusable QA-style intermediate step.

SSR further extends the interface to global 3D grounding with a canonical 7-DoF object parameterization,

$M_{sg} = (O_t, E_t)$4

and a unified coordinate system whose origin is the camera position in the first frame, whose positive $M_{sg} = (O_t, E_t)$5-axis is the projection of the camera optical axis onto the ground plane, and whose frame is right-handed Cartesian. This normalization is used to unify metadata from ScanNet, ScanNet++, ARKitScenes, VLA3D, ScanRefer, ReferIt3D, and Multi3DRefer. At the empirical level, SSR reports roughly 5.6M samples for stage 1, 917K samples for stage 2, and a structured scene dataset of about 190K samples. At a 7B parameter scale, SSR-3D scores 73.9 on VSI-Bench, exceeding InternVL3.5-241B by 4.4 points; SSR-2D alone scores 71.9, still exceeding InternVL3.5-241B by 2.4 points; on VSI-Bench Debiased, SSR-3D exceeds Cambrian-S by 13 points; on SpaCE-10, SSR-2D beats InternVL3.5 by 10.7 points; and LocalCogMap achieves a mean prediction error of 0.71 units, better than the global scheme introduced in VSI-Bench (Zhang et al., 28 Feb 2026).

These results motivate a central interpretation of SSI in spatial intelligence: the interface is not merely a storage format, but a scaffold that makes geometric inference legible to language-centric models.

4. Interactive editing and programmatic scene generation

SceneCraft instantiates SSI as an interactive control layer for image editing. Its motivation is that text-only editing becomes unreliable in complex multi-object scenes because natural language is too ambiguous for precise spatial and relational control (Phan et al., 15 Jun 2026). The system therefore parses the image into a scene graph, allows direct graph editing, and translates those edits into context-aware prompts sent to multiple image-editing models. The graph is built automatically in two stages. First, Detic proposes candidate object regions broadly, Grounding DINO localizes the main objects precisely, boxes are merged by IoU-based matching, and an LLM assigns semantic labels and unique IDs such as “kitten 1” or “ball 1.” Second, given the object set and the original image, an LLM infers semantic relations from bounding-box context.

The editing workspace contains a Scene Graph Editor and an Image Editor. Users manipulate the graph through removal, addition, and replacement operations. A deletion removes a node while preserving surrounding background context. An addition creates a new node and connects it by dragging an edge to an anchor object so that placement is constrained. A replacement swaps one node’s concept while preserving layout, relational context, and lighting consistency. The prompt pipeline is explicitly a two-step bridge: graph edit $M_{sg} = (O_t, E_t)$6 raw instruction $M_{sg} = (O_t, E_t)$7 LLM refinement into a detailed prompt. The paper characterizes the LLM as a semantic compiler from structural edits to natural-language instructions. SceneCraft then dispatches the structured prompt to FLUX.1 Kontext, Qwen Image Editing, and Gemini 2.5 Flash Image in order to provide multiple candidates.

Its evaluation uses Element Composition (EC), Relationship Alignment (RA), Image Quality (IQ), pairwise winning rate,

$M_{sg} = (O_t, E_t)$8

720 pairwise judgments, and 1–5 MOS ratings. SceneCraft reports MOS scores of 4.2 for EC, 4.1 for RA, and 4.4 for IQ, whereas baseline scores are described as roughly in the 3.1–3.9 range. Its pairwise win rates are 71.0% EC, 69.8% RA, and 68.5% IQ versus Qwen Image Editing; 77.3%, 75.1%, and 74.2% versus FLUX.1 Kontext; and 72.6%, 70.3%, and 70.6% versus Gemini 2.5 Flash Image. The system study further reports significantly lower cognitive workload and frustration, with Wilcoxon signed-rank tests at $M_{sg} = (O_t, E_t)$9 (Phan et al., 15 Jun 2026). The paper is equally explicit about limitations: success depends on upstream detection and parsing quality, the system inherits the limitations of the underlying generators, and users still inspect multiple results manually.

“The Scene Language” presents a different SSI design. Rather than a node-link editor, it uses a representation $G(O,R)$0 in which structure is encoded by executable entity programs, semantics by words, and visual identity by neural embeddings (Zhang et al., 2024). The representation is inferred in a training-free manner from pre-trained LLMs, with CLIP text encoding for text inputs and GroundingSAM plus textual inversion for image-conditioned inference. Rendering is modular: the scene program can be mapped to primitive-based renderers such as Mitsuba, neural or SDS-based renderers such as a Gaussian splatting renderer, asset-based renderers such as Minecraft, or hybrid text-to-image renderers such as MIGC. Compared with scene graphs, the paper argues that the Scene LLMs hierarchy, repetition, and precise transforms more explicitly, while preserving editability and identity.

Taken together, these two systems show that SSI in image generation and editing need not be limited to prompt augmentation. It can be a direct manipulation interface, a semantic compiler, or an executable scene program.

5. Embodied robotics and queryable world models

In robotics, SSI functions as the perception–action interface. SSI-Policy defines SSI as a robot-agnostic, RGB-only intermediate representation that decouples perception from control and is trainable from action-free video (Wang et al., 25 Jun 2026). Its three components are monocular depth features from Depth Anything V2, a language-grounded layout map $G(O,R)$1 produced with Grounding DINO and rasterized by filling each detected region with its confidence score while retaining the maximum under overlap, and instruction-conditioned 2D motion trajectories $G(O,R)$2. Trajectory start points are sampled half uniformly across the image and half from high-confidence regions in $G(O,R)$3. Downstream, a Diffusion Action Planner fuses modality-specific CNN features, appended trajectory tokens, a learnable $G(O,R)$4 token, positional and modality-type encodings, and proprioception to form the condition vector $G(O,R)$5, then optimizes

$G(O,R)$6

On LIBERO with 10 demonstrations per task, SSI-Policy reports Spatial 83.50, Object 95.00, Goal 82.50, Long 60.67, and Average 80.42, compared with 65.67 average for ATM-DP, matching the claim of nearly 15% improvement. In the 50-demo setting it reports Average 91.25, which is second-best among methods trained without external data and only 0.65% behind OFT. Ablations show Motion + Depth at 76.92 average, Motion + Layout map at 77.09 average, the full model at 80.42, and a drop from 80.42 to 78.46 when the hybrid trajectory start-point sampling is removed. An SSI-only policy retaining SSI plus proprioception preserves about 98% of full performance. Real-world validation spans 13 tasks; on six spatial tasks it reports 80.0% versus 43.33% for Diffusion Policy, and on contact-rich manipulation 54% versus 29% (Wang et al., 25 Jun 2026).

RoBoSR adopts a more explicitly symbolic SSI. It formulates manipulation as step-wise state transitions over a semantically grounded, object-centric scene graph $G(O,R)$7 derived from RGB-D input (Hu et al., 23 Jun 2026). Nodes represent objects with functional components, articulated child parts, discrete states such as open/closed, and keypoints when relevant; edges encode relations such as on, inside, adjacent, tilted against, and partially occluded. The training objectives include scene abstraction $G(O,R)$8, goal-conditioned planning $G(O,R)$9, forward scene prediction $O=\{o_1,o_2,\dots,o_n\}$0, inverse action reasoning $O=\{o_1,o_2,\dots,o_n\}$1, and goal interpretation $O=\{o_1,o_2,\dots,o_n\}$2. Its Manip-Cognition-1.6M dataset is constructed from Epic-Kitchens-100, EgoPlan, Behavior-1K, and ENACT, with 6k task trajectories and 1.6 million samples, including 15k Scene Abstraction Data, 1.3M Action Planning Data, and 0.3M Goal Interpretation Data. On GSR-bench, RoBoSR-8B reports SOD 0.700/0.610/0.520, SAS 0.330/0.230/0.150, and GCG 0.300/0.190/0.080 across easy/medium/difficult, while RoBoSR-8B-FT reports SOD 0.975/0.917/0.892, SAS 0.934/0.812/0.762, and GCG 0.891/0.291/0.131. The paper frames this as causally constrained state-space reasoning reinforced by rewards for step-wise action constraint, scene-graph grounding, and termination correctness (Hu et al., 23 Jun 2026).

A third robotic SSI design appears in “Structured Interfaces for Automated Reasoning with 3D Scene Graphs.” Here the issue is not learning a scene representation from raw pixels, but grounding language in a large hierarchical 3DSG stored as a graph database (Ray et al., 18 Oct 2025). The paper argues that serializing the entire scene graph into prompt text does not scale: the kilometer-scale outdoor graph is about 17 million tokens in serialized form. Instead, an LLM accesses the graph through Cypher. The framework evaluates instruction grounding to PDDL and scene question answering over a small indoor graph with 65 objects, 96 mesh places, and 5 regions, and a large outdoor graph with 314 objects, 15,944 mesh places, and 124 regions. For GPT-4.1, agentic Cypher achieves QA 0.85 on the small graph and 0.77 on the large graph, and PDDL 0.65 and 0.56 respectively. The context-window baseline reports QA 0.75 and 0.33, and PDDL 0.65 and 0.41. Token counts show the sharpest contrast: on the PDDL task, context-window input is 8,735 tokens on the small graph and 582,202 on the large graph, whereas agentic Cypher uses 2,366 and 2,395 input tokens, with tool tokens 205 and 1,596 (Ray et al., 18 Oct 2025).

These systems differ in modality and task, but they converge on the same principle: explicit scene interfaces allow the model to reason over geometry, objects, relations, and transitions without forcing all structure into free-form text or an end-to-end latent.

6. Situated meaning and non-visual scene interfaces

Scene Abstraction extends SSI into lexical semantics. Its central claim is that a word’s meaning in context can be represented through the interpretive scene in which it participates, rather than through a dictionary definition or a dense embedding alone (Cho et al., 21 May 2026). Each usage instance consists of a Contextual Scene and an Expression Profile. The Contextual Scene contains Events, Entities, and Setting; the Expression Profile contains Engaged Events, Generalizable Properties, and Evoked Emotions. For the sentence “The man sat alone at the kitchen table, drinking whiskey late at night,” the paper’s structured output includes events such as PersonX sits at ObjectY and PersonX drinks ObjectZ, entities such as PersonX (the man): solitary, melancholic and ObjectZ (whiskey): alcoholic, comforting, a setting of Kitchen; late at night; somber and reflective, and a whiskey-centered profile including engaged events, generalizable properties, and evoked emotions (Cho et al., 21 May 2026).

The scene abstractions are produced by few-shot prompting of gpt-4o-mini with $O=\{o_1,o_2,\dots,o_n\}$3 examples. The prompting procedure is governed by four principles: Generalization, Detail omission, Interpretability, and Context sensitivity. The representation is evaluated on COCA-Scenes, a dataset of 520 sentences built from 26 keywords, 4 scene types per keyword, and 5 sentences per scene type, all drawn from the fiction genre of COCA. The odd-scene-out experiment reports human accuracy 82.37%, individual accuracy 73.08%–94.23%, and mean Gwet’s AC1 of 0.761, interpreted as substantial agreement. Using SentenceBERT embeddings, text only scores 0.575, text + Event + Property + Emotion scores 0.693, and scene only scores 0.627; among individual components, Properties score 0.661, Events 0.609, and Emotions 0.562. In a second experiment, Scene Profiles are preferred to an ATOMIC-based alternative in 86.4% of 1,026 valid evaluations, with 91.4% preference for Engaged Events, 89.2% for Generalizable Properties, and 77.1% for Evoked Emotions (Cho et al., 21 May 2026).

This work suggests that SSI is not restricted to physical perception. A scene interface can also be a structured description of the situational frame through which meaning is interpreted. The paper is explicit, however, that these profiles are inferential rather than objective facts, that they may inherit LLM bias and hallucination, and that the evaluation is mainly on English-language fiction.

7. Recurring advantages, limitations, and research directions

Across domains, the literature repeatedly attributes three benefits to SSI. First, it reduces ambiguity by exposing scene logic directly. SceneCraft argues that SSI reduces linguistic ambiguity and gives fine-grained control over object-level, spatial, and relational edits (Phan et al., 15 Jun 2026). SSR argues that efficient feature alignment and structured scene reasoning are the cornerstones of authentic spatial intelligence (Zhang et al., 28 Feb 2026). Scene Abstraction shows that structured scene profiles align more closely with human interpretation of words in context than ATOMIC-based alternatives (Cho et al., 21 May 2026).

Second, SSI improves data efficiency and computational efficiency by relocating structured operations to the representation layer. SSI-Policy explicitly ties its low-data gains to decoupling perception from control, allowing perception to learn from action-free video while the control policy learns from few demonstrations (Wang et al., 25 Jun 2026). The Cypher-based 3DSG interface shows that structured querying scales significantly better than context serialization on large graphs while also substantially reducing token count (Ray et al., 18 Oct 2025). The Scene Language similarly separates structure, semantics, and identity so that editing can target the relevant component without rewriting the entire scene representation (Zhang et al., 2024).

Third, SSI supports longer-horizon reasoning by making state and transitions explicit. RoBoSR frames this as causally constrained state-space reasoning with subtask dependency enforcement and coherent long-horizon task planning (Hu et al., 23 Jun 2026). SSR’s incremental LocalCogMap construction plays a related role in spatial reasoning: each new object is anchored to two existing objects, reducing ambiguity and preserving global connectivity (Zhang et al., 28 Feb 2026).

The limitations are equally recurrent. Several systems depend on upstream parsing quality. SceneCraft notes that if Detic or Grounding DINO misses an object, the user cannot easily edit it (Phan et al., 15 Jun 2026). The Cypher interface notes that hierarchical reasoning remains hard even when the graph is queryable (Ray et al., 18 Oct 2025). Scene Abstraction emphasizes the subjectivity of affective interpretation and the risk of LLM hallucination (Cho et al., 21 May 2026). SSR avoids dense $O=\{o_1,o_2,\dots,o_n\}$4 triplet enumeration through incremental generation, which indicates that structured representations themselves can become computationally difficult if not carefully factorized (Zhang et al., 28 Feb 2026).

Taken together, these works suggest that the central research question is not whether structure helps, but which kind of structure is best matched to which downstream task. Graph triplets, executable programs, query languages, object-centric state graphs, motion-conditioned tensors, and interpretive semantic schemas each instantiate SSI, but each does so under different assumptions about compositionality, geometry, editability, and control. The present literature therefore treats SSI less as a fixed standard than as a general design principle: make scene structure explicit, make it inspectable, and let downstream models reason through it rather than around it.