SceneLoom: Dual Systems for Scene Understanding

• SceneLoom is a dual-usage framework: one system coordinates data storytelling by aligning visual, semantic, and data dimensions, while the other continuously updates shared 3D scene graphs in dynamic environments.
• It leverages multimodal models like GPT-4 and vision-language encoders to synchronize charts, images, and scene components through structured mapping, scoring, and interactive refinement.
• The data visualization variant supports creative design and animation, whereas the 3D scene graph updater emphasizes real-time multimodal fusion, with performance closely linked to detection accuracy.

SceneLoom is the name of two distinct systems in recent arXiv literature. In one usage, it denotes a Vision-LLM (VLM)-powered system for coordinating data visualization with real-world imagery in data-driven storytelling, where the system aligns visual, semantic, and data dimensions on the basis of narrative intent (Gao et al., 22 Jul 2025). In another usage, it denotes a hypothetical host system into which the MM-3DSG Updater (“Multi-Modal 3D Scene Graph Updater for Shared and Dynamic Environments”) is embedded so that a shared 3D scene graph can be updated continuously in dynamic environments through multimodal change detection, fusion, and graph primitives (Olivastri et al., 2024). Taken together, these usages place the name at the intersection of scene understanding, multimodal reasoning, and structured representation, but they refer to different application domains and technical pipelines.

1. Dual usage and definitional scope

The name SceneLoom appears in two separate research contexts.

Usage of SceneLoom	Domain	Core function
"SceneLoom: Communicating Data with Scene Context"	Data-driven storytelling	Coordinates data visualization with real-world imagery
Hypothetical SceneLoom with MM-3DSG Updater	Shared and dynamic environments	Maintains a 3D scene graph through multimodal updates

In the data-storytelling system, SceneLoom addresses the problem that data visualizations and contextual images are often presented side by side but remain separated, which limits their combined narrative expressiveness and engagement (Gao et al., 22 Jul 2025). In the dynamic-environment formulation, SceneLoom is the surrounding system within which the MM-3DSG Updater monitors human input, robot actions, robot perception, and time-based priors, then applies Add, Remove, and Move operations to a 3D scene graph (Olivastri et al., 2024).

A plausible implication is that the shared label reflects a common emphasis on scene context rather than a single continuous software lineage. The two systems differ in objectives, input modalities, and outputs: one produces image-driven design alternatives for expressive data communication, whereas the other produces real-time graph updates that keep a semantic map synchronized with a changing physical environment.

2. SceneLoom for communicating data with scene context

"SceneLoom: Communicating Data with Scene Context" is situated in data journalism and data videos, where data visualizations are often presented alongside real-world imagery (Gao et al., 22 Jul 2025). The paper identifies several challenges in combining these materials: a semantic gap between abstract attributes and concrete entities, perceptual competition between overlaid charts and scene elements, lack of fine-grained alignment in prior “Infomage” or text-to-image–based approaches, and a creative ideation bottleneck.

The system takes three inputs: a CSV-formatted data table, narrative text, and real-world image(s) in PNG or JPG format. Its five-stage pipeline comprises input, preparation, visual perception, reasoning and design mapping, and output. During preparation, GPT-4o performs narrative-intent parsing to identify data-related entities, actions such as “emphasis” and “enter,” and salient facts. GPT-4o also proposes candidate chart types—bar, line, pie, and map—instantiates them with D3.js templates, and renders SVG and PNG. On the image side, Semantic-SAM segments coarse elements such as “pine tree” and “sky,” while HED and M-LSD extract line and edge candidates for planes and lines.

Visual perception is represented with explicit specifications. For charts, GPT-4o receives SVG and PNG together with a declarative schema that enumerates chart type and layout variant, spatial substrate mappings, graphical elements such as bars, circles, and lines, and high-level visual insights including “increasing trend” and “peak at 2020.” For images, elements are specified by granularity, geometric type, bounding boxes, contour shape descriptors, and semantic role. This decomposition is central to the system’s attempt to coordinate scene components with chart components at multiple representational levels.

The output is a ranked set of design alternatives, accompanied by automated evaluation metrics, design cards for selection, optional image editing through inpainting, an interactive canvas editor, and final animation export. The system therefore combines automatic generation with a mixed-initiative selection-and-refinement workflow rather than relying on a single one-shot composition.

3. Coordination design space and mapping model

The conceptual basis of SceneLoom in the storytelling setting comes from a formative study consisting of a corpus analysis of 54 data-video cases and two expert interviews (Gao et al., 22 Jul 2025). The corpus analysis coded six analytical dimensions: visualization components, scene components, compositional layout, visual inspiration, semantic inspiration, and narrative intent. From this, the authors distilled a design space organized along two axes: visual alignment and semantic coherence.

Visual alignment is described across several mappings: point to axes/origin, line to baselines/trends, plane to canvas regions, and grouped elements to series shapes. Semantic coherence is framed as data-content to entity-objects and data-content to scenario/context. Figure 1 in the paper further organizes these relationships at three levels: spatial substrate, graphical elements, and graphical properties. The result is a structured account of how scene structure can become a substrate for data encoding rather than merely a background image.

The system’s coordination logic follows four design considerations: spatial organization, shape similarity, layout consistency, and semantic binding. Spatial organization concerns which scene elements serve as chart axes or canvas, and how grouped or individual scene items correspond to data points or series. Shape similarity matches geometric primitives such as circles with pie slices or dot marks, lines with trend or axis lines, and organic shapes with area-chart fills. Layout consistency preserves relative positions and scene symmetries. Semantic binding may be direct, metaphorical, or contextual.

The reasoning layer uses Chain-of-Thought style prompts to guide GPT-4o through these four considerations. The model returns mapping proposals, data-level must adjust, view-level transform operations, and tool-invocation code such as “move bar-3 to x=150, rotate 10°.” This means the system does not merely score correspondences; it also operationalizes them into concrete editing actions over chart and image assets.

4. Feature extraction, scoring, and alternative generation

SceneLoom represents visual and semantic features explicitly. Spatial layout is encoded with normalized positions $p=(x/\mathrm{width}, y/\mathrm{height})$ from segmentation masks. Object shapes are derived from HED edges and encoded as shape descriptor vectors $e_{\mathrm{shape}}$ via a small CNN. Scene primitives include line orientation $\theta$, plane area $A$, grouping centroid, and inter-object distances. Semantic features include an entity embedding $e_{\mathrm{sem}}$ from a vision-language encoder such as CLIP-VN, a scenario embedding from GPT-4o summarization of scene context, and a data content embedding $e_{\mathrm{data}}$ from LLM-extracted facts (Gao et al., 22 Jul 2025).

Similarity computation is typically cosine similarity,

$\mathrm{sim}(u,v)=\frac{u\cdot v}{\|u\|\|v\|},$

with all feature vectors $\ell_2$-normalized prior to similarity computation. For each scene element $I_i$ and chart element $V_j$, SceneLoom computes four partial scores:

$e_{\mathrm{shape}}$0

$e_{\mathrm{shape}}$1

$e_{\mathrm{shape}}$2

and

$e_{\mathrm{shape}}$3

where grouping alignment yields 1 and misalignment yields 0. The overall mapping score is

$e_{\mathrm{shape}}$4

with default $e_{\mathrm{shape}}$5.

Candidate mappings are generated by GPT-4o in CoT mode as several high-level strategies; one example given is “map the two Christmas trees to two pie slices; drop the ‘no-tree’ category.” Each strategy is translated into code through a tool-invocation schema using D3.js, SVG transforms, and image edits. The system executes all $e_{\mathrm{shape}}$6 proposals in parallel to produce $e_{\mathrm{shape}}$7 rendered composites, after which an automated evaluator ranks them by data accuracy and visual saliency. The user then receives design cards containing a thumbnail, the mapping description, and quantitative scores.

This architecture combines deterministic feature-based scoring with LLM-mediated proposal generation. A plausible implication is that the scoring layer constrains the proposal space, while the language-model layer expands it creatively within the bounds of the encoded design considerations.

5. Interaction model, animation, evaluation, and limitations

After alternative generation, SceneLoom provides an interactive canvas editor that supports direct manipulation of SVG layers through drag, rotate, scale, color-pick, and text-edit operations (Gao et al., 22 Jul 2025). Internally, element transforms are stored as $e_{\mathrm{shape}}$8 and fed back into the LLM to re-evaluate readability and attention. For animation generation, narrative-intent actions such as enter and emphasis drive a frame-by-frame plan. Each element is assigned keyframes, and Anime.js provides smooth tweening with easing functions such as “easeOutQuad.” A timeline UI supports preview and timing adjustment.

The reported evaluation includes a user study with 10 participants (6F/4M), ages 20–35, with mixed backgrounds including data analysts, designers, and journalists. Expertise self-ratings on a 1–5 scale were 3.8±1.0 for data visualization, 3.4±0.8 for visual design, and 3.4±1.2 for video editing. In the procedure, each participant chose 2 of 10 pre-provided datasets together with 3–4 related images and completed a 30-minute creative task in SceneLoom, followed by an exit questionnaire and semi-structured interview.

Quantitatively, the study reports 5-point Likert means and standard deviations for multiple criteria: System Design Usability $e_{\mathrm{shape}}$9; User Experience $\theta$0; Creativity Support $\theta$1; Final Presentation Quality $\theta$2; Overall Design Experience $\theta$3; and Willingness to Recommend/Reuse $\theta$4. Qualitatively, design-card externalizations helped users break fixation and discover non-obvious mappings, and narrative-intent extraction helped keep designs focused on story goals. Failure cases arose when images lacked clear semantic cues or were overly complex, in which case participants either switched images or used fallback overlay. Latency of approximately 77 seconds per mapping was considered acceptable but also identified as a possible bottleneck.

The paper also states several limitations: dependence on clear visual-semantic cues in input images, restriction to basic chart types including bar, line, pie, and area, LLM/VLM latency in the range of 10–120 seconds, and occasional hallucinated tool calls or unsupported operations in GPT-4o prompts. Proposed future directions include full video input with keyframe extraction and temporal consistency mapping, generative-model subroutines for glyphs and scene edits, extension to interactive scrollytelling and AR/VR environments, fine-tuning or retrieval-augmented prompting, and user preference models for mixed-initiative co-creation.

6. SceneLoom as a host for multimodal 3D scene-graph updating

A different use of the name SceneLoom appears in the description of how the MM-3DSG Updater can be embedded into a hypothetical system called SceneLoom (Olivastri et al., 2024). In that formulation, SceneLoom is not a data-storytelling platform but a system for maintaining a shared, dynamic 3D scene graph in real time. The motivating problem is that 3D Scene Graphs capture both metric and semantic information, yet often assume a static world, whereas homes and offices are dynamic and even small changes can affect task performance.

The update pipeline has three high-level stages. The first is change detection per modality. A Human-Input Module listens for natural-language notifications via an LLM and extracts a candidate operation tuple

$\theta$5

A Robot-Action Module intercepts pick/place commands and uses known task semantics to emit the same tuple. A Robot-Perception Module queries the current 3D scene graph $\theta$6 to obtain the set of expected visible objects $\theta$7 from the current camera pose $\theta$8, runs an RGB-D detector/segmenter to obtain

$\theta$9

and performs semantic and geometric association. Unmatched visible objects become candidates for removal, unmatched detections become candidates for addition, and semantically matched objects with large pose change become move candidates. A Time-Decay Module computes a decrease in “staticness” over $A$0 using a logistic decay prior and emits a “possible change” flag when $A$1.

The second stage is multimodal fusion. It takes up to four event streams,

$A$2

assigns each a confidence or confidence weight $A$3, and combines them into a unified update command $A$4 using either a weighted-sum rule or a Bayesian rule. The third stage is the Scene Graph Updater, which parses $A$5 into one of three primitives—Add, Remove, or Move—and applies the primitive atomically on $A$6 to produce $A$7.

At each timestep, the system senses human chatter, observes robot pick/place actions, acquires RGB-D imagery, and computes $A$8 for dynamic nodes. For each modality $A$9, it extracts zero or more candidate operations

$e_{\mathrm{sem}}$0

These are fused into a consolidated list $e_{\mathrm{sem}}$1 by merging duplicates and combining estimated poses and bounding boxes. Each operation then invokes one of the graph primitives: Add$e_{\mathrm{sem}}$2, Remove$e_{\mathrm{sem}}$3, or Move$e_{\mathrm{sem}}$4, and the updated graph $e_{\mathrm{sem}}$5 is returned.

7. Graph formalism, training regime, and reported behavior in dynamic environments

The MM-3DSG formulation embedded in SceneLoom specifies explicit mathematical rules for change detection, fusion, and graph consistency (Olivastri et al., 2024). Semantic consistency between an expected object $e_{\mathrm{sem}}$6 and observed object $e_{\mathrm{sem}}$7 is

$e_{\mathrm{sem}}$8

while geometric consistency is based on

$e_{\mathrm{sem}}$9

and

$e_{\mathrm{data}}$0

The combined per-pair score, treated as a loss for data association, is

$e_{\mathrm{data}}$1

where $e_{\mathrm{data}}$2 balances semantic and geometric terms.

For fusion, if each modality reports a Bernoulli event that object $e_{\mathrm{data}}$3 has changed with confidence $e_{\mathrm{data}}$4, a weighted-sum rule gives

$e_{\mathrm{data}}$5

with change declared if $e_{\mathrm{data}}$6. Under an independence assumption, a Bayesian fusion variant is

$e_{\mathrm{data}}$7

The scene graph itself is defined as $e_{\mathrm{data}}$8 with object nodes $e_{\mathrm{data}}$9 and room nodes $\mathrm{sim}(u,v)=\frac{u\cdot v}{\|u\|\|v\|},$0. Object node attributes are $\mathrm{sim}(u,v)=\frac{u\cdot v}{\|u\|\|v\|},$1, while room nodes carry $\mathrm{sim}(u,v)=\frac{u\cdot v}{\|u\|\|v\|},$2. Edges include inter-layer “belongs_to” relations from room to object, and optionally intra-layer “adjacent_to” relations from room to room. Spatial indexing may be implemented with a k-d tree or octree over object centroids and adjacency lists for traversal. Consistency constraints require no duplicate object nodes with the same label in the same room, and the graph must remain two-layer with no cycles.

The training regime is intentionally limited. There is no end-to-end training of the 3DSG updater itself. Instead, GPT-4 is prompted to extract operation tuples and estimate $\mathrm{sim}(u,v)=\frac{u\cdot v}{\|u\|\|v\|},$3 for each semantic class; YOLOv8 and the Segment-Anything model are used in pretrained form; decay rates $\mathrm{sim}(u,v)=\frac{u\cdot v}{\|u\|\|v\|},$4 are manually tuned on a small “living room” validation set to match human priors, with values in $\mathrm{sim}(u,v)=\frac{u\cdot v}{\|u\|\|v\|},$5; and no additional gradient-based optimization is performed on the graph updater.

Reported performance is summarized as success rate per update type averaged over 10 simulated runs:

$\mathrm{sim}(u,v)=\frac{u\cdot v}{\|u\|\|v\|},$6

The definition is

$\mathrm{sim}(u,v)=\frac{u\cdot v}{\|u\|\|v\|},$7

Qualitative observations indicate that additions such as a new book on a bed were always detected by perception and fused with human input, while removals and moves of small objects such as a TV remote failed when YOLOv8 missed the object. Temporal priors successfully flagged long-unseen objects for active re-inspection. This suggests that, in this usage of SceneLoom, performance is tightly coupled to detector recall and to the reliability of multimodal corroboration rather than to an end-to-end learned updater.