Diorama-to-Virtual Pipeline Workflow

• Diorama-to-Virtual Pipeline is a design pattern that transforms bounded physical or digital scenes into interactive virtual environments using scans, images, or textual prompts.
• The pipeline typically follows three stages—ingestion, interpretation, and deployment—ensuring accurate geometric registration and efficient engine integration.
• It leverages multimodal techniques such as point cloud alignment, CAD retrieval, and scene graph inference, yielding measurable performance and fidelity in AR/VR applications.

Searching arXiv for the cited works to ground the article in current literature. Diorama-to-Virtual Pipeline denotes a family of computational and production workflows that transform a bounded scene representation—physical dioramas, pre-existing 3D scenes, single images, everyday photographs, staged textual descriptions, or selected sub-volumes—into operational virtual content for augmented reality, virtual reality, mixed reality, simulation, or structured digital-twin representations. Across the recent literature, these pipelines share a common abstraction strategy: they derive an intermediate representation of scene geometry, semantics, or relations; impose constraints appropriate to the target medium; and export a virtual artifact that is inspectable, interactable, or deployable in an engine such as Unity or Unreal. The term therefore spans several technically distinct paradigms, including constrained point-cloud registration for AR scene adaptation (Caetano et al., 2024), scan-based photorealistic VR reconstruction (Bilau et al., 25 Mar 2026), zero-shot CAD-based indoor scene modeling from a single RGB image (Wu et al., 2024), text-to-motion-to-IMU synthesis (Leng et al., 2023), mobile-photogrammetric XR attraction production (Wang et al., 12 Aug 2025), diorama-style focus-and-context interaction in medical VR (Hellum et al., 2022), graph induction from RGB-depth infrastructure imagery (Diessner et al., 8 Dec 2025), and dynamic mixed-reality diorama generation from personal photos (Ihara et al., 8 Apr 2026).

1. Conceptual scope and representational regimes

The literature uses the underlying idea of a diorama in multiple ways. In some works, the source is an actual physical miniature or built environment scanned into a real-time engine. The reproducible Reality-to-VR workflow documents capture of an as-built kitchen with Terrestrial Laser Scanning (TLS), followed by point cloud processing, manual retopology, and Unreal Engine integration; the same four stages are explicitly generalized to a physical diorama, where TLS captures accurate miniature geometry and manual retopology yields efficient meshes for VR deployment (Bilau et al., 25 Mar 2026). The XR narrative attraction "Promisedland" similarly uses hand-crafted scale models, 3D scanning, and Unreal integration as a low-cost, high-fidelity production workflow (Wang et al., 12 Aug 2025).

Other works use “diorama” more abstractly. ARfy treats any pre-existing Unity 3D scene as a virtual scene abstraction and converts it into an AR-ready experience by sampling it into a point cloud $P_v$ and aligning that cloud to an unknown physical environment point cloud $P_e$ at runtime (Caetano et al., 2024). Diorama, in contrast, reconstructs a structured indoor scene from a single monocular RGB image by combining architecture reconstruction, CAD retrieval, 9-DoF pose estimation, and differentiable scene layout optimization, yielding a compositional CAD scene that can be exported to graphics engines (Wu et al., 2024). MemoryDiorama starts from a small set of personal photos and synthesizes a dynamic 3D mixed-reality diorama through multimodal scene analysis, 3D object generation, lighting inference, geographic grounding, and animated spatial composition (Ihara et al., 8 Apr 2026).

A further expansion appears in modality-transfer pipelines. One work describes a text-driven process that begins from short staged activity descriptions, generates 3D human motion with T2M-GPT, converts motion to virtual IMU streams through inverse kinematics and IMUSim, and uses the result for human activity recognition training (Leng et al., 2023). Another pipeline infers a graph representation of hydraulic infrastructure from RGB images and depth data by detecting components, projecting them into 3D, estimating endpoints, and constructing a rule-constrained relational graph $G=(V,E)$ (Diessner et al., 8 Dec 2025). In medical VR, Maserama does not reconstruct an external world but dynamically “scoops” a spherical sub-volume of a complex anatomical scene into a magnified handheld diorama for efficient selection and annotation (Hellum et al., 2022).

These variants suggest that “Diorama-to-Virtual Pipeline” is best understood not as a single algorithm but as a design pattern for converting bounded scene evidence into virtual structure, geometry, behavior, or interaction.

2. Core pipeline architectures

Despite heterogeneity in source modality and target artifact, the surveyed systems exhibit a recurring staged architecture. The first stage is ingestion and abstraction of the source scene. In ARfy, Unity meshes are sampled into a virtual scene point cloud $P_v$ using either surface sampling or support-point sampling based on the lowest XZ face of an object’s axis-aligned bounding box; Unity layers determine inclusion and sampling mode (Caetano et al., 2024). In the scan-based VR workflow, raw TLS captures are registered in Faro SCENE, cleaned, exported as .e57, validated in Autodesk ReCap Pro, then retopologized in SketchUp Pro (Bilau et al., 25 Mar 2026). In the photogrammetric XR attraction workflow, RealityScan produces a textured mesh from 150–300 photographs with 60–80% overlap, after which Blender is used for cleanup, decimation, UV handling, and LOD generation (Wang et al., 12 Aug 2025).

The second stage is structural interpretation. Diorama decomposes interpretation into open-world perception and CAD-based scene modeling: OWLv2 and SAM provide instance masks, Metric3D v2 supplies metric depth and normals, GPT-4o generates a support scene graph, DuoDuoCLIP retrieves candidate CAD assets, and DINOv2-based correspondences seed pose estimation (Wu et al., 2024). The infrastructure graph pipeline uses YOLOv8 pose for pumps, tanks, and valves; YOLOv8 instance segmentation for pipes; pinhole back-projection into world coordinates; DBSCAN and statistical outlier removal; then heuristic endpoint estimation and graph construction (Diessner et al., 8 Dec 2025). MemoryDiorama extracts EXIF metadata, segments scene elements with SAM 3, produces a structured JSON over five cue layers with Gemini 3, then uses top-down map annotation and OpenCV thresholding to derive positions, areas, and routes for virtual placement (Ihara et al., 8 Apr 2026).

The third stage is optimization or deployment into an executable virtual form. ARfy solves a constrained registration problem at runtime to compute a single global similarity transform $T=(R_y,t,s)$ aligning the full scene to the environment (Caetano et al., 2024). The single-image Diorama system performs stage-wise semantic-aware layout optimization over orientation, placement, space, and refinement stages, ultimately exporting instantiated CADs and architectural meshes (Wu et al., 2024). The scan-based VR pipeline imports retopologized assets into Unreal Engine 5.3.2 via Datasmith, preserving semantic hierarchy and metadata while configuring Lumen global illumination, OpenXR interaction, and collision primitives (Bilau et al., 25 Mar 2026). MemoryDiorama composes geo-referenced terrain, image-to-3D assets, lighting, routes, and particles in Unity and deploys to Meta Quest 3 (Ihara et al., 8 Apr 2026).

This repeated decomposition—abstraction, interpretation, constrained assembly—marks the central systems-level regularity across otherwise disparate Diorama-to-Virtual pipelines.

3. Geometric registration, reconstruction, and scene optimization

One major class of Diorama-to-Virtual pipelines is fundamentally geometric. ARfy formalizes adaptive AR placement as similarity registration between virtual and physical point clouds. With pose variables $x=(t,R_y,s)$, nearest-neighbor correspondences $c(i)$, and yaw-only rotation, it minimizes mean squared nearest-neighbor distance

$$J(x) = \frac{1}{N_v} \sum_{i=1}^{N_v} \left\| R_y \left( s \, p_i \right) + t - q_{c(i)} \right\|_2^2$$

subject to yaw-only rotation and positive uniform scale (Caetano et al., 2024). The mapped scene vertex is then

$x_{AR} = R_y \left( s \, x_{scene} \right) + t,$

or, when a diorama-specific normalization factor is used,

$$x_{AR} = R_y \left( s \, s_{diorama} \, x_{diorama} \right) + t$$

(Caetano et al., 2024). The baseline optimization strategy is ICP, initialized from the scene’s origin and the current device world origin.

The single-image Diorama system solves a different geometric problem: reconstruction of a structured indoor scene from one RGB observation. It back-projects metric depth with

$P_e$0

fits planar architecture through normals clustering, RANSAC plane fitting, and weak Manhattan-world refinement, and estimates object pose by combining multiview semantic correspondences with an Umeyama-style alignment objective

$P_e$1

(Wu et al., 2024). Its global scene layout optimization then incorporates semantic alignment, support placement, spacing consistency, collision penalties, and containment into a stage-wise differentiable objective (Wu et al., 2024).

Scan-based pipelines emphasize acquisition fidelity before real-time reduction. The reproducible Reality-to-VR workflow uses FARO Focus S70 TLS with systematic ranging error of $P_e$2 mm at 10 m and target-based registration reaching a mean point error of 1.2 mm in Faro SCENE (Bilau et al., 25 Mar 2026). The resulting dense cloud is not imported directly into UE5; instead, manual retopology is used specifically to avoid stochastic high-polygon artifacts incompatible with real-time engines (Bilau et al., 25 Mar 2026). Promisedland uses mobile photogrammetry rather than TLS, but still formalizes physical-to-virtual alignment with a similarity transform

$P_e$3

and recommends Procrustes estimation with optional ICP refinement (Wang et al., 12 Aug 2025).

These methods share a consistent logic: source evidence is first expressed in a geometry-rich space—point clouds, planes, meshes, trajectories—then reduced under explicit structural assumptions such as parallel ground planes, Manhattan layouts, support hierarchies, or rigid similarity transforms.

4. Semantic grounding, multimodal inference, and relational structure

A second major class of techniques augments geometry with semantic or relational priors. Diorama is explicitly zero-shot and open-world: it avoids end-to-end training by composing pretrained foundation models for detection, segmentation, metric geometry, retrieval, language-mediated reasoning, pose estimation, and optimization (Wu et al., 2024). GPT-4o, prompted with set-of-mark overlays, produces a scene graph $P_e$4 whose edges encode support relations such as “placed on” and “mounted on,” and this graph constrains downstream layout optimization (Wu et al., 2024).

MemoryDiorama also uses a multimodal LLM as a scene-analysis engine, but its role is different. Gemini 3 produces a deterministic JSON spanning objects, humans, geography, lighting, and particles, and these semantic layers directly determine asset generation, placement annotations, animation verbs, environmental effects, and mixed-reality composition (Ihara et al., 8 Apr 2026). The result is not merely a reconstruction of visible structure but an augmented cue environment designed to enrich autobiographical recall.

The infrastructure graph pipeline introduces semantics through object class detection and user-defined heuristics. Nodes correspond to pipes, pumps, tanks, valves, PipeCrossing, or Reducer/Expander, while edges represent physical connections weighted by Euclidean endpoint distance (Diessner et al., 8 Dec 2025). Rule-based refinement then deletes violating edges until constraints are satisfied: pumps and valves have at most two connections, tanks at most one, pipes at most three, neighbors of the same non-pipe object cannot be connected, isolated nodes are disallowed, cycles are disallowed, and certain topologies trigger relabeling to Reducer/Expander or PipeCrossing (Diessner et al., 8 Dec 2025). The authors emphasize that this transparent rule framework is a virtue for high-stakes infrastructure settings.

Even the text-to-IMU pipeline depends on semantic structure. ChatGPT is constrained to produce prompts of 15 words or less, with a single actor and minimal environment; these prompts then condition T2M-GPT to synthesize 22-joint motion sequences, which are later converted to virtual sensor streams (Leng et al., 2023). Here the “diorama” is a succinct semantic scene specification rather than a geometric miniature.

A plausible implication is that Diorama-to-Virtual research is increasingly organized around hybrid representations: geometry alone is often sufficient for alignment, but semantics are needed for support reasoning, animation selection, graph topology, or user-relevant cue design.

5. Engine integration, interaction, and runtime constraints

The conversion from reconstructed or inferred scene structure to a usable virtual system depends strongly on engine integration and interaction design. ARfy exposes both a Unity plugin and a Python Web API. The Unity side handles scene sampling and runtime application of the computed transform to Unity transforms and anchors, while ARKit or ARCore supply the physical point cloud and world-coordinate frame at runtime (Caetano et al., 2024). The system is designed for mobile and head-mounted AR and can optionally expose error heatmaps for debugging (Caetano et al., 2024).

The scan-based Reality-to-VR workflow uses Unreal Engine 5.3.2, Datasmith, Lumen dynamic global illumination, OpenXR, and collision capsules. Datasmith is preferred over FBX because it preserves semantic hierarchy and metadata, which in turn supports efficient swapping of design variables without losing coordinate references (Bilau et al., 25 Mar 2026). Promisedland instead uses Unreal Engine 4.27 and adds spline-driven movement, Sequencer-authored narrative choreography, Niagara particle systems, and optimization strategies tailored to Meta Quest deployment, including aggressive LODs, instancing, and static lighting (Wang et al., 12 Aug 2025).

MemoryDiorama uses Unity 6000.1.15f1 with Cesium for Unity v1.19.0, Google Photorealistic 3D Tiles, and Meta XR All-in-One SDK v81.0.0, deploying to Meta Quest 3 (Ihara et al., 8 Apr 2026). Its interaction style is explicitly worlds-in-miniature: users inspect the mini scene from multiple angles rather than manipulating a dense control interface (Ihara et al., 8 Apr 2026). Maserama in medical VR provides a different interaction paradigm. A spherical selection volume attached to the right controller defines a sub-volume

$P_e$5

which is then materialized as a magnified handheld diorama attached to the left controller for laser-based structure selection (Hellum et al., 2022). The main anatomy remains visible, yielding a focus-and-context configuration rather than a global clipping view.

Runtime constraints vary by platform but are central across papers. The reproducible Reality-to-VR pipeline explicitly targets stable 90 Hz and reports a mean frame rate of 90.4 FPS with average frame time 11.1 ms on an RTX 3080 workstation, with a final scene asset of approximately 450,000 polygons (Bilau et al., 25 Mar 2026). Promisedland recommends a visible triangle budget of roughly 300k–500k per frame on Quest-class devices and typical draw-call targets below 100–150, although the paper itself did not publish frame metrics (Wang et al., 12 Aug 2025). ARfy does not provide explicit runtime benchmarks and notes that performance depends on point counts and device capabilities (Caetano et al., 2024).

6. Evaluation practices, empirical results, and recurring limitations

Evaluation protocols differ according to target modality, but several patterns recur: quantitative fidelity measures, real-time performance constraints, and task-level user validation.

ARfy evaluates adaptive placement on ARKit Scenes by random downsampling the virtual scene cloud to 1,000 points, running constrained ICP for each environment, and computing mean squared placement error. For the Unity Game Kit scene adapted to ARKit Scenes, the best placement is reported for video id 47332911 with error $P_e$6, while the worst is video id 45261190 with $P_e$7; empty rectangular rooms produce lower errors, whereas mirrors and multi-floor geometries challenge registration (Caetano et al., 2024).

The reproducible Reality-to-VR pipeline combines geometric and human-subject validation. Target-based registration in Faro SCENE achieves mean point error 1.2 mm; the final environment maintains 90.4 FPS at 11.1 ms frame time; and a within-subject study with 17 older adults shows no significant increase in SSQ total scores from baseline to either condition, while NASA-TLX workload is significantly lower in Open Shelving than Closed Cabinets with $P_e$8 and effect size $P_e$9 (Bilau et al., 25 Mar 2026). This ties scene fidelity and frame stability directly to ecological validity and cybersickness mitigation.

Diorama reports extensive reconstruction metrics. For 9-DoF alignment on SSDB, the system with estimated depth attains $G=(V,E)$0, $G=(V,E)$1, $G=(V,E)$2, $G=(V,E)$3, collision 7.37, and relation 0.25; full layout optimization reduces collisions from 7.26 to 4.43 and improves structure overall to 0.92 (Wu et al., 2024). PlainRecon with Metric3D v2 reports IoU 58.6, Pixel Error 9.6, Edge Error 18.9, CDb 0.447, and 29 s runtime, succeeding on 344/344 scenes (Wu et al., 2024).

The text-to-IMU pipeline evaluates Macro F1 under leave-one-subject-out cross-validation. Mixed Real+Virtual training improves over Real-only on PAMAP2 from $G=(V,E)$4 to $G=(V,E)$5, on RealWorld from $G=(V,E)$6 to $G=(V,E)$7, and on USC-HAD from $G=(V,E)$8 to $G=(V,E)$9 (Leng et al., 2023). Notably, on RealWorld, Virtual-only exceeds Real-only, which the paper attributes to prompt-driven diversity (Leng et al., 2023).

MemoryDiorama evaluates its mixed-reality output in a within-subject study with 18 participants. Compared with Photo-Only and Static Diorama, MemoryDiorama increases internal details, in-cue details, perceptual details, MCQ visual details, and enjoyment, while NASA-TLX shows no significant workload difference across conditions (Ihara et al., 8 Apr 2026). The system also reports a technical feasibility test over 25 photo sets with average end-to-end time 15.70 minutes, photo analysis success 100%, element generation 92.58%, and placement/route success 75.25% (Ihara et al., 8 Apr 2026).

Limitations are strikingly consistent across the literature. Reflective or specular surfaces cause ghosting in scan-based pipelines (Bilau et al., 25 Mar 2026); mirrors and multi-level spaces degrade registration in ARfy (Caetano et al., 2024); depth noise, heavy occlusions, and LMM hallucinations destabilize single-image scene modeling (Wu et al., 2024); semantic ambiguity in prompts can produce motion-model confusion in text-driven IMU synthesis (Leng et al., 2023); foliage, glossy surfaces, and mobile performance constraints complicate photogrammetric XR deployment (Wang et al., 12 Aug 2025); small-object detection and heuristic endpoint estimation remain weak points in graph induction (Diessner et al., 8 Dec 2025); and MemoryDiorama raises explicit concerns about hallucinated augmentations and false-memory risk (Ihara et al., 8 Apr 2026). This suggests that data ambiguity, not merely algorithmic inefficiency, remains the dominant failure source in current Diorama-to-Virtual systems.

7. Synthesis, applications, and research directions

The surveyed pipelines occupy different application domains—adaptive AR authoring, ecological VR experiments, indoor scene understanding, activity-recognition augmentation, XR storytelling, medical navigation, infrastructure digital twins, and autobiographical memory support—but their convergence is methodologically significant.

First, they increasingly separate source capture from target deployment through explicit intermediate abstractions. In ARfy, both source scene and target environment are reduced to point clouds (Caetano et al., 2024). In scan-based VR and photogrammetric XR, dense captures are reduced to retopologized meshes and structured engine assets (Bilau et al., 25 Mar 2026, Wang et al., 12 Aug 2025). In single-image reconstruction, the abstraction is a CAD scene plus support graph and architectural planes (Wu et al., 2024). In infrastructure modeling, it is a relational graph (Diessner et al., 8 Dec 2025). In MemoryDiorama, it is a five-layer semantic-and-spatial composition schema (Ihara et al., 8 Apr 2026).

Second, these pipelines often trade exact reconstruction for operational utility. ARfy does not model semantics, affordances, collision, or occlusion in its core optimization (Caetano et al., 2024). Diorama prioritizes compositional CAD structure rather than exact material reproduction (Wu et al., 2024). The text-to-IMU pipeline does not seek physically exhaustive motion realism, but enough semantically faithful diversity to improve classifier generalization (Leng et al., 2023). Promisedland explicitly values “material expressiveness” and narrative warmth over metrically exhaustive replication (Wang et al., 12 Aug 2025). This suggests that virtual adequacy is domain-specific: the relevant criterion may be placement plausibility, ecological validity, relational transparency, user comfort, or recall enrichment rather than geometric identity alone.

Third, future directions implied by the literature are highly compatible. ARfy identifies outlier detection and improved optimization as future work (Caetano et al., 2024). Scan-based VR workflows point to semantic separation and rigging for interactive elements (Bilau et al., 25 Mar 2026). Diorama’s results suggest gains from better scene graph reliability, depth robustness, and retrieval precision (Wu et al., 2024). Infrastructure graph generation would benefit from explicit classes for elbows and T-fittings, stronger thin-structure reconstruction, and possibly interpretable GNNs (Diessner et al., 8 Dec 2025). MemoryDiorama points toward more reliable placement, improved human generation, and explicit study of memory accuracy and false-memory formation (Ihara et al., 8 Apr 2026).

A plausible implication is that future Diorama-to-Virtual pipelines will become increasingly multimodal and constraint-aware: geometric capture or inference will remain central, but robust deployment will depend on semantic reasoning, uncertainty handling, and application-specific optimization layered on top of that geometry.