Realiz3D: Controllable 3D Generation

• Realiz3D is a research motif encompassing AR-driven image-to-3D pipelines and domain-aware diffusion frameworks for generating controllable 3D content.
• The AR system uses user-guided segmentation, instance detection, and Shap-E–based 3D conversion to transform 2D images into manipulable 3D objects.
• The diffusion framework employs lightweight residual Domain Shifters to balance photorealism with 3D consistency under geometric and material controls.

Searching arXiv for papers directly relevant to “Realiz3D” and closely related usage contexts. Realiz3D is a label used in recent arXiv literature for two technically distinct systems. One is an augmented-reality pipeline for turning a user-captured 2D image into a manipulable 3D object placed in situ, with the stated goal of making 3D content creation accessible to non-experts (Behravan et al., 27 Apr 2025). The other is a domain-aware diffusion fine-tuning framework for producing outputs that are simultaneously photorealistic and 3D-consistent under controls such as geometry, materials, and viewpoint (Sobol et al., 25 Mar 2026). Across these uses, the name denotes not a single standardized architecture but a recurring research concern: how to couple 3D controllability with practical usability, whether in interactive AR authoring or in controllable image generation.

1. Scope of the term and antecedent themes

The name “Realiz3D” appears in two separate arXiv titles, and its meaning is therefore contextual rather than singular. In the 2025 usage, it denotes a human-in-the-loop AR workflow built around object selection, segmentation, and image-to-3D generation. In the 2026 usage, it denotes a lightweight training framework for diffusion models that explicitly separates visual domain from 3D control signals. The former is a deployment system; the latter is a learning strategy for controllable generation (Behravan et al., 27 Apr 2025).

A longer intellectual backdrop can be identified in earlier work on democratizing 3D media production. “Realizacao domestica e escolar de foto e video 3D” argued that stereoscopic image production should not remain tied to cinema, specialized industry, or expensive dedicated equipment, and instead showed how ordinary digital cameras, simple editing procedures, and anaglyph glasses could support domestic and school-scale 3D practice (Lunazzi, 2012). This suggests a persistent line of research in which accessibility, low-cost tooling, and non-expert workflows are treated as central design constraints rather than secondary dissemination concerns.

Within that broader trajectory, Realiz3D marks two different technical responses to the same structural problem. One response focuses on front-end interaction in AR capture and 3D asset generation; the other focuses on the back-end learning problem that arises when 3D controls are available mainly in synthetic data while photorealistic appearance is learned from real imagery.

2. Realiz3D as an AR image-to-3D generation pipeline

In the 2025 system, Realiz3D is presented as a practical AR pipeline that converts a user-captured 2D image into a 3D object that can be manipulated and displayed in augmented reality. The stated motivation is that conventional 3D modeling is slow, skill-intensive, and dependent on professional software, whereas AR applications increasingly require diverse 3D assets on demand. The system therefore wraps OpenAI’s Shap-E in an AR-specific preprocessing and interaction loop so that it can operate under cluttered, real capture conditions rather than only on isolated object images (Behravan et al., 27 Apr 2025).

The interaction model is explicitly user-guided. Through an AR hand menu, the user selects either “capture zone” or “capture objects.” In the zone-based mode, a lasso-like red boundary is drawn around the target region; the drawing ends with a pinch gesture; and after a 3-second delay, the system captures an image containing both the scene and the red selection marking. The captured image is cropped to the selected region, converted to HSV color space, processed to detect the red lasso line, and reduced to a bounding box using OpenCV contour processing. Within that crop, Mask R-CNN performs instance segmentation, and the resulting object crops are labeled with class names, encoded in base64, and sent via an API to the 3D generation backend. Before generation, the system presents a menu of detected objects so that the user can choose which extracted object or objects should be converted into 3D. Only then is the selected crop sent to Shap-E. The resulting 3D model is simplified by vertex reduction, then loaded and rendered back into the AR environment.

This architecture is hybrid by construction. It does not rely solely on fully automatic vision, because the lasso provides explicit user intent; nor does it rely solely on manual cropping, because Mask R-CNN provides object-level delineation inside the chosen zone. The paper emphasizes that this hybrid design is necessary because direct use of Shap-E on cluttered headset imagery fails in realistic AR conditions. Two failure examples are singled out: a multi-object tabletop scene yields essentially a single cube, and a mug shown on a monitor is reconstructed as something like a laptop, because the screen structure dominates the image.

The implementation details are partially specified. The front end is described as running in an AR environment, likely on Microsoft HoloLens 2, which is explicitly recommended because of its display resolution, sensors, environment mapping, and hand tracking. The backend uses OpenCV, Mask R-CNN, and Shap-E; Linux distributions are mentioned for API development; Windows 10 or higher is recommended for AR environment development; and the system automatically selects GPUs when available for detection. The specific AR software framework is not specified.

The reported stage times make clear that the system is interactive in-session rather than real-time in the strict frame-rate sense.

Stage	Reported time
Image processing for object detection	5.2 s
Image-to-3D conversion	43.2 s
Model simplification	9.1 s
Load and render	10.3 s

These figures sum to an end-to-end delay of about 67.8 seconds, with the dominant bottleneck being the Shap-E generation stage. The paper also reports Average GPU utilization: 61% and GPU memory consumption: 6.8 GB.

3. Usability profile, failure modes, and technical limits of the AR system

The AR Realiz3D paper evaluates the system through both runtime measurements and a usability study with 35 participants, aged 18 to 40, with 63% male and 37% female participation and a user population mostly characterized by rare or no usage of AR/VR and only slight or none familiarity with 3D design (Behravan et al., 27 Apr 2025).

The primary quantitative usability result is an overall System Usability Scale score of 69.64. The paper interprets this as slightly above the standard benchmark of 68. It also reports a statistically significant subgroup difference by AR/VR usage frequency: the Rarely/Never group achieved a mean SUS of 64.38, whereas the Regularly/Sometimes/Often group reached 80.71; the variances were 50.32 and 42.17 respectively; the group sizes were 20 and 15; and the reported ANOVA statistics were F = 18.212 and p < 0.001, with the table giving p = 0.000032. The practical implication is not that the system is unusable for novices, but that familiarity with immersive interaction substantially changes perceived usability.

The technical limitations are equally explicit. The paper identifies sensitivity to background complexity, multiple objects in one frame, objects displayed on monitors, and errors induced by lighting and viewing angle. Segmentation failures propagate directly into poor generation because the wrong object may be cropped or labeled. Generated models are judged primarily visually; the paper does not provide geometric accuracy metrics, and it does not specify dynamic occlusion handling between generated objects and the real world. It also notes that generated meshes may be too heavy for practical headset rendering, which is why vertex reduction is a mandatory postprocessing stage rather than an optional optimization.

A recurring misconception is corrected by the paper’s own measurements: despite the phrase “real-time” in the title, the reported timing does not correspond to instantaneous graphics-time synthesis. The contribution is better characterized as user-driven, in-session 3D generation inside an AR workflow. In that sense, the novelty lies less in Shap-E itself than in the system-level recipe: user-guided region selection, instance segmentation, object verification, backend generation, mesh simplification, and AR-native display.

4. Realiz3D as a domain-aware diffusion framework

The 2026 Realiz3D paper addresses a different problem entirely: how to make a diffusion generator both photorealistic and 3D-consistent under controls such as geometry, materials, and viewpoint, when those controls are available mainly from synthetic renders of 3D assets (Sobol et al., 25 Mar 2026). Its central diagnosis is that standard fine-tuning on synthetic 3D renders creates a spurious association between the presence of control signals and synthetic-looking outputs. In the paper’s formulation, the issue is not only catastrophic forgetting, but also a learned shortcut in which “having control” becomes correlated with “looking synthetic.”

The framework’s solution is to explicitly separate visual domain from 3D control. Realiz3D introduces a binary domain co-variate, $$e_{\text{domain}} \in \{e_{\text{real}}, e_{\text{syn}}\}$$, and injects it through lightweight low-rank residual modules called Domain Shifters. For a latent representation $X \in \mathbb{R}^d$, a Domain Shifter applies the update

$$X = X + D(\text{domain}) = X + W_{\text{left}} W_{\text{right}} e_{\text{domain}}.$$

Here, $W_{\text{left}} \in \mathbb{R}^{d \times r}$, $W_{\text{right}} \in \mathbb{R}^{r \times d}$, and the implementation uses $r = 8$. The update is residual and block-level, which reflects the paper’s claim that real and synthetic visual modes already exist in the pretrained model and only require a lightweight latent-space shift.

Training is two-stage. In Stage 1, the diffusion backbone is frozen and only the Domain Shifters are trained, with null control $c = \varnothing$ for both real and synthetic samples. This stage learns visual domain independently of control. In Stage 2, the Domain Shifters are frozen and the backbone is fine-tuned to learn controllability while preserving the earlier disentanglement. The paper keeps the standard diffusion objective. For synthetic samples, the loss is

$$\mathcal{L} = \mathbb{E}_{z,\epsilon \sim q_z,\epsilon \sim \mathcal{N}(0,1),\, t} \left[ \left\| \epsilon - \epsilon_\theta(z_t, t, c, e_{\text{syn}}) \right\|_2^2 \right] \qquad \text{(at stage 1, } c=\varnothing \text{)},$$

and for real samples it is

$$\mathcal{L} = \mathbb{E}_{z,\epsilon \sim \mathcal{N}(0,1),\, t} \left[ \left\| \epsilon - \epsilon_\theta(z_t, t, c=\varnothing, e_{\text{real}}) \right\|_2^2 \right].$$

A further mechanism, called Representation Binding, exploits the paper’s observation that early denoising steps and early layers are more domain-agnostic and more responsible for coarse structure, whereas later steps and later layers dominate fine appearance. This leads to two additional strategies. Layer-Aware Training updates real samples primarily in later blocks while freezing a prefix of earlier blocks; the implementation uses $T_B = 40\%$ of total blocks. Domain Reassignment reassigns some early blocks to synthetic mode while processing real samples, with probability $X \in \mathbb{R}^d$0, in order to improve transfer of synthetic control knowledge into real-domain outputs. At inference, the default setting is $X \in \mathbb{R}^d$1 with non-null control, but the framework also allows partial inference-time domain shifting, using synthetic mode for early blocks or high-noise timesteps. The reported settings are $X \in \mathbb{R}^d$2 of blocks and $X \in \mathbb{R}^d$3.

5. Tasks, training configuration, and empirical performance of domain-aware learning

The 2026 framework is evaluated on two tasks: text-to-multiview generation and multiview texturing from 3D inputs (Sobol et al., 25 Mar 2026). In the first task, the model generates $X \in \mathbb{R}^d$4 images from text plus camera or viewpoint information. In the second, it generates four RGB views conditioned on normal maps and position maps. In both cases, the four views are arranged into a $X \in \mathbb{R}^d$5 grid, with each view at $X \in \mathbb{R}^d$6 resolution, so that self-attention can operate across views and encourage consistency.

The training setup is unusually explicit. The synthetic training corpus contains 120K synthetic 3D assets, each rendered from $X \in \mathbb{R}^d$7 viewpoints with RGB, normal, and position maps. The real-domain dataset is generated from the base text-to-image model itself, using the same textual descriptions as prompts and producing matched-size sets of realistic images with white background. The system is trained for 10 epochs on 64 NVIDIA H100 GPUs, with batch size 8 and learning rate $X \in \mathbb{R}^d$8. The Domain Shifters are trained separately for 3 epochs. Sampling uses DDIM with 50 steps, and classifier-free guidance is applied only to the text condition.

The central empirical claim is that Realiz3D improves realism substantially while preserving most of the controllability of synthetic-only baselines. In multiview texturing, the synthetic-only baseline reports PSNR 25.76, SSIM 0.9269, LPIPS 0.0831, FID$X \in \mathbb{R}^d$9 168.21, KID$$X = X + D(\text{domain}) = X + W_{\text{left}} W_{\text{right}} e_{\text{domain}}.$$0 0.0240, FID$$X = X + D(\text{domain}) = X + W_{\text{left}} W_{\text{right}} e_{\text{domain}}.$$1 218.29, KID$$X = X + D(\text{domain}) = X + W_{\text{left}} W_{\text{right}} e_{\text{domain}}.$$2 0.0431, and CLIP 0.2628. Realiz3D reports PSNR 24.78, SSIM 0.9153, LPIPS 0.0865, FID$$X = X + D(\text{domain}) = X + W_{\text{left}} W_{\text{right}} e_{\text{domain}}.$$3 141.90, KID$$X = X + D(\text{domain}) = X + W_{\text{left}} W_{\text{right}} e_{\text{domain}}.$$4 0.0121, FID$$X = X + D(\text{domain}) = X + W_{\text{left}} W_{\text{right}} e_{\text{domain}}.$$5 200.24, KID$$X = X + D(\text{domain}) = X + W_{\text{left}} W_{\text{right}} e_{\text{domain}}.$$6 0.0291, and CLIP 0.2674. The result is a modest decrease in strict 3D consistency metrics but a large improvement in realism metrics.

The same pattern holds in text-to-multiview generation. The synthetic-only baseline reports PSNR 19.66, SSIM 0.8779, LPIPS 0.0964, FID$$X = X + D(\text{domain}) = X + W_{\text{left}} W_{\text{right}} e_{\text{domain}}.$$7 168.60, KID$$X = X + D(\text{domain}) = X + W_{\text{left}} W_{\text{right}} e_{\text{domain}}.$$8 0.0204, FID$$X = X + D(\text{domain}) = X + W_{\text{left}} W_{\text{right}} e_{\text{domain}}.$$9 215.57, KID$W_{\text{left}} \in \mathbb{R}^{d \times r}$0 0.0363, and CLIP 0.2541. Realiz3D reports PSNR 19.02, SSIM 0.8631, LPIPS 0.1075, FID$W_{\text{left}} \in \mathbb{R}^{d \times r}$1 122.01, KID$W_{\text{left}} \in \mathbb{R}^{d \times r}$2 0.0056, FID$W_{\text{left}} \in \mathbb{R}^{d \times r}$3 196.01, KID$W_{\text{left}} \in \mathbb{R}^{d \times r}$4 0.0171, and CLIP 0.2643. The framework therefore does not eliminate the realism–control tradeoff, but it shifts the Pareto frontier toward more photorealistic outputs than prior adapter, switcher, LoRA, and training-free baselines.

The paper is also explicit about remaining limits. Fine-grained details such as fur, hair, and high-frequency material effects remain harder to keep perfectly consistent across views. The authors also note that the pretrained text-to-image prior carries lighting biases that can induce view-inconsistent illumination, even when geometric or viewpoint controls are followed correctly.

6. Related systems and recurring interpretive cautions

The two Realiz3D works sit inside a broader landscape of “3D realization” systems, but they occupy different positions within it. Mobile reconstruction work such as “Hologram: Realtime Holographic Overlays via LiDAR Augmented Reconstruction” focuses on on-device sensing, metric depth, and live overlay rendering, achieving around 50 FPS on an iPhone 14 Pro in its LiDAR + TrueDepth pipeline (Agrawal, 2024). Desktop display systems such as “A Real-time 3D Desktop Display” convert ordinary 2D camera, video, or screen-region inputs into light-field output for devices such as Looking Glass, reporting 10 Hz and about 100 ms/frame under the tested configuration (Tenze et al., 9 Jun 2025). Low-cost viewing systems such as “Real-time 3D Light-field Viewing with Eye-tracking on Conventional Displays” combine a standard monitor, a webcam, and red-cyan anaglyph glasses, operating at 30 FPS on CPU-only hardware (Pham et al., 22 Aug 2025). Mixed-reality fabrication systems such as “I-nteract 2.0” integrate CSG, single-view mesh reconstruction, workspace-aware resizing, and additive manufacturing (Malik et al., 2020). Scene-level scan conversion systems such as “LiteReality” reconstruct compact, object-centric, graphics-ready indoor scenes from RGB-D scans (Huang et al., 3 Jul 2025). View-conditioned neural object realization in VR is represented by “Deep Billboards,” which uses a billboard for visual interaction and an invisible rough mesh for physics interaction (Kondo et al., 2022).

Against this background, two misconceptions should be avoided. First, Realiz3D is not a single unified platform spanning AR, diffusion training, mobile holography, and fabrication. The exact term names two separate systems with different outputs, assumptions, and evaluation protocols. Second, the word “3D” is not used identically across those systems. In the AR paper, it denotes image-to-3D object generation for AR display after segmentation and user selection (Behravan et al., 27 Apr 2025). In the diffusion paper, it denotes 3D-consistent multiview or texture generation under viewpoint, geometry, and material control, not direct recovery of a full explicit scene or CAD model (Sobol et al., 25 Mar 2026).

Taken together, the literature indicates that Realiz3D is best understood not as a settled technical standard but as a research motif centered on controllable 3D realization under practical constraints. One branch emphasizes capture, object isolation, and novice-facing AR interaction; the other emphasizes synthetic-to-real transfer in controllable generative models. Their common denominator is the attempt to make 3D control operational without surrendering either realism or usability.