Stable Video Materials 3D (SViM3D)

• SViM3D is a unified framework that generates multi-view consistent 3D assets by predicting relightable PBR materials, surface normals, and geometric cues from a single image.
• It leverages latent video diffusion with explicit camera trajectory conditioning to ensure spatial coherence and accurate multi-view synthesis for downstream inverse rendering.
• The approach supports AR/VR, movies, and games by enabling real-time relighting and precise appearance edits, streamlining digital asset creation with physically based rendering.

Stable Video Materials 3D (SViM3D) denotes a class of frameworks that directly address the joint generation of multi-view-consistent appearance and physically based rendering (PBR) materials from a single image, enabling relightable and editable 3D asset synthesis under explicit camera control. SViM3D extends latent video diffusion by predicting not only RGB images but also spatially varying PBR parameters and surface normals for every generated view, thus serving as a unified neural prior for both forward synthesis and inverse rendering pipelines. The addition of camera trajectory conditioning ensures spatial coherence across synthesized orbital videos and facilitates controlled appearance edits, novel view synthesis, and full 3D reconstruction. This approach resolves long-standing challenges in single-image inverse rendering by delivering state-of-the-art outputs and broad applicability in AR/VR, movies, and games (Engelhardt et al., 9 Oct 2025).

1. Model Architecture and Output Structure

SViM3D builds on latent video diffusion architectures, repurposing them for multi-channel output. The backbone comprises a denoising UNet operating in a latent space defined by a pretrained variational autoencoder (VAE). Channel dimensions are extended to predict for each view:

• RGB image (3 channels)
• PBR material parameters (basecolor/albedo, roughness, metallic; total 5 channels)
• Surface normals (3 channels)

For a standard 21-view orbit, the output is a stack of 11-channel images aligned to sequenced camera poses. Condition inputs consist of (i) the source image, and (ii) a camera trajectory, typically as a sequence of (elevation, azimuth) pairs. The denoising process is governed by a simplified EDM loss,

$$\min\, E\left[\left\|\epsilon_\theta(z_t, t, \text{cond}) - \epsilon\right\|^2\right]$$

where $z_t$ denotes the noisy latent, $t$ the timestep, and “cond” concatenates both input image and trajectory. Material maps are processed as images, enabling use of the same latent space for both RGB and PBR channels.

This architecture permits joint prediction of view-consistent materials and geometric cues, which is essential for subsequent relighting and 3D reconstruction tasks.

2. Physically Based Material Parameter Prediction

A central innovation of SViM3D is the direct prediction of spatially-varying PBR maps—albedo, roughness, and metallic—alongside per-view normals. The output channels closely follow the schema of real-time graphics engines (e.g., the ORM “Occlusion-Roughness-Metallic” layout).

Material modeling relies on the Cook–Torrance microfacet BRDF, with predictions parameterized and arranged for compatibility with physically based rendering:

• Albedo (base color): $\mathbf{a} \in \mathbb{R}^3$
• Roughness: $r \in [0,1]$
• Metallic: $m \in [0,1]$
• Surface normal: $\mathbf{n} \in \mathbb{R}^3$, normalized

At rendering time, the network’s outputs are combined with environmental maps using a split-sum approximation for lighting, in accordance with: $$L(\omega_o) \approx [\text{precomputed BRDF lookup}] \times [\text{pre-filtered environment map}]$$ Material edits and relighting are accomplished by direct manipulation of these predicted maps. This approach enables reliable fine-grained control of appearance without further inverse estimation.

3. Camera Trajectory Conditioning and Spatial Consistency

Explicit camera control is enforced by wiring the trajectory (sequence of elevation–azimuth tuples) into the conditioning of the diffusion UNet. Each view is aligned to a specified camera pose, ensuring spatial coherence and accurate multi-view alignment.

This explicit trajectory conditioning separates intrinsic material properties from view-dependent effects, increasing robustness to perspective distortion and enabling high-quality multi-view synthesis suited for downstream optimization (e.g., NeRF or DMTet pipelines). Spatial consistency is further maintained through view-dependent masking: regions of low perspective distortion receive higher weight in loss computation, mitigating multi-view inconsistencies.

4. Relighting and Appearance Editing

Because SViM3D predicts full PBR parameters and normals for each view, the generated output can be relit or re-edited under novel lighting conditions. The physically based rendering module evaluates the Cook–Torrance BRDF for each pixel of every view: $$f(\omega_i, \omega_o) = \frac{D\, G\, F}{4(\omega_o \cdot \mathbf{n})\,(\omega_i \cdot \mathbf{n})}$$ where $D$ is the normal distribution function (e.g., GGX), $G$ is geometric attenuation, and $F$ is the Fresnel term.

Relighting is achieved using a fast environment-based lighting engine that leverages pre-filtered environment maps and split-sum lookups. This apparatus enables real-time relighting and appearance edits, with the physically based outputs supporting high-fidelity visual effects.

5. Performance Evaluation and Generalization

SViM3D demonstrates state-of-the-art performance on multiple object-centric benchmarks, including the Poly Haven and Stanford Orb datasets. Quantitative metrics such as PSNR, SSIM, FID, LPIPS, and CLIP-based scores confirm superior perceptual quality and appearance consistency compared to methods that decouple RGB and material prediction.

Ablation studies highlight the efficacy of components such as view-dependent masking and homography correction for multi-view consistency and high-fidelity geometric reconstructions. Generalization is evidenced by robust outputs across diverse natural source images and lighting conditions.

6. Applications in Digital Content Creation

The joint prediction of relightable appearance and material maps provides immediate utility in content creation domains:

• AR/VR: View-consistent, relightable 3D assets for immersive environments
• Movies and games: Physically accurate appearance transferable across scenes, supporting controlled edits and dynamic lighting
• Asset pipelines: Unified neural prior for 3D reconstruction and appearance editing, enabling end-to-end differentiable content workflows

Because the predictions align with the conventions of real-time engines (PBR channel layouts, normals), SViM3D assets can be integrated with minimal adaptation.

7. Context and Significance

SViM3D advances the field by resolving a core ill-posed problem: joint inverse rendering of both geometry and materials from a single image under explicit camera control. In contrast to previous frameworks limited to RGB or simple reflectance estimation, SViM3D produces multi-view consistent materials and geometric cues suitable for downstream relighting and 3D asset synthesis. As a neural prior compatible with advanced reconstruction methods, it streamlines production pipelines and enables precise control over appearance under user-specified lighting and viewpoints (Engelhardt et al., 9 Oct 2025).

A plausible implication is that the framework’s extension to multi-channel latent diffusion modeling may serve as a template for future research in end-to-end differentiable inverse rendering, single-image asset synthesis, and physically-based editing in digital media creation.