Know3D: Advanced 3D Generation Framework

• Know3D is a 3D generation framework that fuses semantic cues from text and images to synthesize plausible, unobserved back-view regions.
• It employs a three-stage pipeline—semantic-aware 2D bridge, latent hidden-state extraction, and sparse-voxel 3D synthesis—to integrate geometric and semantic information.
• It achieves superior alignment and improved metrics (ULIP, Uni3D, PSNR, Chamfer) compared to baselines, enabling more controllable and realistic 3D asset generation.

Know3D is a 3D generation framework that leverages knowledge from multimodal vision-LLMs (VLMs) via latent hidden-state injection to enable controllable, language-guided synthesis of the unobserved back-side of 3D assets. It addresses the longstanding challenge of stochastic and implausible geometry in single-image 3D generation by fusing semantic cues from textual prompts and visual input at the latent level, yielding improved alignment of unseen regions with user intentions and physical plausibility (Chen et al., 24 Mar 2026).

1. Three-Stage Pipeline Architecture

Know3D operates as a three-stage pipeline:

A. Semantic-Aware Novel View Generation (2D Bridge)

• Inputs: A single front RGB image $I_{\text{front}}$ and a textual prompt $P$ (describing desired back view).
• VLM Encoder: Qwen2.5-VL encodes $(I_{\text{front}}, P)$ as high-level semantic features $H_{\text{vlm}}$.
• VAE Encoder: $E$ encodes $I_{\text{front}}$ into spatial latents $Z_{\text{front}} = E(I_{\text{front}})$.
• Diffusion Transformer (MMDiT): Starts from noisy latent $Z_t \sim (1-t)Z_{\text{back}} + t\epsilon$, then iteratively denoises conditioned on $(H_{\text{vlm}}, Z_{\text{front}})$ via cross-attention, generating an edited back-view latent $\hat{Z}_{\text{back}}$ decoded to $I_{\text{back}}$.

B. Knowledge Extraction

• At fixed diffusion timestep $t^* = 0.25$, intermediate hidden states are extracted from $n$ layers of MMDiT, concatenated along the channel dimension to form $$H_{\text{DiT}} = \text{Concat}(h_{t^*}^{(1)}, \ldots, h_{t^*}^{(n)})$$.

C. 3D Generative Model (TRELLIS2-based)

• Two-stage Sparse-Voxel Diffusion:
  • Stage 1: $G_{\text{ss}}/D_{\text{ss}}$ generate a coarse sparse structure $V_{\text{ss}}$.
  • Stage 2: $G_{\text{geo}}/D_{\text{geo}}$ refine to high-resolution geometry $V_{\text{geo}}$ conditioned on $V_{\text{ss}}$.
• Dual Conditioning: Both $F_{\text{front}}$ (from VAE) and semantic-structural signal $H_{\text{DiT}}$ are injected via parallel cross-attention into each DiT block of the 3D generator.

2. Latent Hidden-State Injection Mechanism

Latent hidden-state injection fuses geometric cues (front-view) with semantic priors (from MMDiT hidden states) at each DiT block via parallel cross-attention branches. Formally:

• Let $F$ be the 3D DiT block activation.
• Classical attention:
  • $F_{\text{sa}} = \text{SelfAttn}(F)$
  • $$A_{\text{img}} = \text{CrossAttn}(F_{\text{sa}}, F_{\text{img}}, F_{\text{img}})$$
• Injected branch from VLM-diffusion:
  • $$A_{\text{DiT}} = \text{ZeroInitLinear}(\text{CrossAttn}(F_{\text{sa}}, H_{\text{DiT}}, H_{\text{DiT}}))$$
• Output:
  • $$F_{\text{out}} = F + A_{\text{img}} + A_{\text{DiT}}$$

In compact notation: $z' = z + W h$, where $h$ is the projected and layer-normalized $H_{\text{DiT}}$, $W$ includes cross-attention weights and zero-initialized scaling. This structure allows semantic-structural priors to dynamically inform volumetric geometry synthesis.

3. Diffusion Processes and Conditioning

Forward Diffusion (2D Bridge)

A variance-exploding linear schedule on $t \in [0,1]$ produces:

$$Z_t = (1-t) Z_\text{back} + t\,\epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$

Reverse Denoising

The conditional denoiser $v_\theta$ predicts the "velocity":

$$v_\theta(Z_t, t; H_{\text{vlm}}, Z_{\text{front}}) \approx \epsilon - Z_{\text{back}}$$

The reverse update is:

$$Z_{t-\Delta t} \approx Z_t + \Delta t \cdot v_\theta(Z_t, t; \cdot)$$

Conditioning is realized via cross-attention to $H_{\text{vlm}}$ in MMDiT and latent injection with $H_{\text{DiT}}$ in the 3D generator.

4. Training Paradigms and Objective Functions

2D Semantic-Aware View Generation

Employs Conditional Flow Matching (CFM):

$$\mathcal{L}_{2D} = \mathbb{E}_{t, Z_{\text{back}}, \epsilon} \| v_\theta(Z_t, t; H_{\text{vlm}}, Z_{\text{front}}) - (\epsilon - Z_{\text{back}}) \|^2$$

3D Geometry Generation (Both Stages)

$$\mathcal{L}_{3D} = \mathbb{E}_{T, x_0, \epsilon} \| v_\phi(x_T, T; F_{\text{img}}, H_{\text{DiT}}) - (\epsilon - x_0) \|^2$$

No additional KL or explicit reconstruction losses are introduced beyond the VAE auto-encoding component in the 2D bridge stage.

5. Implementation Specifications

• VLM Bridge: Base model is Qwen-Image-Edit-2511 (Qwen2.5-VL + DiT), fine-tuned via LoRA (rank=64) on 32 × A800 GPUs, batch size 32, over 20k steps, learning rate $1\times 10^{-4}$. The paired front/back-view dataset includes 5k high-quality TexVerse meshes, 12 views/mesh, with component-level text annotations.
• 3D Generator: TRELLIS2 backbone, with two-stage sparse-voxel DiTs (at $64^3$ and $512^3$ resolutions). Injected modules (parallel cross-attn) are fully fine-tuned; original model weights LoRA-adapted (rank=64). Training uses 60k meshes, 70k steps, batch 64, $1\times 10^{-4}$ learning rate on 32 × A800 GPUs.
• Feature Extraction: Hidden states are extracted from MMDiT blocks {20, 30, 40} at $t^* = 0.25$ and channel-concatenated (per-layer dim $\approx$ 768–1024).

6. Evaluation Protocols and Comparative Results

• Dataset: HY3D-Bench held-out single-image 3D generation benchmark.
• Metrics:
  • Semantic alignment: ULIP, Uni3D (higher is better).
  • Geometry: PSNR/SSIM/LPIPS (back-view renders); Chamfer distance (meshes).
• Baselines: Hunyuan3D-2.1, TRELLIS2, Hunyuan3D-2 mv, Step1X-3D, Hi3DGen, Direct3D-S2.
• Key Results:
  • ULIP / Uni3D: Know3D achieves 0.2174 / 0.3518 (best), outperforming TRELLIS+HUN (0.2143 / 0.3434).
  • PSNR/SSIM: +0.8–1.0 dB gain; LPIPS reduced by 0.04; Chamfer distance improved by 10–15%.
  • Qualitative: Know3D yields more plausible and controllable back-views (Figure 1 & 6), and supports diverse backside structures per text prompt (Figure 2, 10, 11).

Metric	Know3D	Best Baseline (T+H)
ULIP	0.2174	0.2143
Uni3D	0.3518	0.3434
Chamfer (↓)	-10-15%	Reference

7. Analysis and Ablation Studies

Diffusion-Timestep $t$ Selection for $H_{\text{DiT}}$

• $t=0.25$ achieves optimal IoU (0.352) and Chamfer distance (2.262). Earlier ($t=0$) underrepresents semantics; later ($t>0.5$) admits excessive noise.

Feature Representations

• VAE encoder latents: IoU=0.308, CD=2.803.
• DINOv3 features: IoU=0.342, CD=2.385.
• MMDiT hidden states ($t=0.25$): IoU=0.352, CD=2.262 (strongest).

Multi-View Baselines

• Know3D outperforms Hunyuan3D-2 mv on identical input: ULIP +0.006, CD −0.007, validating that injecting diffusion transformer hidden states surpasses using raw synthesized back-views.

A plausible implication is that latent-level semantic-structural information, extracted via diffusion transformer hidden states at appropriately chosen timesteps, provides a stronger prior for geometry generation than either image-based or earlier feature representations.

---

Know3D demonstrates that bridging vision-LLM knowledge and 3D generative diffusion via latent state injection enables semantically-controllable, visually plausible completion of unobserved object regions, outperforming both single-view and naive multi-view generation approaches across multiple quantitative and qualitative benchmarks (Chen et al., 24 Mar 2026).