---
title: 'Know3D: Advanced 3D Generation Framework'
url: https://www.emergentmind.com/topics/know3d-framework
type: topic
---

# Know3D: Advanced 3D Generation Framework

Know3D is a 3D generation framework that leverages knowledge from multimodal vision-language models (VLMs) via latent hidden-state injection to enable controllable, language-guided synthesis of the unobserved back-side of 3D assets. It addresses the longstanding challenge of stochastic and implausible geometry in single-image 3D generation by fusing semantic cues from textual prompts and visual input at the latent level, yielding improved alignment of unseen regions with user intentions and physical plausibility [2603.22782].

## 1. Three-Stage Pipeline Architecture

Know3D operates as a three-stage pipeline:

**A. Semantic-Aware Novel View Generation (2D Bridge)**  
- **Inputs**: A single front RGB image $I_{\text{front}}$ and a textual prompt $P$ (describing desired back view).
- **VLM Encoder**: Qwen2.5-VL encodes $(I_{\text{front}}, P)$ as high-level semantic features $H_{\text{vlm}}$.
- **VAE Encoder**: $E$ encodes $I_{\text{front}}$ into spatial latents $Z_{\text{front}} = E(I_{\text{front}})$.
- **Diffusion Transformer (MMDiT)**: Starts from noisy latent $Z_t \sim (1-t)Z_{\text{back}} + t\epsilon$, then iteratively denoises conditioned on $(H_{\text{vlm}}, Z_{\text{front}})$ via cross-attention, generating an edited back-view latent $\hat{Z}_{\text{back}}$ decoded to $I_{\text{back}}$.

**B. Knowledge Extraction**  
- At fixed diffusion timestep $t^* = 0.25$, intermediate hidden states are extracted from $n$ layers of MMDiT, concatenated along the channel dimension to form $H_{\text{DiT}} = \text{Concat}(h_{t^*}^{(1)}, \ldots, h_{t^*}^{(n)})$.

**C. 3D Generative Model (TRELLIS2-based)**  
- **Two-stage Sparse-Voxel Diffusion**:  
    - Stage 1: $G_{\text{ss}}/D_{\text{ss}}$ generate a coarse sparse structure $V_{\text{ss}}$.  
    - Stage 2: $G_{\text{geo}}/D_{\text{geo}}$ refine to high-resolution geometry $V_{\text{geo}}$ conditioned on $V_{\text{ss}}$.
- **Dual Conditioning**: Both $F_{\text{front}}$ (from VAE) and semantic-structural signal $H_{\text{DiT}}$ are injected via parallel cross-attention into each DiT block of the 3D generator.

## 2. Latent Hidden-State Injection Mechanism

Latent hidden-state injection fuses geometric cues (front-view) with semantic priors (from MMDiT hidden states) at each DiT block via parallel cross-attention branches. Formally:

- Let $F$ be the 3D DiT block activation.
- Classical attention:
  - $F_{\text{sa}} = \text{SelfAttn}(F)$
  - $A_{\text{img}} = \text{CrossAttn}(F_{\text{sa}}, F_{\text{img}}, F_{\text{img}})$
- Injected branch from VLM-diffusion:
  - $A_{\text{DiT}} = \text{ZeroInitLinear}(\text{CrossAttn}(F_{\text{sa}}, H_{\text{DiT}}, H_{\text{DiT}}))$
- Output:
  - $F_{\text{out}} = F + A_{\text{img}} + A_{\text{DiT}}$

In compact notation: $z' = z + W h$, where $h$ is the projected and layer-normalized $H_{\text{DiT}}$, $W$ includes cross-attention weights and zero-initialized scaling. This structure allows semantic-structural priors to dynamically inform volumetric geometry synthesis.

## 3. Diffusion Processes and Conditioning

**Forward Diffusion (2D Bridge)**  
A variance-exploding linear schedule on $t \in [0,1]$ produces:
$$
Z_t = (1-t) Z_\text{back} + t\,\epsilon, \quad \epsilon \sim \mathcal{N}(0, I)
$$

**Reverse Denoising**  
The conditional denoiser $v_\theta$ predicts the "velocity":
$$
v_\theta(Z_t, t; H_{\text{vlm}}, Z_{\text{front}}) \approx \epsilon - Z_{\text{back}}
$$
The reverse update is:
$$
Z_{t-\Delta t} \approx Z_t + \Delta t \cdot v_\theta(Z_t, t; \cdot)
$$

Conditioning is realized via cross-attention to $H_{\text{vlm}}$ in MMDiT and latent injection with $H_{\text{DiT}}$ in the 3D generator.

## 4. Training Paradigms and Objective Functions

**2D Semantic-Aware View Generation**  
Employs Conditional Flow Matching (CFM):
$$
\mathcal{L}_{2D} = \mathbb{E}_{t, Z_{\text{back}}, \epsilon} \| v_\theta(Z_t, t; H_{\text{vlm}}, Z_{\text{front}}) - (\epsilon - Z_{\text{back}}) \|^2
$$

**3D Geometry Generation (Both Stages)**
$$
\mathcal{L}_{3D} = \mathbb{E}_{T, x_0, \epsilon} \| v_\phi(x_T, T; F_{\text{img}}, H_{\text{DiT}}) - (\epsilon - x_0) \|^2
$$

No additional KL or explicit reconstruction losses are introduced beyond the VAE auto-encoding component in the 2D bridge stage.

## 5. Implementation Specifications

- **VLM Bridge**: Base model is Qwen-Image-Edit-2511 (Qwen2.5-VL + DiT), fine-tuned via LoRA (rank=64) on 32 × A800 GPUs, batch size 32, over 20k steps, learning rate $1\times 10^{-4}$. The paired front/back-view dataset includes 5k high-quality TexVerse meshes, 12 views/mesh, with component-level text annotations.
- **3D Generator**: TRELLIS2 backbone, with two-stage sparse-voxel DiTs (at $64^3$ and $512^3$ resolutions). Injected modules (parallel cross-attn) are fully fine-tuned; original model weights LoRA-adapted (rank=64). Training uses 60k meshes, 70k steps, batch 64, $1\times 10^{-4}$ learning rate on 32 × A800 GPUs.
- **Feature Extraction**: Hidden states are extracted from MMDiT blocks {20, 30, 40} at $t^* = 0.25$ and channel-concatenated (per-layer dim $\approx$ 768–1024).

## 6. Evaluation Protocols and Comparative Results

- **Dataset:** HY3D-Bench held-out single-image 3D generation benchmark.
- **Metrics:**
  - Semantic alignment: ULIP, Uni3D (higher is better).
  - Geometry: PSNR/SSIM/LPIPS (back-view renders); Chamfer distance (meshes).
- **Baselines:** Hunyuan3D-2.1, TRELLIS2, Hunyuan3D-2 mv, Step1X-3D, Hi3DGen, Direct3D-S2.
- **Key Results:**  
    - ULIP / Uni3D: Know3D achieves 0.2174 / 0.3518 (best), outperforming TRELLIS+HUN (0.2143 / 0.3434).
    - PSNR/SSIM: +0.8–1.0 dB gain; LPIPS reduced by 0.04; Chamfer distance improved by 10–15%.
    - Qualitative: Know3D yields more plausible and controllable back-views (Figure 1 & 6), and supports diverse backside structures per text prompt (Figure 4, 10, 11).

| Metric       | Know3D        | Best Baseline (T+H) |
|--------------|---------------|---------------------|
| ULIP         | 0.2174        | 0.2143              |
| Uni3D        | 0.3518        | 0.3434              |
| Chamfer (↓)  | -10-15%       | Reference           |

## 7. Analysis and Ablation Studies

**Diffusion-Timestep $t$ Selection for $H_{\text{DiT}}$**  
- $t=0.25$ achieves optimal IoU (0.352) and Chamfer distance (2.262). Earlier ($t=0$) underrepresents semantics; later ($t>0.5$) admits excessive noise.

**Feature Representations**  
- VAE encoder latents: IoU=0.308, CD=2.803.
- DINOv3 features: IoU=0.342, CD=2.385.
- MMDiT hidden states ($t=0.25$): IoU=0.352, CD=2.262 (strongest).

**Multi-View Baselines**  
- Know3D outperforms Hunyuan3D-2 mv on identical input: ULIP +0.006, CD −0.007, validating that injecting diffusion transformer hidden states surpasses using raw synthesized back-views.

A plausible implication is that latent-level semantic-structural information, extracted via diffusion transformer hidden states at appropriately chosen timesteps, provides a stronger prior for geometry generation than either image-based or earlier feature representations.

---

Know3D demonstrates that bridging vision-language model knowledge and 3D generative diffusion via latent state injection enables semantically-controllable, visually plausible completion of unobserved object regions, outperforming both single-view and naive multi-view generation approaches across multiple quantitative and qualitative benchmarks [2603.22782].

Source: https://www.emergentmind.com/topics/know3d-framework