Know3D: 3D Knowledge-Driven Generative Framework

• Know3D is a 3D generative framework that integrates vision-language models with diffusion techniques to create semantically coherent back views and complete 3D assets from single images.
• It employs a three-stage pipeline incorporating VLM feature injection, sparse-voxel and structured latent diffusion with flow-matching losses for precise geometric synthesis.
• Empirical results show Know3D outperforms baselines with enhanced semantic alignment and approximately 10% lower Chamfer Distance compared to TRELLIS2.

Know3D is a 3D generative framework designed to inject knowledge from multimodal vision-LLMs into the synthesis pipeline, enabling language-controlled generation of the back-view and completion of 3D assets from single-view inputs. By leveraging latent hidden-state injection from vision-LLMs (VLMs) into a diffusion-based 3D architecture, Know3D addresses the challenge of reconstructing plausible and semantically accurate unseen regions, where conventional single-view methods often produce stochastic and uncontrolled geometries (Chen et al., 24 Mar 2026).

1. Architectural Foundations

Know3D implements a three-stage VLM-diffusion-3D pipeline. The upstream vision-LLM employed is Qwen2.5-VL, which processes the input front-view image and prompt $P = P_{view} + P_{back}$. $P_{view}$ is a fixed phrase guiding reverse-view synthesis, while $P_{back}$ introduces stochastically sampled semantic descriptions of object back-side components. The image is encoded into VAE latents $Z_{front}$, and the prompt-image pair $(I_{front}, P)$ is converted into high-level hidden states $H_{VLM}$ via Qwen2.5-VL.

A Diffusion Transformer (MMDiT) is used to generate the reverse view latents ($Z_{back}$) by denoising noisy samples $Z_t$, bridging the semantic gap from VLM outputs to 2D representations. At a selected intermediate diffusion timestep $t^*=0.25$, hidden states $h^{(i)}_{t^*}$ from layers $P_{view}$0 are extracted and concatenated ($P_{view}$1).

The 3D generator is based on the "Trellis2" backbone and operates in two stages:

• Coarse generation: sparse-voxel diffusion $P_{view}$2 outputs $P_{view}$3
• Refinement: structured latent diffusion $P_{view}$4 outputs $P_{view}$5

Both stages are optimized with flow-matching losses.

2. Mathematical Formulation and Training Objectives

Know3D's diffusion process at sampling step $P_{view}$6 is: $P_{view}$7 with noise schedule $P_{view}$8, cumulative noise $P_{view}$9, denoiser $P_{back}$0 parameterized by the DiT, and $P_{back}$1 controlling stochasticity.

The latent hidden-state injection is defined as $P_{back}$2: $P_{back}$3 where $P_{back}$4 is the incoming feature, $P_{back}$5 is the front-view feature, and $P_{back}$6 the projected DiT hidden states. The additional branch is zero-initialized to preserve pretrained priors.

Back-view image diffusion is trained with Conditional Flow Matching (CFM): $P_{back}$7 where $P_{back}$8, $P_{back}$9. Both 3D diffusion stages use a similar CFM objective. No explicit image-space reconstruction loss is applied; geometric fidelity is an emergent property of the diffusion priors and evaluation.

3. Language Control and Prompt Engineering

The language control mechanism constructs prompts as $Z_{front}$0. $Z_{front}$1 is a static instruction: “Show the reverse side of the subject as seen from rotating the camera 180 degrees.” $Z_{front}$2, with probability $Z_{front}$3, samples component-level descriptions (e.g., "A small balcony on the wall.").

The VLM Qwen2.5-VL tokenizes the prompt and cross-attends to image patches, generating hidden states $Z_{front}$4. These hidden states are later used to guide the generation of back-view latents. Conditional generation with different $Z_{front}$5 descriptions produces varying geometric completions of the unseen side, allowing granular semantic control. When $Z_{front}$6 is omitted, completions are plausible but generic.

4. Implementation and Dataset Considerations

Key implementation details include:

• A linear $Z_{front}$7 schedule in DiT, with injection timestep $Z_{front}$8 selected via ablation.
• Qwen2.5-VL employs LoRA adapters (rank=64) across all attention layers for fine-tuning.
• The diffusion transformer (MMDiT) for 2D and Trellis2 DiT for 3D (64³ sparse and 512³ structured resolution).
• Injection employs a new cross-attention branch in each DiT block, with a zero-initialized linear projection for stability.

Training employs 20k steps (back-view), 70k steps (3D), batch sizes of 32/64, a learning rate of $Z_{front}$9, and 32×A800 GPUs per stage. Back-view generation leverages 5k meshes from TexVerse with 12 azimuth views and semantic annotations. Full end-to-end 3D generation uses 60k meshes, and evaluation is on the HY3D-Bench with previously unseen 3D assets.

5. Empirical Evaluation and Ablations

Quantitative results demonstrate Know3D surpasses single-view baselines such as Hunyuan3D-2.1 and TRELLIS2 in semantic alignment (ULIP, Uni3D), geometric metrics (PSNR, SSIM, LPIPS), and reduces Chamfer Distance by approximately 10% relative to TRELLIS2. Against front+generated back multi-view baselines (Hunyuan3D-2mv), Know3D achieves superior semantic and geometric scores.

Qualitative analysis reveals that even unconditional back-view completions improve plausibility compared to TRELLIS2. Prompt-guided variants showcase controlled, user-specified modifications (e.g., appearance of specific components). The method also mitigates common distortions present in multi-view baselines, such as duplicated structures.

Ablation studies underscore the criticality of intermediate timestep extraction ($(I_{front}, P)$0 yielding optimal IoU and Chamfer Distance) and feature source selection, with MMDiT hidden states outperforming DINOv3 features and fully-denoised VAE latents.

Baseline	Semantic Alignment (ULIP/Uni3D)	Chamfer Distance Reduction
Hunyuan3D-2.1	Lower	Baseline
TRELLIS2	Lower	–
Hunyuan3D-2mv	Lower than Know3D	–
Know3D	Highest	~10% lower than TRELLIS2

6. Limitations and Prospects

Know3D's performance is sensitive to the VLM's semantic interpretation; misinterpretation of $(I_{front}, P)$1 or low-quality back-view image generation can misguide downstream 3D synthesis. The system's capability to adapt to novel concepts is currently constrained by the use of LoRA adapters and reliance on fixed pretrained backbones. Computational costs for two-stage diffusion training are high.

Potential directions for further research include deploying stronger multimodal VLM backbones with improved spatial grounding, experimenting with continuous or learnable injection schedules for $(I_{front}, P)$2, integrating mesh-based refinement or differentiable rendering objectives, and augmenting textual annotation schemes to enable richer compositionality and support for multi-object scenes (Chen et al., 24 Mar 2026).

7. Significance and Outlook

Know3D presents a methodologically distinct approach to bridging vision-language semantic knowledge and 3D generative modeling via latent hidden-state injection, enabling explicit control over the synthesis of otherwise ambiguous or unobserved regions. The framework offers a pathway to semantically robust and user-controllable 3D asset generation and motivates further exploration at the intersection of language-driven reasoning and spatial reconstruction in generative AI systems (Chen et al., 24 Mar 2026).