Know3D: Prompting 3D Generation with Knowledge from Vision-Language Models

Abstract: Recent advances in 3D generation have improved the fidelity and geometric details of synthesized 3D assets. However, due to the inherent ambiguity of single-view observations and the lack of robust global structural priors caused by limited 3D training data, the unseen regions generated by existing models are often stochastic and difficult to control, which may sometimes fail to align with user intentions or produce implausible geometries. In this paper, we propose Know3D, a novel framework that incorporates rich knowledge from multimodal LLMs into 3D generative processes via latent hidden-state injection, enabling language-controllable generation of the back-view for 3D assets. We utilize a VLM-diffusion-based model, where the VLM is responsible for semantic understanding and guidance. The diffusion model acts as a bridge that transfers semantic knowledge from the VLM to the 3D generation model. In this way, we successfully bridge the gap between abstract textual instructions and the geometric reconstruction of unobserved regions, transforming the traditionally stochastic back-view hallucination into a semantically controllable process, demonstrating a promising direction for future 3D generation models.

• The paper introduces a framework that decouples semantic and geometric guidance using latent hidden-state prompting for enhanced 3D generation.
• It employs a fine-tuned vision-language model to generate back-view images, achieving superior semantic alignment and structural fidelity versus existing baselines.
• Experimental results show improved ULIP, Uni3D, PSNR/SSIM scores and demonstrate robust language-level control over unseen 3D structures.

Know3D: Knowledge-Guided Semantic Control for Single-View 3D Generation

Problem Definition and Background

Single-view image-to-3D generation remains intrinsically ill-posed due to severe information loss regarding unobserved geometry. Existing methods relying on generative priors, such as diffusion or autoregressive models, can hallucinate occluded regions but their predictions for unseen surfaces are stochastic, lack semantic grounding, and are prone to producing physically implausible or user-misaligned results. These limitations are exacerbated by the scarcity and low diversity of high-quality text-3D paired data, constraining the ability of 3D generation models to internalize sufficient world knowledge and structural commonsense.

Recent large-scale Vision-LLMs (VLMs) and Multimodal LLMs (MLLMs) have demonstrated superior capabilities in capturing semantic, structural, and commonsense priors from internet-scale multimodal data. However, naïve strategies for injecting VLM knowledge—such as autoregressive 3D synthesis with LLMs or direct feature transfer—result in performance degradation due to either lost modality-specific priors or insufficient spatial/geometric grounding.

Know3D Framework

Know3D introduces a novel semantic control paradigm for 3D synthesis, guided by the rich knowledge in multimodal foundation models. The core design decouples semantic and geometric guidance by leveraging a fine-tuned VLM-diffusion model (Qwen-Image-Edit-2511) as an intermediate bridge. The process involves:

1. Semantic-Aware Back-View Generation: Given a front-view image $I_\text{front}$ and a textual description $T$ for the unseen side, Qwen-Image-Edit is fine-tuned to generate plausible back-view images, conditioned on both $I_\text{front}$ and $T$. A stochastic prompting scheme ensures the model learns both unconditional and semantically controlled back-view synthesis. The Conditional Flow Matching (CFM) objective is adopted for stable training.
2. Latent Hidden-State Prompting: Rather than relying on pixel-space back-view images or high-level VLM embeddings, Know3D harnesses the intermediate layer hidden states from the diffusion transformer's denoising trajectory (MMDiT). These latents capture a fusion of spatial structure, object attributes, and prompt-conditioned semantics, providing effective priors for 3D shape completion.
3. Geometric Synthesis: Building on TRELLIS2, a state-of-the-art sparse voxel and latent-structure diffusion model, Know3D injects the extracted MMDiT hidden states via a parallel cross-attention conditioning branch. Shape synthesis then proceeds in two stages (coarse structure $\to$ fine-grained geometry), each optimizing a conditional velocity field estimator under the CFM objective.

Experimental Results and Analysis

Semantic Control and Generation Quality:

• Know3D achieves superior semantic alignment between generated meshes and input images, as evidenced by ULIP and Uni3D scores on the standard HY3D-Bench dataset.
• Compared to leading baselines such as Hunyuan3D-2mv, TRELLIS2, Step1X-3D, and Hi3DGen, Know3D yields stronger quantitative performance for both semantic and geometric fidelity (e.g., test ULIP 0.2174; Uni3D 0.3518; highest PSNR/SSIM and lowest CD/LPIPS).
• The method enables language-level control of backside geometry, supporting precise, user-driven manipulation of unseen regions.

Feature Design and Ablation:

• MMDiT intermediate hidden states (at $t=0.25$ in the denoising trajectory) provide optimal tradeoffs between spatial/semantic richness and low-level details, outperforming both VAE and DINOv3-based feature promptings.
• Ablation studies establish that later denoising stages are contaminated by noise, and earlier stages lack abstract structural cues.
• The modular design (frozen VLM backbone, LoRA adaptation for new attention branches) ensures stable fine-tuning and effective knowledge transfer.

Failure Modes:

• Structural and semantic robustness is ultimately upper-bounded by the underlying MLLM knowledge; if the VLM misinterprets user prompts or fails on spatial commonsense, 3D generation will reflect these errors.

Implications and Future Directions

Know3D sets a new direction in knowledge-guided, semantically controllable 3D generation. The framework demonstrates that leveraging multimodal diffusion intermediates as conditioning priors can bridge the gap between high-level language instructions and geometric shape completion—a longstanding open problem in the community.

The practical implications are substantial for content creation pipelines (fast, directed asset modeling for games/film/VR), and for embodied AI, which requires coherent reasoning about unobserved 3D environments. On the theoretical side, Know3D highlights the importance of intermediate semantic-structural states in multimodal models, suggesting analogous strategies for other ill-posed inverse generation tasks.

Key future directions include:

• Scaling and hardening the semantic backbone with more advanced MLLMs for stronger spatial reasoning and world knowledge.
• Exploring joint, end-to-end training paradigms for even tighter vision-language-geometry coupling, potentially fusing attention across both the image and 3D diffusion domains.
• Extending semantic control to more granular conditioning (e.g., part-level, relational prompts), compositional scene generation, and integration with interactive editing loops.

Conclusion

Know3D represents a significant step forward in bridging semantic and geometric reasoning for single-view 3D generation. Through latent hidden-state knowledge prompting and diffusion-based semantic conditioning, the framework achieves both high-fidelity geometry reconstruction and user-controllable synthesis for unseen regions. This approach opens new avenues for harnessing generalizable world knowledge in multimodal generative modeling and underscores the central role of VLM intermediates in future 3D content generation systems.