- The paper introduces a modular pipeline that decomposes image-to-3D asset generation into four specialized modules, achieving state-of-the-art fidelity on furniture benchmarks.
- The method leverages a two-stage DiT framework combined with VAE-based latent SDF modeling for high-fidelity geometry reconstruction and accurate detail recovery.
- It employs multiview texture completion and cross-modal material assignment to ensure physically consistent, editable 3D design assets.
Home3D 1.0: Modular High-Fidelity Image-to-3D Asset Generation for Interior Design
System Motivation and Problem Scope
Home3D 1.0 targets the specialized and demanding challenge of generating product-faithful 3D furniture and décor models from a single reference image, with explicit suitability for interior design and e-commerce applications. Unlike generic 3D object generation, where category plausibility may suffice, the requirements in this domain are more stringent: results must preserve specific geometric details, style, construction semantics, and material boundaries. Common 3D generation defects such as mesh fragmentation, poor structural fidelity, or texture inconsistency often render assets unusable in these settings.
Modular Pipeline Architecture
The core design philosophy of Home3D 1.0 is the decomposition of the image-to-3D task into four tightly coupled, specialized modules: Geometry Reconstruction, Texture Generation, Material Assignment, and Material-Aware Part Decomposition.
Geometry: High-Fidelity, Product-Faithful Mesh Reconstruction
Geometry synthesis leverages a two-stage, image-conditioned pipeline combining VAE-based latent SDF modeling with a coarse-to-fine Flow Matching DiT framework. The pipeline is pretrained on diverse 3D assets (ObjaverseXL, proprietary high-quality models) and fine-tuned on curated furniture. Key preprocessing steps include canonicalization of orientation, watertight SDF computation, and quality filtering.
At inference, the pipeline first generates a globally plausible coarse geometry using DiT with visual conditioning from a reference image. This is refined by a second DiT stage for sharp boundaries and thin structures. The VAE decoder finally constructs SDF fields for mesh extraction. This separation of global topology and local refinement yields robust surface completions even in occluded or weakly observed regions.
Strong numerical results: Home3D 1.0 achieves state-of-the-art scores on a 100-case furniture benchmark—Chamfer Distance 0.4936 (×10−3), EMD 5.174 (×10−2), [email protected] 0.6329—outperforming leading closed-source baselines.
Texture: Reliable, Multiview Albedo and 3D Texture Completion
Home3D’s texture module introduces a multiview texture generation pipeline. Rather than independently producing textures for each view (which leads to inconsistency), the model predicts all canonical-view albedo maps jointly from the input image and geometry. These are then reprojected onto the mesh; missing regions from occlusion or limited field of view are filled by a 3D continuous texture completion module that operates in surface space, avoiding UV artifacts and topological sensitivity.
The training procedure incorporates realistic illumination, consistency regularization, and gradient-based losses to disentangle intrinsic color from environmental effects, enhancing cross-view coherence.
Material Assignment and PBR Retrieval: Physically Consistent, Editable Materials
Material decomposition and assignment are addressed by a part-aware segmentation framework called MatWeaver. Segmentation is accomplished by video-based tracking using multi-view renderings, providing robust mapping between image and mesh surfaces, followed by UV-space consensus voting for accuracy.
For each segment, a hierarchical multi-modal retrieval pipeline matches mesh components to materials from a curated, hierarchically annotated PBR material database. Cross-modal embeddings (visual and semantic) and VLM-based reranking (including Qwen3-VL, Gemini, expert VLM judges) identify the most physically and visually coherent materials. Baking consolidates the selected PBR material attributes—metallic, roughness, normal maps—into compact, engine-ready textures. The system also incorporates pose estimation and dense feature alignment for projection accuracy.
Material-Aware Parts: Efficient, Direct Part Structure Generation
Unlike standard function-aware part decompositions, Home3D focuses on material-editable semantic regions. The parts module is built around a multi-head SDF VAE (PartVAE) that learns a shared latent space encoding all part channels simultaneously. Decoding produces all part-specific SDF fields in a single evaluation, addressing efficiency and allowing for direct, independent material edits.
Image-conditioned DiT transformers, conditioned on DINO features and geometry latents, denoise the shared code for part structure prediction. The result is a consistent segmentation of objects into granularity suitable for design tasks (e.g., wood, metal, glass, fabric), facilitating seamless material replacement operations without laborious manual segmentation.
Evaluation and Comparative Results
Each pipeline component is individually and jointly evaluated.
Geometry: Home3D 1.0 demonstrates superior fidelity across standard mesh metrics and qualitative comparisons, particularly in the recovery of thin structures and occluded details critical to furniture.
Texture/Material: Quantitatively, it leads in both CLIP-I and LPIPS appearance similarity metrics. Qualitatively, high-frequency material cues—such as fabric weave, wood grain, and glass reflections—are materially more consistent and physically plausible compared to baselines.
Part Structure: The part-aware VAE enables constant-time inference regardless of the number of parts (versus linear scaling in per-part decoders), with competitive part reconstruction scores. Material-aware segmentation consistently results in clean, editable boundaries, as demonstrated by qualitative visualization and editability.
Discussion: Practical and Theoretical Implications
Practically, Home3D 1.0 represents a significant step towards scalable, automated 3D asset creation for real-world applications in e-commerce visualization and interior design. The explicit modularity—geometry, texture, material, parts—addresses specific failure modes and supports downstream use cases, such as rapid material iteration and editing.
Theoretically, the decoupling of product-faithful geometry synthesis, semantically aware material assignment, and direct part SDF field decoding sets a new direction for interpretable, controllable 3D asset, moving beyond black-box texture generation or function-oblivious segmentation. The heavy use of VLMs for semantic supervision and retrieval illustrates a promising convergence between visual-LLMs and traditional computer graphics pipelines.
Future Prospects
Potential advances include cross-module feedback (using retrieved materials to regularize geometry and texture), addressing articulated and deformable assets with dynamic part modules, further scaling of material databases, and integration of broader scene context for holistic interior synthesis. Robustness to more severe domain shifts—such as in-the-wild images, occlusions, or variations in lighting—remains an open challenge.
Conclusion
Home3D 1.0 introduces a modular, high-fidelity approach to image-to-3D asset generation, specifically tailored for the demands of interior design and e-commerce. Through the separation of geometry, texture, material, and parts, and with strong empirical results, it enables production-level asset creation and editing with a degree of product-faithful specificity and physical realism not found in prior systems. This work informs both practical deployment and future research into interpretable, editable 3D generation.