Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials

Abstract: We present Meta 3D AssetGen (AssetGen), a significant advancement in text-to-3D generation which produces faithful, high-quality meshes with texture and material control. Compared to works that bake shading in the 3D object's appearance, AssetGen outputs physically-based rendering (PBR) materials, supporting realistic relighting. AssetGen generates first several views of the object with factored shaded and albedo appearance channels, and then reconstructs colours, metalness and roughness in 3D, using a deferred shading loss for efficient supervision. It also uses a sign-distance function to represent 3D shape more reliably and introduces a corresponding loss for direct shape supervision. This is implemented using fused kernels for high memory efficiency. After mesh extraction, a texture refinement transformer operating in UV space significantly improves sharpness and details. AssetGen achieves 17% improvement in Chamfer Distance and 40% in LPIPS over the best concurrent work for few-view reconstruction, and a human preference of 72% over the best industry competitors of comparable speed, including those that support PBR. Project page with generated assets: https://assetgen.github.io

• The paper introduces a novel two-stage pipeline that uses SDF-driven geometry and dual-channel inputs to achieve explicit PBR material decomposition.
• It leverages fine-tuned diffusion for multi-view image synthesis alongside a sparse-view neural reconstructor to generate high-fidelity 3D meshes.
• Empirical results highlight a 17% reduction in Chamfer Distance, a 40% LPIPS improvement, and a 72% human preference rate over competing methods.

Meta 3D AssetGen: High-Fidelity Text-to-Mesh Generation with Physical Material Control

Introduction

Meta 3D AssetGen introduces an architecture for text- or image-driven 3D mesh generation with explicit physically-based rendering (PBR) material control, synergizing rapid generation speed with high-fidelity geometry and texture, and, critically, decomposed material properties for downstream relighting and graphics workflows. Unlike prior generators that primarily bake appearance into per-object textures, AssetGen enforces a disentangled albedo/metalness/roughness model, yielding relightable, reusable asset outputs.

Figure 1: Meta 3D AssetGen: text/image-conditioned 3D mesh generator with PBR outputs (albedo, metalness, roughness), enabling high-fidelity relighting across environments.

Two-Stage Generation Architecture

AssetGen adopts a two-phase pipeline inspired by fast view-conditioned pipelines:

1. Text-to-Multi-View Image Synthesis: A large-scale diffusion model, finetuned with 3D captions, predicts a 4-view grid of images per prompt. Each view contains a 3-channel shaded image and a 3-channel intrinsic albedo image, leveraging semantic scene priors for view-consistency and material ambiguity resolution.
2. Conditional Image-to-3D Reconstruction: A sparse-view neural reconstructor processes the multiview, dual-channel input to yield a continuous SDF field and dense 3D fields for albedo, roughness, and metalness, enabling direct mesh extraction and material mapping. Texture refinement is performed in UV-space via a transformer conditioned on view projections for spatial detail recovery.

   Figure 2: Overview of AssetGen’s pipeline: (blue) text-to-multiview with shaded/albedo, (orange) image-to-3D with SDF/UV PBR extraction, (green) UV-space texture refinement.

Key Implementation Details

• SDF-Driven Geometry: Employing a triplane SDF representation improves isosurface extraction, geometric regularity, and supports direct supervision of signed-distance values, overcoming early volumetric-opaque-field artifacts.
• Physically-Based Material Decomposition: Through multichannel input (shaded+albedo), the network exploits per-pixel Bruneton-style deferred shading losses, efficiently approximating ground-truth renderings during backprop with low memory overhead.
• Texture Refinement: A UNet-based transformer fuses extracted mesh UVs with reprojected source views via cross-view attention, enhancing microtexture and repairing information loss during volumetric rendering.

Empirical Evaluation

Sparse-View 3D Reconstruction

AssetGen exhibits clear improvements over established image-to-3D baselines (Instant3D-LRM, InstantMesh, GRM, MeshLRM, LightplaneLRM) in both geometric and textural metrics. Specifically, a 17% reduction in Chamfer Distance and a 40% improvement in LPIPS are reported relative to state-of-the-art few-view mesh reconstructor baselines. Notably, geometry fidelity is substantially improved by direct SDF loss and SDF-based differentiable rendering.

Figure 3: AssetGen qualitative performance on sparse view mesh reconstruction, showing superior geometry (orange overlays) and texture compared to baselines; texture refinement yields further improvements.

Text-to-3D Generation with PBR Decomposition

Compared to industry and academic solutions supporting PBR, AssetGen achieves a 72% preference rate in human studies over the best fast baselines (Meshy v3, Luma Genie) in both visual quality and text-prompt alignment, while operating at significantly reduced time-to-asset (30s asset synthesis vs. minutes for "refinement" competitors).

Figure 4: Text-to-3D: AssetGen produces well-decoupled, high-fidelity materials; better metalness/roughness separation than Luma Genie and superior visual fidelity vs. baselines.

Figure 5: Demonstration of AssetGen’s PBR decomposition: precise albedo, metalness, and roughness separation, visually highlighting nuanced material control and relighting capability.

Evaluating Material Disentanglement

An explicit ablation of input strategies demonstrates that dual-channel input (shaded and albedo) outperforms single-channel or direct PBR prediction from the 2D generator—resolving material ambiguity and improving PBR map accuracy.

Figure 6: Prompt-driven material control (“a cat made of <MATERIAL>”): AssetGen yields plausible, physically meaningful material maps under environmental lighting.

Texture Refiner and Loss Formulation

AssetGen’s UV texture refiner is built on a dual-stream UNet with cross-view attention, where the main branch receives the coarse PBR/UV and geometric attributes, and side branches accept backprojected source views. Cross-view attention enables per-texel aggregation of the sharpest view-aligned evidence, further regularized by a deferred shading loss computed in rendered image-space.

Figure 7: (a) Architecture of the cross-view attention in the texture refiner; (b) deferred shading loss enforces physically correct material channel recovery by penalizing discrepancies in relit images.

Material Model and PBR Details

Meta 3D AssetGen’s material model implements a composite BRDF as:

$$f(,|k(x),n) = \frac{R}{\pi} + \frac{F(h|n) D(h|n) G_1(n,) G_1(n,)}{4 (n \cdot ) (n \cdot )}$$

where $R = \rho_0 (1-\gamma)$, $F_0 = \mathbf{1}(1-\gamma)+\rho_0\gamma$, and $k(x)=(\rho_0,\gamma,\alpha)$. The model is parameterized for pipeline compatibility with PBR-based shading engines and enables real-time relighting under dynamic environment maps.

Limitations and Future Directions

• View Consistency: The 2D diffusion model, even when fine-tuned for view-consistency, is not always perfect, causing artefacts or Janus issues in extreme scenarios.
• Sparse SDF Representation: The triplane SDF is optimal for memory scaling but is inherently inefficient for sparse or object-centric scenes, suggesting future transitions towards sparse voxel/octree encodings.
• Limited Channel Scope: AssetGen focuses on RGB/metalness/roughness, omitting emissive and AO channels; further work is needed to support universal asset compatibility.
• Object-Scale Only: Current evaluation is limited to object-scale 3D asset synthesis; the methodology provides a foundation for scene-scale and multi-object procedural modeling.

Implications and Prospects

AssetGen establishes a scalable paradigm for text-to-mesh 3D synthesis with full PBR material support at inference speeds amenable to interactive or iterative design loops. The explicit decomposition into relightable components advances the ecosystem for asset creation in AR/VR, digital content creation, and gaming, lowering the barrier for non-experts while maintaining integrability with physically-plausible render pipelines.

On the theoretical front, the combination of SDF-based geometric priors, dual-stream multi-view supervision, and UV-positioned refinement integrates the advantages of neural implicit and explicit mesh pipelines, challenging the established dichotomy in 3D asset generation.

Further research should investigate Octree/SparseHash-based volumetric fields for better scalability, integrate differentiable simulators for dynamic or physical property transfer, and extend material models to full spectral, emission, and subsurface domains.

Conclusion

Meta 3D AssetGen demonstrates that decoupling the stochastic material assignment from geometry prediction, leveraging an SDF-centric reconstruction pipeline, and executing UV-domain texture refinement enables high-quality, scalable, PBR-compatible text-to-mesh synthesis. Its results challenge the trade-off between synthesis speed and visual/material fidelity, and provide substantial empirical and methodological advances for future physically-plausible 3D asset generation.