Shap-E: Conditional 3D Asset Generation

• Shap-E is a conditional generative model for 3D assets that uses a two-stage method combining an encoder and latent diffusion to map 3D data into implicit function parameters.
• It employs a dual implicit representation with NeRF and STF fields via an MLP architecture, enabling both textured mesh extraction and volumetric rendering.
• Shap-E achieves rapid text- and image-conditional 3D asset generation, outperforming previous methods in fidelity, speed, and geometric detail.

Shap-E is a conditional generative model for 3D assets that operates by directly producing the parameters of implicit functions suitable for rendering as both textured 3D meshes and neural radiance fields (NeRFs). Unlike prior 3D generative modeling approaches that target a fixed explicit representation such as point clouds or voxel grids, Shap-E employs a two-stage method: an encoder deterministically maps a 3D asset into the parameters of a representation MLP, and a latent diffusion model is trained on these MLP parameters for conditional synthesis from text or images. As a result, Shap-E is capable of generating diverse, high-fidelity 3D content in seconds and supports both zero-shot text-conditional and image-conditional generation (Jun et al., 2023).

1. Implicit Function Representation and Model Architecture

Shap-E represents each 3D asset by a 6-layer, bias-free MLP that defines two coupled implicit fields: a neural radiance field (NeRF) and a signed distance & texture field (STF). The MLP receives as input a sinusoidal embedding $\varphi(x)$ of spatial coordinates $x \in \mathbb{R}^3$, specifically a 15-frequency sinusoidal (Fourier) encoding as in NeRF and 3DiM ($\varphi(x) \in \mathbb{R}^{93}$). The MLP structure consists of:

• First four layers (256 units each): Instance-specific, with weights output via the encoder for each asset.
• Final two layers: Shared globally across instances.

The MLP branches produce:

• NeRF: density $\sigma(x)$ via sigmoid, view-dependent color $c(x, d)$.
• STF: signed distance $s(x)$ via tanh and color $t(x)$ via sigmoid.

The asset-specific weights $W^1 \ldots W^4$, along with the shared weights, fully define an implicit field and thus enable volumetric rendering or mesh extraction via Marching Cubes (Jun et al., 2023).

2. Encoder and Latent Space

The encoder in Shap-E processes both a dense RGB point cloud (16,384 points) and multiple rendered 2D views of the asset (20 views at $256 \times 256$ resolution). The pipeline involves:

• PointConv for feature extraction from the point cloud,
• Cross-attention between point features and both point embeddings and image patch embeddings,
• A transformer for global integration,
• Outputting a $1024 \times 1024$ latent feature tensor (constituting the latent $r$ or $h$),
• A final projection yielding the first four weight matrices of the MLP decoder.

The output latent ($r \in \mathbb{R}^{1024 \times 1024}$) is clamped to $[-1, 1]$ and is the basis for the downstream diffusion model. Shap-E does not impose an explicit prior on these latents beyond the diffusion modeling (Jun et al., 2023, Chen et al., 2023).

3. Conditional Diffusion Prior

A transformer-based U-Net is trained as a latent diffusion model over the encoder latents. Diffusion operates by sequentially noising the latent with a variance schedule $\bar{\alpha}_t = e^{-12t}$ for $t \in [0, 1]$ and learning a denoising model parameterized to predict either $x_0$ or $\epsilon$ as in DDPM. Classifier-free guidance is implemented for both text and image conditioning, with text guidance scales $s \approx 20$ and image guidance $s \approx 3$. At inference, the model generates a new latent from pure noise via iterative denoising, then decodes via the MLP to a 3D implicit function.

The underlying transformer U-Net architecture is based on Point-E's (300M parameters), adapted to the $1024$-dimensional token width, and can operate in either text-conditional or image-conditional modes (Jun et al., 2023, Chen et al., 2023).

4. Training Protocol and Data

Shap-E is trained in two distinct phases:

1. Encoder Pre-training:
   • Datasets: ≈3 million CAD assets (ShapeNet, ABC, Objaverse, 1M high-quality assets), with ≈120,000 human-labeled text captions.
   • NeRF Pre-training: 600k iters, Adam (lr = 1e-4, batch 64, 16-bit).
   • STF Distillation: 50k iters, Adam (lr = 1e-5, batch 64), using a pretrained SDF regressor and RGB nearest neighbor supervision.
   • STF Fine-tuning: 65k iters, Adam (lr = 1e-5), mesh extraction with differentiable Marching Cubes.
2. Latent Diffusion Training:
   • On all (clamped-noised) encoder latents, same dataset, transformer diffusion as in Point-E (batch size 512, 64 GPUs, lr = 2e-4, mixed precision).

Guidance scale and sampling schedules are tuned empirically to maximize CLIP-based retrieval performance (Jun et al., 2023).

5. Sampling, Rendering, and Downstream Usage

Shap-E supports text-conditional and image-conditional sampling by mapping prompts (via CLIP) to initial noise and running diffusion in the latent space. The final latent is decoded into MLP weights, yielding a function $f_\theta$ for downstream usage:

• Mesh extraction: Query STF SDF on a $128^3$ grid, compute mesh via Marching Cubes, and texture color per vertex.
• NeRF rendering: Volumetric integration of density and color along camera rays for novel viewpoints.

Sampling produces 3D assets in ≈13 seconds on a V100 for text conditioning, and ≈1 minute for image conditioning (for batch inference) (Jun et al., 2023).

6. Empirical Performance and Comparisons

Evaluation uses CLIP R-Precision (ViT-B/32 and ViT-L/14) and CLIP score for image-to-text similarity. Shap-E surpasses Point-E in text-conditional R-Precision (37.8% vs 33.6% for ViT-B/32; 40.9% vs 35.5% for ViT-L/14), with lower sampling latency. Qualitatively, Shap-E captures finer geometric and topological details, especially in thin structures, and converges to high visual fidelity faster than Point-E. Failure modes include typical cross-modal difficulties: occluded parts, attribute binding, and counting. Text-conditional sampling remains an open challenge for multi-object and relational prompts (Jun et al., 2023).

7. Significance, Limitations, and Extensions via Shap-Editor

Shap-E’s compact $1024 \times 1024$ latent manifold supports both efficient generation and semantic latent space manipulation. The Shap-Editor framework extends Shap-E for one-shot, instruction-guided 3D editing by learning a feed-forward network $g(r^s, y) \to r^e$ that modifies latent codes under text prompt control. Shap-Editor employs score distillation sampling with dual 2D teacher networks (TI2I, T2I) and enables instance edits in ≈1 s—orders of magnitude faster than test-time optimization. The latent space demonstrates partial linearity, supporting edit chaining and strength control. Limitations include reliance on the expressive power of the latent space and the 2D teacher models, as well as scaling issues for open-vocabulary or out-of-distribution categories (Chen et al., 2023).

A plausible implication is that Shap-E marks a shift in 3D generative modeling: by amortizing the optimization of 3D fields and supporting fast generator/encoder/latent-edited workflows, it provides a unified substrate for research in conditional synthesis, interactive editing, and multimodal 3D understanding.