DiT³ Diffusion Models for 3D Generation

• The paper introduces DiT³, which replaces U-Net backbones with plain transformers to generate high-fidelity, diverse 3D shapes from voxelized point clouds.
• It details methodological innovations including voxelization, 3D patching, and windowed self-attention to encode spatial structure while reducing computational load.
• The approach incorporates Increment-Calibrated Caching (ICC) to accelerate sampling by reusing transformer activations, yielding faster inference with improved quality metrics.

Diffusion Transformers (DiTs) are a class of generative models that replace traditional U-Net backbones in diffusion probabilistic models (DPMs) with transformer architectures. DiT$^3$ refers to the extension of this paradigm to 3D data, where plain transformer models are leveraged to generate high-fidelity 3D shapes from noise, typically represented as voxelized point clouds. DiT-3D models incorporate novel strategies for representing spatial structure, reducing computational burden, and enabling efficient transfer from pre-trained 2D generative models. This approach has established new benchmarks in 3D generation, particularly in metrics assessing fidelity and diversity, and additionally motivates new methods for accelerating sampling, such as Increment-Calibrated Caching (ICC) with channel-aware low-rank corrections.

1. Foundation: Diffusion Transformers and the Move to 3D

Diffusion transformers (DiT) operate as denoisers within the diffusion modeling framework, where the core objective is to iteratively reconstruct clean data from progressively noised inputs using a learned model. The original DiT architecture demonstrated that a plain Vision Transformer (ViT)-style model, composed of stacked self-attention and feed-forward blocks, could replace U-Nets in 2D image generation with scalable and competitive results. Central to this design is the patch-based tokenization of data, combined with time and condition embeddings added to the transformer input sequence (Mo et al., 2023).

The DiT-3D framework extends this principle to unordered point clouds for 3D generation tasks. DiT-3D explores whether the same minimalistic transformer design, coupled with adaptations for 3D spatial structure, can surpass 3D-specific architectures, such as point-voxel CNNs and U-Net variants, traditionally used in DDPMs for 3D data.

2. Architectural Adaptations for 3D Generative Modeling

DiT-3D adapts the plain transformer architecture to handle 3D data through several key modifications:

• Voxelization and 3D Patching: Each input point cloud is voxelized into a tensor $v \in \mathbb{R}^{V \times V \times V \times 3}$, where $V$ indicates the spatial resolution and $3$ denotes coordinate channels. Non-overlapping cubic patches of size $p^3$ are extracted and projected to $D$-dimensional embeddings via small 3D convolutions.
• 3D Positional Embeddings: To encode spatial context, each patch receives a frequency-based sine-cosine positional encoding in 3D. For each coordinate $(x, y, z)$, positional encodings are computed as:

$$\mathrm{PE}_{(x, y, z), 2i} = \sin(\omega_i \cdot [x, y, z]), \quad \mathrm{PE}_{(x, y, z), 2i+1} = \cos(\omega_i \cdot [x, y, z])$$

These are added to patch embeddings, with further embeddings for class and time conditions.

• 3D Windowed Self-Attention: Direct application of global self-attention is computationally infeasible due to the cubic increase of tokens. DiT-3D partitions the input into non-overlapping 3D spatial windows (e.g., of size $R^3$) and applies local self-attention within each window, significantly reducing compute and memory to $\mathcal{O}(L^2 / R^3)$ where $v \in \mathbb{R}^{V \times V \times V \times 3}$0 is the number of tokens.
• Transformer Block and Prediction Head: Each block follows the ViT template—LayerNorm, (windowed) multi-head self-attention, residual connection, another LayerNorm, a two-layer MLP with GeLU, and a final residual. Following $v \in \mathbb{R}^{V \times V \times V \times 3}$1 such blocks, a linear projection and unpatchifying operation are used to reconstruct the 3D shape from token embeddings, followed by devoxelization to output coordinates (Mo et al., 2023).

3. Diffusion Process and Training Protocols

The generative process adheres to the standard DDPM formulation:

• Forward process: Data $v \in \mathbb{R}^{V \times V \times V \times 3}$2 is gradually corrupted into $v \in \mathbb{R}^{V \times V \times V \times 3}$3 through a series of Gaussian noise additions:

$v \in \mathbb{R}^{V \times V \times V \times 3}$4

where $v \in \mathbb{R}^{V \times V \times V \times 3}$5 are noise schedules and $v \in \mathbb{R}^{V \times V \times V \times 3}$6 can also be computed directly from $v \in \mathbb{R}^{V \times V \times V \times 3}$7.

• Reverse process: A transformer network $v \in \mathbb{R}^{V \times V \times V \times 3}$8 learns to predict the noise component at each timestep. The simplified loss is:

$v \in \mathbb{R}^{V \times V \times V \times 3}$9

where $V$0 denotes any conditioning information.

• Efficient fine-tuning: The paralleling of DiT-2D and DiT-3D architectures allows for direct transfer of weights pre-trained on large-scale 2D datasets (such as ImageNet) using the DiffFit protocol. Here, only a small set of scale, bias, and normalization parameters specific to 3D components are trained, reducing the number of learnable parameters from 32.8 MB (full model) to 0.09 MB for adaptation, while yielding measurable improvements in downstream 3D fidelity (Mo et al., 2023).

4. Accelerating DiT-based Generation: Increment-Calibrated Caching

The deployment of diffusion models is computationally intensive because of their iterative structure. Increment-Calibrated Caching (ICC) addresses this by exploiting temporal redundancies in DiT's transformer-layer activations during sampling (Chen et al., 9 May 2025). The two-stage method is as follows:

• Offline Calibration: For each transformer layer, the weight matrix is decomposed via low-rank Singular Value Decomposition (SVD). Channel-aware SVD (CA-SVD or CD-SVD) incorporates channel sensitivity by scaling inputs and outputs based on activation statistics before SVD, enhancing the accuracy and robustness of the correction, especially in the presence of outliers.
• Accelerated Sampling: During inference, full computation is performed only for a subset of timesteps/layers (determined by a fixed period $V$1). For other steps, previous activations are reused, and an efficient low-rank correction is applied to account for the difference in inputs. Specifically:

$V$2

where cached activations $V$3 and $V$4 are stored at regular intervals, and $V$5 are the channel-aware low-rank calibration parameters.

This process is entirely training-free and can be implemented post hoc on any pre-trained DiT.

5. Empirical Results and Benchmarking

DiT-3D Performance: On ShapeNet, DiT-3D achieves state-of-the-art results on 1-Neighborhood Nearest Accuracy (1-NNA) and Coverage (COV) metrics for key object classes. For example, on the Chair dataset, DiT-3D reports 1-NNA@CD=49.11 (vs. 53.70 for LION) and Coverage@CD=52.45 (vs. 48.94 for LION), marking significant improvements in both fidelity and diversity over baselines, including GAN-based, flow-based, and mesh-based models (Mo et al., 2023). Ablation studies confirm that each architectural adaptation—3D positional encoding, windowed attention, and direct denoising of voxels—contributes to final performance.

Acceleration via ICC: On DiT-XL/2 with 35-step DDIM sampling, ICC achieves a reduction in computation (MACs) of approximately 45%, resulting in sampling 1.8× faster. Quantitatively, this is accompanied by an improvement in Inception Score (IS) (from approximately 235 in the baseline to 251 with ICC) and negligible increase in FID (<0.06). The results also hold for text-to-image models such as PixArt-α, demonstrating that ICC consistently outperforms naïve caching and matches or exceeds quality achieved by simply reducing diffusion steps (Chen et al., 9 May 2025).

6. Computational Complexity and Scalability

The complexity of DiT-3D arises from both the high dimensionality of 3D voxel grids and the quadratic costs of transformer self-attention. Windowed attention mitigates this by restricting attention to local spatial regions, reducing training time from ~92 hours to ~42 hours in experiments. ICC further addresses inefficiency at inference by reducing the number of full forward passes, ensuring that extra memory for caching and calibration parameters remains minor relative to total model size (Chen et al., 9 May 2025, Mo et al., 2023).

Furthermore, DiT-3D is highly scalable in both resolution (through finer patches or larger voxel grids) and model size (from Small to XL). Pretrained models on 2D datasets can be efficiently adapted to novel 3D domains or object classes with minimal fine-tuning.

7. Limitations and Directions for Future Research

DiT-3D models have been validated primarily on voxelized point clouds. Extending to alternative 3D representations such as signed distance fields and meshes remains open. The voxel grid's inherent memory requirements may restrict the rendering of fine geometric detail at high resolutions; sparse or adaptive tokenizations, such as octrees, are plausible future pathways.

On the acceleration front, ICC employs a static caching schedule and uniform rank selection. Adaptive or learned schedules, as well as per-layer rank adaptation, could yield better trade-offs between compute and fidelity. Outlier residuals pose stability challenges for low-rank correction at small ranks, suggesting investigation into nonlinear or higher-order correction approaches. ICC is orthogonal to advancements in diffusion ODE samplers; integration may further reduce sample complexity (Chen et al., 9 May 2025).

In summary, DiT$V$6 diffusion models demonstrate that plain transformer backbones, when adapted to 3D spatial structure and combined with principled acceleration techniques such as ICC, provide a powerful and efficient solution for scalable generative modeling of 3D shapes, setting new quantitative standards and enabling new regimes of cross-modal transfer and deployment efficiency (Mo et al., 2023, Chen et al., 9 May 2025).