Pixal3D: Pixel-Aligned 3D Generation

• Pixal3D is a 3D generative model that creates assets by back-projecting multi-scale image features into a voxelized 3D space.
• It utilizes a camera coordinate framework to enforce explicit image-to-3D alignment, eliminating ambiguity typical of cross-attention methods.
• Empirical evaluations reveal notable improvements in IoU, PSNR, and SSIM, underscoring its superior reconstruction fidelity over prior approaches.

Pixal3D is a 3D generative modeling paradigm that enables high-fidelity 3D asset creation from images through explicit pixel-to-3D alignment. In contrast to prior 3D-native generators that operate in canonical object space and condition on images via attention, Pixal3D generates 3D content in the camera coordinate frame using a pixel back-projection mechanism. This architecture explicitly lifts multi-scale image features into a voxelized 3D feature volume, directly conditioning generation on aligned image cues and thereby addressing core ambiguity in the 2D–3D correspondence problem (Li et al., 11 May 2026).

1. Architectural Motivation and Design Principles

Conventional 3D-native generators such as TRELLIS and Direct3D-S2 utilize cross-attention to inject image features into a canonical-space latent, relying on the network to implicitly discover pixel-to-3D correspondences. This approach introduces inherent ambiguity, as 3D tokens must independently infer which image regions to attend, leading to misalignment and degraded image fidelity—especially problematic for repetitive or fine-scale structures.

Pixal3D departs from this paradigm by generating directly in the view-centric (camera) space. The method carves out a fixed voxelized cube (typical resolution: $64^3$) in the camera frustum, where every voxel corresponds to a known 3D location defined relative to the input camera. The core innovation is the back-projection of multi-scale image features into this 3D voxel grid, creating a pixel-aligned feature volume that the generative backbone (latent diffusion VAE, dense-to-sparse DiTs, SDF decoder) utilizes. Critically, the noise latent is fused with the pixel-aligned features by direct addition, ensuring one-to-one spatial correspondence and eliminating the need for cross-attention in the conditioning process.

2. Pixel-Aligned Back-Projection Conditioning

The pixel-aligned conditioning mechanism centers on geometric back-projection. For each voxel center $\mathbf{x}_{ijk}$ in the 3D grid, the corresponding image location is computed using the camera intrinsics $K$: $(u_{ijk}, v_{ijk}, 1)^T = K\,\mathbf{x}_{ijk}.$ For each scale $i$ in a set of $L$ multi-scale image feature maps $F_i$, features are bilinearly sampled at $(u_{ijk}, v_{ijk})$ and averaged: $$V(\mathbf{x}_{ijk}) = \frac{1}{L}\sum_{i=1}^L F_i(\pi(\mathbf{x}_{ijk})),$$ with $\pi$ denoting 3D-to-2D projection. During iterative diffusion, this volumetric feature $\mathbf{x}_{ijk}$0 is summed with the evolving noise latent $\mathbf{x}_{ijk}$1 at every voxel, enforcing strict alignment of every spatial index.

Multi-scale feature maps (e.g., DINOv2 patch tokens, upsampled high-res features) allow the construction of dense, high-frequency–aware 3D volumes, critical for capturing intricate pixel-level cues. The result is a 3D generation pipeline that maintains unambiguous correspondences between input image pixels and the generated geometry.

3. 3D Feature Volume Processing and Refinement

Pixal3D discretizes the scene into a normalized camera-space cube, initially voxelized at low resolution. The generative process typically initiates with a "dense" latent diffusion stage that estimates a global occupancy grid. Subsequent masking identifies occupied voxels, which are then processed at higher resolution using "sparse" transformer-based DiTs.

At each diffusion timestep $\mathbf{x}_{ijk}$2, the noise latent $\mathbf{x}_{ijk}$3 (with $\mathbf{x}_{ijk}$4 voxels) is refined by a 3D transformer $\mathbf{x}_{ijk}$5, described by: $\mathbf{x}_{ijk}$6 where all attention operations are replaced by direct addition of the pixel-aligned feature $\mathbf{x}_{ijk}$7. This update preserves explicit voxel-to-voxel mapping throughout the denoising process, ensuring that image information is neither diffused nor scrambled.

4. Volumetric Rendering and Training Losses

Upon reaching the final latent state $\mathbf{x}_{ijk}$8, a VAE-based decoder transforms voxel values into signed distance field (SDF) densities $\mathbf{x}_{ijk}$9 and RGB color $K$0. Rendered color along a ray $K$1 utilizes volumetric transmittance: $K$2 The primary objective is an $K$3 pixel reconstruction loss comparing the input image $K$4 and rendered $K$5: $K$6 Additional regularizers, such as TV-smoothness or normal consistency, may be applied to the SDF; latent diffusion losses from DiT training are preserved.

5. Multi-View Aggregation and Scene-Level Generation

For multi-view generation, Pixal3D aggregates pixel-aligned feature volumes across available images with known camera intrinsics and extrinsics $K$7: $K$8 This additive fusion propagates consistent cues from multiple viewpoints, increasing certainty of the reconstruction where observations overlap and leveraging learned priors to hallucinate unobserved regions. Multi-view fusion straightforwardly extends the single-image paradigm without additional architectural modifications.

The scene synthesis pipeline integrates segmentation (with tools such as SAM$K$9), 2D inpainting (Qwen-image-edit), object-centered pixel-aligned 3D generation (Pixal3D), and global alignment via dense depth estimation (MoGe). Object scales $(u_{ijk}, v_{ijk}, 1)^T = K\,\mathbf{x}_{ijk}.$0 and translations $(u_{ijk}, v_{ijk}, 1)^T = K\,\mathbf{x}_{ijk}.$1 are estimated to align each object's Pixal3D mesh with the MoGe-derived depth map, solving

$(u_{ijk}, v_{ijk}, 1)^T = K\,\mathbf{x}_{ijk}.$2

for coherent, scene-level 3D assembly.

6. Empirical Evaluation and Comparative Metrics

Pixal3D demonstrates significant improvements in reconstruction fidelity over prior state-of-the-art. On the Toys4K dataset for single-view input, Pixal3D achieves Intersection over Union (IoU) of 93.57% (compared to 79.48%), PSNR of 24.21 dB (vs. 20.98 dB), SSIM of 0.897 (vs. 0.883), LPIPS of 0.108 (vs. 0.204), and mean normal error of 16.63° (vs. 25.00°), relative to leading baselines including TRELLIS, TripoSG, Hunyuan3D-2.1, and Direct3D-S2.

In-the-wild evaluations reflect robust alignment with image-text similarity metrics (e.g., CLIP-based ULIP2: 45.04 vs. 44.76; Uni3D: 42.11 vs. 41.09), and user studies yield mean fidelity/quality ratings of 4.91/4.74 out of 5.

For multi-view generation on Toys4K, increasing the number of views (2/4/6) systematically improves Chamfer distance (5.27/4.73/4.16), Earth Mover's Distance (1.13/1.05/1.00), and F-Score (64.94/67.85/69.04), with multi-view fusion yielding up to 4× improvement over previous approaches such as TRELLIS.

Ablation studies indicate that omitting high-resolution feature upsampling results in loss of fine details, and reverting to cross-attention–based conditioning degrades convergence and fidelity.

7. Implications, Limitations, and Future Directions

Pixal3D establishes a new paradigm in 3D generation by providing deterministic, pixel-aligned correspondence between input images and synthesized geometry. The explicit geometric prior resolves core ambiguity in previous methods and enables scalable, high-fidelity, and multi-view 3D asset and scene synthesis. The scene synthesis framework further supports object-separated 3D scene assembly with strong geometric coherence.

Limitations include dependence on accurate camera parameters for pixel back-projection and scalability to unconstrained scenes with complex occlusion and lighting. Extending robust pixel-aligned modeling to broader, less-structured domains remains a central research avenue.

Further, Pixal3D's methodology suggests promising opportunities for integrating dense optical flow, learned depth priors, or neural correspondence fields to relax calibration requirements and enhance generalization.

Pixal3D defines a reproducible, modular approach for pixel-aligned generative 3D modeling, offering a substantial uplift in image-to-3D fidelity and establishing a foundation for further advances in learned geometric reasoning and high-fidelity asset creation (Li et al., 11 May 2026).