PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimation

Abstract: State-of-the-art single-image 3D reconstruction methods often rely on complex hybrid architectures and loss functions, or compress geometry into latent spaces in order to leverage pre-trained latent diffusion models. In this work, we show that such architectural overhead and intricate loss formulations are unnecessary. We introduce a minimalist pixel-space Diffusion Transformer, built on a plain ViT, that operates directly on raw 3D point map patches and is conditioned on image tokens from a pre-trained DINOv3. Unlike existing latent diffusion approaches, we train our diffusion backbone entirely from scratch, eliminating the need for point map tokenizers. Despite its simplicity, our approach surpasses complex latent-based diffusion models while remaining significantly simpler than hybrid alternatives. Notably, it produces sharper geometric structure and is more robust in highly ambiguous regions, such as transparent objects.

• The paper demonstrates that a ViT-based pixel-space diffusion model directly estimates dense 3D point maps without relying on VAEs or hybrid architectures.
• It employs end-to-end training with flow matching and x-prediction to achieve superior geometric fidelity and efficient, single-step inference.
• Empirical results show state-of-the-art boundary sharpness and depth accuracy, establishing its potential for robotics, AR/VR, and spatial reasoning applications.

Minimalist Pixel-Space Diffusion for Monocular Geometry: An Analysis of PointDiT

Problem Setting and Motivation

The estimation of dense 3D geometry from a single RGB image—monocular geometry estimation—is central to 3D scene understanding and vision-based spatial reasoning. Prior paradigms fall into two dominant categories: (1) deterministic regression, generally using hybrid architectures (e.g., ViT+CNN) to produce point/depth maps but with inherent oversmoothing due to averaging in the presence of geometric ambiguity, and (2) generative latent diffusion, which employs Variational Autoencoders (VAEs) to encode point maps or similar signals into a latent space for denoising diffusion, risking information loss and fundamentally limited reconstruction fidelity.

The PointDiT framework directly challenges these conventions by removing reliance on VAEs and hybrid architectures. Instead, it employs a minimalist diffusion model entirely in pixel space, demonstrating that these architectural complexities and auxiliary losses are unnecessary for high-fidelity monocular 3D reconstruction. This is accompanied by end-to-end training, eliminating staged pipelines and domain-specific tokenizers.

Model Architecture: Diffusion Transformer in Pixel Space

PointDiT reformulates point map estimation as probabilistic generative modeling using a Vision Transformer (ViT)-based diffusion model. The model consists of the following core components:

• Direct Point Map Modeling: The regression target is a dense $H \times W \times 3$ point map in camera coordinates, thus removing the need for camera intrinsic knowledge for 3D reconstruction.
• Pixel-Space Diffusion: Training operates directly on raw 3D point map patches using a flow-matching framework, avoiding VAE-based compression artifacts.
• Transformer Backbone: A plain ViT, with input constructed from concatenated "tokens" of patchified noisy point map and image priors.
• DINOv3 Conditioning: Conditioning of the diffusion model uses robust, frozen DINOv3 features for the RGB input; tokens from four DINOv3 layers are concatenated and aligned with point map patches.
• x-Prediction Training: The model predicts the clean data (point map) directly at every diffusion step, empirically shown crucial for 3D signal fidelity and superior to v-prediction (velocity prediction).

By fusing noise-interpolated point map patches with DINOv3-image tokens via concatenation before Transformer processing, the architecture bridges powerful pretrained visual priors with the image-conditioned 3D data manifold.

Training and Inference

PointDiT is trained end-to-end with synthetic datasets (SceneNet-RGBD, Hypersim, VKITTI2, and others) to obtain "pixel-perfect" point map ground truth, critically leveraging synthetic-to-real transfer to minimize label imperfections.

• Normalization: Point maps are normalized per sample to ensure stable diffusion paths regardless of scene scale.
• Noise Scheduling: Flow matching uses a logit-normal noise time schedule with explicit zero-initialization to remove train-test discrepancy.
• Loss Function: The dominant training signal is flow matching loss (velocity-based MSE), with an auxiliary relative point loss to emphasize recovery of local geometric details.

At inference, ODE integration with explicit Euler steps transforms pure noise to the final 3D point map. Notably, even single-step denoising yields near-optimal results with high robustness to noise realization; additional steps primarily sharpen boundaries.

Empirical Results

PointDiT achieves SOTA or near-SOTA performance on several real-world benchmarks (KITTI, DIODE, NYUv2, ETH3D, HAMMER, iBims-1, Booster):

• Boundary Sharpness (BF1): PointDiT sets a new high-water mark for geometric boundary metrics, substantially outperforming all baselines, including hybrid deterministic regressors and latent diffusion models.
• Absolute Accuracy: The largest model (PointDiT-H) demonstrates best-in-class depth accuracy (in relative error and threshold-based metrics) compared to methods like MoGe, GeometryCrafter, and UniDepthV2.
• Efficiency: Single-step inference time is nearly 20x lower than latent diffusion approaches, with competitive or superior accuracy.
• Robustness: The model’s output is invariant to noise realization at inference and supports deterministic, feedforward deployment.

Ablation studies further highlight:

• The criticality of x-prediction for 3D signals; using v-prediction leads to catastrophic collapse in geometric quality.
• The benefit of DINOv3 over linear or other task-specific embeddings for boundary recovery.
• Superiority of the generative paradigm (flow matching) over deterministic regression in handling ambiguous and transparent regions.

Theoretical and Practical Implications

PointDiT demonstrates that direct pixel-space diffusion is both tractable and advantageous for high-dimensional, structured signals such as 3D point maps. This empirical evidence challenges the dogma that 3D geometry demands hybrid or latent-space methods for computational feasibility or detail retention. The architectural simplicity (plain ViT, direct x-prediction objective) is shown not only to be sufficient but in fact superior for reconstructing both coarse and fine geometry.

Practically, this results in models that are:

• End-to-end trainable, removing the need for staged encoder/decoder pipelines
• Flexible in inference, supporting both single-step feed-forward and multi-step refinement
• More suitable for ambiguous and out-of-distribution inputs, such as transparent or thin structures
• Deployable at lower computational cost, enabling practical adoption in time-constrained systems (e.g., robotics, AR/VR).

Limitations and Future Directions

Current limitations include training at fixed resolutions (256/512), with room for improvement in outdoor scene generalization and multi-modal prediction. Richer training data (especially for outdoor domains), as well as extensions to mixed-resolution training and joint prediction of geometry and appearance, are highlighted as viable future directions. The architecture’s inherent flexibility suggests straightforward applicability to multi-view, multi-modality, or alternative 3D representations by minor architectural adjustment.

Conclusion

PointDiT exemplifies the power and sufficiency of minimalist, pixel-space diffusion Transformers for monocular geometry estimation. It removes the need for VAEs, staged training, and complex regularizations, setting new standards in both geometric fidelity and boundary sharpness. The paradigm shift proposed by PointDiT is likely to guide further methodological simplification in geometrically-aware generative modeling, with significant implications for 3D/4D generative models, spatial reasoning, and downstream AI tasks.

---

Reference:

"PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimation" (2607.02515)