PointDiffusion: Diffusion-Based Scene Completion in the Point Cloud Domain

Abstract: Reconstructing dense 3D scenes from sparse LiDAR point clouds is a fundamental challenge in autonomous driving, where latent diffusion models offer a promising solution. However, existing approaches rely on object-level autoencoders that collapse into unstable global representations at outdoor scale and suffer from ground truth data corrupted by odometry drift that systematically degrades supervision quality. Furthermore, multi-step diffusion inference incurs prohibitive latency for real-time deployment. We propose a novel multi-token Gaussian VAE with cross-attention pooling for stable scene-scale LiDAR compression, combined with an anchor-based ICP ground truth refinement pipeline that eliminates drift-induced noise from training supervision. Together, these components enable a scaffold-free single-step diffusion completion model that achieves an approximately 16x reduction in squared Chamfer distance on SemanticKITTI seq. 08 (0.396 m^2 to 0.024 m^2), surpasses LiDiff and ScoreLiDAR by 17-19% and 10-11%, respectively, and operates at 25-143x lower inference latency. Our results demonstrate that data quality dominates model design in this regime and that multi-token latent spaces provide a stable first stage for latent diffusion-based scene completion.

• The paper introduces a multi-token Gaussian VAE that improves point cloud reconstruction by leveraging spatially specialized latent tokens.
• It refines ground truth using an anchor-based ICP pipeline, reducing squared Chamfer Distance 16-fold and eliminating common artifacts.
• The single-step diffusion completion network achieves state-of-the-art reconstruction quality with drastically lower computational latency.

Diffusion-Based Scene Completion for Large-Scale Point Clouds: An Analysis of PointDiffusion

Introduction

"PointDiffusion: Diffusion-Based Scene Completion in the Point Cloud Domain" (2606.16048) addresses the longstanding problem of reconstructing dense, unoccluded 3D geometry from inherently sparse LiDAR input, critical for autonomous navigation. The work tackles both architectural and data-centric bottlenecks impeding scene-scale deployment of generative models, focusing especially on scalable latent representations and ground truth fidelity. The approach integrates a multi-token Gaussian variational autoencoder (VAE), an anchor-based ICP ground truth refinement pipeline, and a scaffold-free, single-step diffusion completion network—collectively yielding substantial improvements in both reconstruction quality and computational efficiency compared to prior art.

Methodological Contributions

Multi-Token Gaussian VAE

The core architectural advance is a VAE that substitutes earlier global or discrete latent encodings with 32 continuous, spatially specialized Gaussian latent tokens. This is instantiated by a PointNet backbone for per-point feature extraction, followed by a learned cross-attention pooler generating latent tokens (Figure 1), each parameterizing a Gaussian posterior. Decoding is performed via a stack of transformer blocks that map these tokens to dense reconstructions of point clouds, effectively distributing and preserving spatial context that is lost in prior global-pooled or VQ representations.

Figure 1: VAE reconstruction quality on SemanticKITTI seq. 08, frame 500: ground truth, 32-token Gaussian VAE reconstruction, and overlay.

The empirical ablation demonstrates monotonic Chamfer Distance (CD) improvement as the number of latent tokens is increased, validating the superiority of distributed latent representations over single-vector, codebook-based, or object-scale approaches when scaled to complex outdoor scenes.

Drift-Reduced Ground Truth Refinement

Semantic scene completion benchmarks are widely hampered by ground truth aggregated using odometry-predicted poses, which induce systematic misalignments—most pronounced as duplicated surfaces and “thick walls.” The proposed anchor-based ICP pipeline compensates for this by aligning scan windows locally to reduce pose drift, discarding corrections exceeding a displacement threshold and applying statistical and radius-based outlier removal. The resulting ground truth (v2) is $50\times$ denser and removes prevalent systematic artifacts (Figure 2, Figure 3).

Figure 2: Full scene renderings, without (left) and with (right) ICP-based ground truth refinement—showing elimination of redundant surfaces.

Figure 3: Car detail, without (left) and with (right) ICP: post-ICP, boundaries are sharper and redundancy is removed.

The refinement step alone yields a 16-fold reduction in squared Chamfer Distance (from $0.396$ to $0.024$ m²) for the same model, unequivocally establishing that data and supervision quality dominate model architecture at the targeted scale.

Scaffold-Free Diffusion Completion

For the completion itself, a lightweight denoising network is conditioned on a frozen PTv3 (Point Transformer V3) encoder. A Diffusion Probabilistic Model (DDPM) is trained via $\epsilon$-prediction, but at inference, only a single forward pass is used—eschewing costly, many-step iterative generation typical in prior work. The single-step $x_0$ prediction regime leads to both lower computational latency and reduced output diversity, but achieves state-of-the-art fidelity in Chamfer metrics for practical scene completion (Figure 4).

Figure 4: Diffusion process visualization: clean target scene, partially noised input, near-pure noise, and the model’s single-step $x_0$ completion.

Experimental Evaluation

Quantitative Benchmarks

On SemanticKITTI sequence 08, the VAE achieves squared Chamfer Distance $0.120\pm0.026$ m², avoiding codebook collapse ($\sim$16 m²) seen with VQ-VAE approaches and maintaining competitive parameter and latency profiles (7.1M params, 1.6 ms/frame). When deployed as a denoising teacher for diffusion, the system obtains $0.024\pm0.005$ m² CD on ICP-refined ground truth (v2), compared to $0.396\pm0.090$ m² with standard GT.

In a strictly matched protocol using v2 GT, PointDiffusion's teacher delivers squared CD $0.396$0 m², outperforming state-of-the-art diffusion baselines LiDiff (3.41–3.50 m²) and ScoreLiDAR (3.15–3.19 m²) by 10–19%, at $0.396$1 to $0.396$2 lower inference latency (comparison in Table below).

Model (Scaffold-Free, v2 GT)	Squared CD (m²)	Relative Latency
LiDiff	3.41–3.50	Baseline
ScoreLiDAR	3.15–3.19	$0.396$3
PointDiffusion (Ours)	2.83	$0.396$4–$0.396$5

Ground Truth Impact

Refinement of ground truth, rather than network architecture, produces the largest absolute gains; the effect size dwarfs that of any tested model or hyperparameter intervention. This underscores that future work in this domain must factor data curation and alignment at least as heavily as network innovations.

Visual and Practical Assessment

Output inspection post-ICP shows qualitative gains matching numerical results: redundancy disappears, surfaces sharpen, and scene ambiguity is perceptibly reduced (Figures 4, 5). The runtime of 209 ms/frame at 20,000-point scale is conducive to real-time deployment, rivaled only by the fastest non-diffusion baselines.

Theoretical and Practical Implications

The results support several key theoretical and practical conclusions:

• Latent Design: Distributed, tokenized Gaussian latent representations are superior for large-scale point cloud VAE bottlenecks, mitigating the spatial collapse of pooling and the learning instability of codebook-based representations.
• Supervision Quality: For large geometric generative models, improvements in training set fidelity can yield gains unattainable by model family alone. ICP-based refinement should become standard practice for large-scale completion supervision.
• Diffusion Efficiency: Single-step $0.396$6 sampling is sufficient for high-quality geometric completion under high-quality ground truth, sharply reducing the inference-time burden and opening the door for deployment in high-throughput/low-latency scenarios.
• Diversity–Fidelity Tradeoff: The scaffold-free, single-step design restricts generative diversity—a requirement in certain use cases—but ensures stable, high-fidelity reconstruction for safety-critical applications.

Limitations and Future Directions

Current evaluations are restricted to urban, outdoor driving datasets (SemanticKITTI). The architecture’s capacity to generalize to indoor, aerial, or off-road environments is unproven. Integrating the multi-token VAE as a latent denoiser bottleneck remains an open avenue. The restriction to single-step denoising—while efficient—does not probe the generative space and may limit stochastic modeling of ambiguity; relaxing this constraint (e.g., via learned-step schedulers or hybrid iterative approaches) could restore diversity if needed.

Figure 5: Illustration of the GT fusion pipeline with anchor-based ICP for ground truth refinement.

Conclusion

This work demonstrates that high-fidelity, low-latency 3D scene completion from sparse LiDAR can be achieved primarily through supervision refinement and multi-token latent design, with the proposed system outperforming existing diffusion-based competitors in both quality and speed. Its robust and generalizable architectural principles—multi-token continuous representations, data-centric supervision, and efficient single-step denoising—will likely inform both future research on point cloud generative modeling and the deployment of real-time perception systems.