Sparse auto-regressive modeling for scene generation from multi-view images

Abstract: Generating complete 3D scenes from sparse, unconstrained views is a fundamental challenge in 3D vision which requires reasoning beyond observed content while remaining computationally tractable. Existing feed-forward reconstruction methods are inherently limited to content visible in the input images, while 3D generative modeling is hindered by the high computational cost of dense volumetric representations and the scarcity of large-scale 3D supervision. We introduce SPAR3S, a sparse voxel-aligned 3D latent generative model for conditional scene completion without requiring ground-truth 3D data for supervision. Our key insight is to formulate 3D scene generation in a structured, compact, voxel-aligned 3D latent space where only occupied voxels are represented. We learn this sparse latent space directly from multi-view images using photometric supervision via differentiable 3D Gaussian Splatting. Given a partial set of observed voxels encoded from sparse input views, scene completion reduces to predicting the missing latent tokens and their spatial support within the voxel grid. To this end, we train a masked autoregressive transformer that jointly models voxel occupancy and latent token values, enabling efficient and spatially consistent generation of unseen regions. We demonstrate the effectiveness of our method on synthetic indoor scenes, achieving higher novel-view quality than prior work. We further validate its generalization on RealEstate10k, highlighting its applicability to real-world data.

• The paper presents SPAR3S, a method for completing 3D scenes from sparse multi-view images using a sparse voxel-aligned latent representation and autoregressive generation.
• Compared to baseline methods, SPAR3S achieves significant improvements in novel-view synthesis with a highest PSNR (15.18 for 3DFront and 16.72 for RealEstate10K.) and a lowest FID (59: 3DFront and 41: RealEstate10K) and LPIPScores.
• Accurate occupancy prediction and representation granularity are key technical constraints of SPAR3S.

Problem formulation and contribution

SPAR3S addresses conditional 3D scene completion from sparse, unconstrained multi-view images. The central difficulty is that feed-forward 3D reconstruction methods generally regress only geometry and appearance visible within the input camera frustums, whereas image-space generative methods require prescribed novel camera poses and may produce geometrically inconsistent views. SPAR3S instead generates missing scene content directly in a structured 3D representation, allowing the completed scene to be rendered from arbitrary viewpoints.

The paper’s main claim is that sparse voxel-aligned latent representations make autoregressive 3D generation tractable without requiring explicit ground-truth 3D supervision. The method combines three components: a multi-view encoder that constructs a sparse 3D latent representation, a 3DGS decoder trained through differentiable rendering, and an occupancy-aware masked autoregressive transformer that completes unobserved latent voxels. The complete system is trained from RGB multi-view observations, using estimated or known cameras and pointmaps, rather than directly optimized against ground-truth meshes, point clouds, or Gaussian scenes (2609.03931).

This formulation separates reconstruction from completion. The encoder establishes a partial, observation-supported scene representation, while the generative model estimates plausible geometry and appearance outside the observed regions. Consequently, SPAR3S is not deterministic reconstruction in the conventional sense: when the observations underdetermine the scene, the latent predictor samples from a conditional distribution over completions.

Sparse voxel-aligned latent representation

The first stage learns a 3D latent space through a variational encoder-decoder. Given a set of RGB images, the encoder receives image tokens together with pointmaps estimated from the views. The union of pointmaps is transformed into a normalized volume, and only voxels containing observed scene content are instantiated. This sparse initialization avoids processing the full cubic voxel grid.

The image tokens and voxel-aligned 3D tokens are updated through bidirectional cross-attention. Geometry-guided attention incorporates pointmap-derived correspondences between image patches and voxels, supplementing learned attention logits with explicit spatial association. After the cross-attention stack, the image tokens are discarded and the 3D tokens are hierarchically downsampled into a compact bottleneck representation.

The decoder upsamples the latent representation and predicts a set of 3D Gaussian primitives for each occupied voxel. Occupancy is predicted before token values during upsampling, allowing empty locations to be removed from subsequent computation. The decoder therefore maps the latent tensor to a 3DGS scene rather than to a conventional dense voxel field.

The representation-learning pipeline is summarized below.

Figure 1: Multi-view images and pointmaps are fused into sparse voxel-aligned 3D tokens, which are decoded into Gaussian splats and trained through differentiable rendering.

The encoder-decoder is optimized by rendering the predicted Gaussian scene into the input camera views and minimizing photometric reconstruction error, with a low-weight KL regularizer on the latent distribution. This is an important methodological choice: the 3D representation is learned as a consequence of enforcing a single scene-level Gaussian bottleneck across multiple views, rather than from direct 3D labels. The resulting representation is therefore shaped by view reconstruction and the inductive structure of voxel alignment, not by explicit geometric supervision.

The use of sparse occupancy is essential for computational feasibility. Dense volumetric processing scales cubically with resolution, while SPAR3S materializes only occupied or potentially occupied voxels. The hierarchical architecture further reduces the number of latent tokens through spatial compression. During training, false-positive occupancy noise is injected into upsampling masks so that the decoder is robust to imperfect occupancy predictions at inference. The ablations identify approximately 25% false-positive injection as a robust operating point for the tested compression ratios.

The latent bottleneck is deliberately narrow but not arbitrarily so. A latent dimension below 32 substantially degrades reconstruction, whereas stronger KL regularization than 0.1 also causes a significant decline. The selected configuration uses a 32-dimensional latent vector and a KL coefficient of 0.1, representing the tightest bottleneck that preserves stable reconstruction quality according to the reported sweep.

Occupancy-aware masked autoregressive completion

The generative stage receives two types of latent representation. A dense set of views is encoded into a target latent scene, while a sparse set of views is encoded into conditioning tokens. Training then masks target voxels and asks the transformer to predict both whether each voxel is occupied and, conditional on occupancy, what latent feature it should contain.

This joint prediction is the key difference from raster-order autoregressive modeling. A conventional autoregressive model would process every position in a predefined dense volume, including empty voxels, and would impose a fixed sequential ordering. SPAR3S instead treats occupancy as a variable to be generated and operates only on the sparse spatial support.

Figure 2: SPAR3S jointly predicts voxel occupancy and latent content for masked spatial tokens, avoiding dense raster-order processing over the full volume.

The model, denoted $H_\psi$, consists of bidirectional attention blocks followed by lightweight voxel-wise occupancy and denoising heads. The occupancy head is trained with a masked BCE/focal objective to address the imbalance between occupied and empty locations. The latent-value head uses a token-wise diffusion objective: target latent tokens are corrupted according to a DDPM noise schedule, and the network predicts the injected noise conditioned on visible tokens and spatial context. A second diffusion head refines conditioning tokens using information from the completed representation.

The diffusion parameterization is not merely an alternative output layer. Removing it and replacing it with a linear latent regression head reduces 3DFront performance from 15.18 to 13.90 PSNR, from 0.62 to 0.53 SSIM, and from 0.50 to 0.54 LPIPS. This result supports the paper’s claim that stochastic token prediction is useful for ambiguous unobserved regions, where a unimodal regression target can average incompatible scene hypotheses.

Training targets need not cover the entire scene. Dense target views may still leave parts of the volume unobserved. SPAR3S explicitly separates occupied, observed-empty, and unobserved positions, forces genuinely unobserved positions to remain in the prediction set, and masks them from the loss because their latent values are unavailable. This is a practical mechanism for learning from incomplete multi-view coverage without treating unobserved content as empty.

At inference, the model completes the scene iteratively. The observed tokens serve as seeds, and candidate voxels are partitioned into spatial groups. SPAR3S uses a sparse $k$-nearest-neighbor graph and breadth-first search to assign depth levels from the observed seeds. Predictions are then generated region by region, with newly accepted occupied tokens added to the context for subsequent iterations. This region-growing ordering imposes a spatially local conditional structure while retaining the flexibility of masked autoregression.

The inference process is illustrated by the complete generative pipeline.

Figure 3: Sparse observed-view latents condition iterative prediction of missing occupancy and latent tokens before decoding into a completed 3D Gaussian scene.

The ablations show that the spatial ordering and occupancy mechanisms are consequential. Removing 3D relative positional encoding reduces PSNR to 13.74 and SSIM to 0.49. Replacing BFS ordering with a random autoregressive order reduces PSNR from 15.18 to 14.81. Removing occupancy-reference refinement reduces PSNR to 14.48 and SSIM to 0.56. An oracle using ground-truth occupancy reaches 17.90 PSNR, 0.68 SSIM, and 0.45 LPIPS, demonstrating that occupancy errors remain a major source of final reconstruction error.

Experimental setting

The evaluation focuses on novel-view synthesis from randomly sampled conditioning views. This protocol is intentionally more difficult than overlap-heavy view-synthesis splits because conditioning cameras may have little or no mutual overlap with one another, and target viewpoints may lie far outside their frustums.

The synthetic 3DFront benchmark contains more than 10,000 furnished indoor scenes with diverse room layouts and appearances. Ground-truth cameras and rendered depth maps are available in this setting, allowing pointmaps to be initialized directly. RealEstate10k provides more than 80,000 real-estate video clips. For this benchmark, camera poses, intrinsics, and pointmaps are estimated with MASt3R-SfM, so the evaluation also tests robustness to real-world geometry and camera estimation. The RealEstate10k camera trajectories are smoother and more densely sampled than the random 3DFront configurations, making it a less severe test of unconstrained extrapolation.

The encoder-decoder and autoregressive model are trained separately. The paper reports approximately four days of training for each model in the main experimental description, with supplementary details specifying large iteration budgets and single-GPU training. The reported experiments use $224 \times 224$ images and evaluate PSNR, SSIM, LPIPS, FID, and KID on rendered novel views.

Quantitative performance

With two conditioning views, SPAR3S provides the strongest results among the evaluated methods on both datasets. The most relevant comparison is against generative 3D methods rather than deterministic reconstruction systems, since the latter are structurally unable to infer extensive unseen content.

Dataset	Method	FID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
3DFront	MVSplat360	111	9.72	0.35	0.70
3DFront	LatentSplat	180	13.92	0.33	0.61
3DFront	SPAR3S	59	15.18	0.62	0.50
RealEstate10k	MVSplat360	66	15.40	0.49	0.40
RealEstate10k	LatentSplat	53	16.09	0.48	0.39
RealEstate10k	SPAR3S	41	16.72	0.57	0.36

On 3DFront, SPAR3S improves over LatentSplat by 1.26 dB PSNR and 0.29 SSIM, while reducing LPIPS from 0.61 to 0.50 and FID from 180 to 59. Relative to MVSplat360, the gains are 5.46 dB PSNR and 0.27 SSIM. On RealEstate10k, the improvements are smaller in PSNR because the benchmark contains more overlapping camera trajectories, but SPAR3S still obtains the best results: 16.72 PSNR, 0.57 SSIM, 0.36 LPIPS, and 41 FID.

The results support a specific architectural conclusion rather than a general claim that 3D generation always dominates reconstruction. The advantage is concentrated in sparse, wide-baseline configurations where substantial scene content is outside the conditioning frustums. In observed or strongly overlapping regions, deterministic methods with geometric priors remain competitive, but they do not provide a mechanism for completing genuinely unobserved space.

Qualitative behavior and conditioning dependence

The qualitative comparisons show that SPAR3S can generate coherent content for target views with little overlap with the two conditioning images. Competing methods often reproduce visible regions but fail under large viewpoint extrapolation, producing missing structures, inconsistent geometry, or appearance artifacts.

Figure 4: From two widely separated conditioning images, SPAR3S produces more coherent extrapolated geometry and appearance than the compared reconstruction and generative baselines.

The reported behavior is consistent with the representation used by the model. Since all predicted content is placed in a common voxel-aligned 3D latent space before rendering, multiple novel views share the same completed scene. Image-space diffusion approaches can generate visually plausible individual views, but without a shared 3D representation they may exhibit view-dependent inconsistencies. SPAR3S does not require target camera poses during generation; camera parameters are used for conditioning and evaluation, while completion occurs in scene coordinates.

Figure 5: Novel-view synthesis from two conditioning images on 3DFront and RealEstate10k, showing scene completion beyond the directly observed regions.

Increasing the number of conditioning views consistently improves reconstruction. The paper reports this trend for one, two, four, eight, and twelve views. More views reduce uncertainty in the latent scene and improve both image fidelity and perceptual similarity. The occupancy threshold also interacts with the amount of context. With few views, lower thresholds generally perform better because omitting occupied regions is more damaging than retaining some spurious voxels. With more views, occupancy estimates become more reliable and higher thresholds can be used without discarding as much valid geometry.

Figure 6: Additional conditioning views reduce ambiguity and improve rendered novel views while preserving plausible completions in unobserved regions.

This dependence exposes the probabilistic nature of the task. SPAR3S does not converge to one fixed scene independent of the observations; it conditions its completion distribution on the available views. Regions that remain ambiguous can therefore vary across samples, while regions constrained by additional evidence become more stable and detailed.

Limitations and open questions

The principal limitation is the dependence on occupancy prediction for sparsification. A false negative at the occupancy stage removes a voxel from subsequent processing, creating holes in walls, furniture, or other scene structures. The oracle occupancy experiment quantifies the importance of this failure mode: replacing predicted occupancies with ground truth raises performance from 15.18 to 17.90 PSNR and from 0.62 to 0.68 SSIM on the reported 3DFront experiment.

The voxel resolution and fixed Gaussian allocation per voxel also constrain local detail. The paper reports localized blur and attributes it to the resolution of the sparse voxel hierarchy and the number of splats allocated to each voxel. Increasing the number of splats improves PSNR but exhibits diminishing returns; the selected configuration uses $6^3$ splats per voxel. Increasing model capacity or dataset scale could mitigate these artifacts, but would also increase the computational cost of occupancy prediction, sparse attention, and autoregressive completion.

The claim of supervision without ground-truth 3D should also be interpreted precisely. SPAR3S does not require ground-truth meshes or Gaussian splats, but it depends on pointmaps and camera estimates. On 3DFront these are obtained from rendered depth and known cameras; on RealEstate10k they are estimated using MASt3R-SfM. Errors in those geometric inputs can therefore affect voxel initialization, geometry-guided attention, and the conditioning representation. The method’s robustness to substantially noisier pointmaps or inaccurate camera scales remains an open empirical question.

Finally, the evaluation is concentrated on indoor scenes and rendered novel-view quality. The paper establishes that the learned representation can support completion and rendering, but it does not fully characterize large-scale outdoor environments, dynamic scenes, long-range spatial extent, or semantic controllability of generated content. Whether the sparse latent and occupancy mechanisms remain computationally effective as scene extent and geometric complexity grow is left unresolved.

Conclusion

SPAR3S presents a coherent formulation of sparse-view 3D scene generation in which completion is performed directly in a sparse voxel-aligned latent space. Its encoder-decoder learns 3D Gaussian representations from photometric supervision, while an occupancy-aware masked autoregressive transformer predicts both the spatial support and latent content of unobserved regions. The combination of sparse computation, stochastic latent prediction, geometry-aware attention, and BFS-based region growing yields substantial improvements in two-view novel-view synthesis, particularly under wide-baseline and low-overlap conditions.

The strongest evidence is the reported 3DFront result of 15.18 PSNR, 0.62 SSIM, 0.50 LPIPS, and 59 FID, together with consistent gains on RealEstate10k. At the same time, the occupancy oracle gap and the reported voxel-resolution artifacts show that occupancy recall and representation granularity remain central technical constraints. The paper’s unresolved question is therefore specific: how far can native 3D autoregressive completion scale before the computational cost and error sensitivity of sparse occupancy modeling outweigh its geometric consistency advantages?