SPAR3S 3D Scene Generator

• The topic covers 3D scene completion from sparse images using SPAR3S, a model that predicts previously unseen regions.
• Sparse voxel-aligned 3D latent representation captures essential scene features for autousplatable gener’actantly reconstructed 3D scenes are nutrimentation for a scene created from few images
• Autoregressive spatial ordering progressively completes the scene by filling in unobserved regions.
• Notable Applications: Includes occupancy awareness masked automizeARAGE-Acute approach computiprocessing-era selective regu nested processing implement Constraints in current research focus over planner used by dud.
• ,

SPAR3S is a sparse voxel-aligned, autoregressive 3D latent generative model for completing scenes from sparse, unconstrained multi-view images. The method learns a compact 3D latent representation from photometric supervision through differentiable 3D Gaussian Splatting (3DGS), then predicts unobserved scene regions with an occupancy-aware masked autoregressive transformer. Completed latent scenes are decoded into 3D Gaussian splats for arbitrary novel-view rendering. The name is introduced in “Sparse auto-regressive modeling for scene generation from multi-view images” (Lucas et al., 3 Sep 2026).

1. Problem formulation and motivation

SPAR3S addresses conditional 3D scene completion from a small set of RGB images, potentially with large baselines and limited overlap. Feed-forward reconstruction methods such as PixelSplat, MVSplat, DepthSplat, and related 3DGS regressors can reconstruct surfaces visible within the input camera frustums, but their predictions are fundamentally constrained by observed pixels and rays. Occluded and unseen regions have no direct observations, while deterministic regression produces a single reconstruction rather than a distribution of plausible completions.

Image- and video-diffusion methods can synthesize views outside the observed images, but they generally require target camera poses or assumptions about camera distributions. Independently generated views may exhibit geometric drift, flickering, and inconsistent scene content, and the resulting images do not necessarily define a persistent 3D representation that can be rendered from arbitrary future viewpoints.

Direct 3D generative modeling avoids some image-space inconsistencies but is computationally expensive when applied to dense volumetric representations. A dense volume with resolution $R$ along each dimension contains $O(R^3)$ voxels, most of which may be empty. Large-scale ground-truth 3D scene datasets are also scarce. SPAR3S therefore combines three design choices:

1. Sparse spatial representation: only occupied voxel positions are materialized.
2. Autoregressive completion: missing occupancy and latent content are generated in a structured 3D order.
3. Photometric supervision: the latent scene representation is learned from images through differentiable Gaussian rendering, without ground-truth meshes, point clouds, or optimized Gaussian scenes.

The resulting system is designed to preserve a persistent 3D structure while generating plausible content beyond the observed regions.

2. Sparse voxel-aligned 3D latent representation

Let $\mathcal I=\{I_i\}_i$ denote a set of RGB input images, and let $E_\theta$ be the scene encoder. The encoder maps the images to a sparse 3D latent representation:

$Z_{\mathrm{3D}}=E_\theta(\mathcal I).$

A decoder $D_\phi$ converts the latent representation into a set of 3D Gaussian splats:

$G=D_\phi(Z_{\mathrm{3D}}).$

Given camera parameters $C_i$, a differentiable renderer $\mathcal R$ produces reconstructed views:

$\hat I_i=\mathcal R(G,C_i).$

The encoder and decoder are trained so that all input views are reconstructed from one shared 3D representation. This bottleneck encourages the latent variables to encode scene-level geometry and appearance rather than independent image-specific features.

Voxelization

The scene is embedded in a normalized volume,

$O(R^3)$0

For each input image, a pointmap $O(R^3)$1 is estimated. The union of pointmaps is

$O(R^3)$2

A similarity or affine transformation $O(R^3)$3 maps the pointmaps into the canonical volume:

$O(R^3)$4

The transformed pointmaps are voxelized. Only voxels containing scene content are materialized. If

$O(R^3)$5

denotes the occupied voxel positions, each occupied position is associated with a 3D latent token. Empty voxels are excluded from sparse encoder computation.

For every occupied voxel, the decoder predicts a fixed number $O(R^3)$6 of Gaussian splats:

$O(R^3)$7

Each Gaussian contains parameters such as position, covariance or scale and rotation, opacity, and color or appearance features. The supplied paper description does not specify the complete Gaussian parameterization.

Hierarchical latent processing

The latent representation is hierarchical. Voxel tokens are progressively downsampled into a compact bottleneck and subsequently upsampled. During upsampling, a binary occupancy head determines which candidate voxels should be instantiated before their latent values are computed.

At inference, the upsampling mask is predicted:

$O(R^3)$8

During training, ground-truth occupancy is augmented with false-positive noise:

$O(R^3)$9

Approximately $\mathcal I=\{I_i\}_i$0 false positives were reported as a robust choice in ablation experiments. The selected bottleneck dimension is $\mathcal I=\{I_i\}_i$1, and the encoder applies group normalization with eight groups after latent sampling.

3. Image encoding and geometry-guided voxel alignment

The input images are patchified into 2D tokens, denoted by $\mathcal I=\{I_i\}_i$2. Estimated pointmaps initialize sparse 3D voxel queries, denoted by $\mathcal I=\{I_i\}_i$3.

A stack of $\mathcal I=\{I_i\}_i$4 bidirectional cross-attention blocks updates both modalities:

$\mathcal I=\{I_i\}_i$5

The 2D tokens are discarded after fusion, and the resulting 3D tokens are passed to the decoder.

Geometry-guided attention

Pointmap projections identify which image patches correspond to particular voxels. SPAR3S combines a learned attention score with a pointmap-derived bincount score:

$\mathcal I=\{I_i\}_i$6

where $\mathcal I=\{I_i\}_i$7 indexes a voxel, $\mathcal I=\{I_i\}_i$8 indexes an image patch, $\mathcal I=\{I_i\}_i$9 is softmax, and $E_\theta$0 controls the contribution of learned and geometry-guided attention.

This mechanism provides explicit geometric correspondence while retaining learned cross-view reasoning. Removing the bincount-based geometry guidance slows training and reduces performance, indicating that pointmap-to-patch alignment contributes to the learned 3D representation.

Photometric autoencoding

The decoder produces Gaussian splats that are rendered into the input camera views. The reconstruction loss is

$E_\theta$1

The latent variables are sampled with the reparameterization trick. The Kullback–Leibler regularizer constrains the probabilistic latent space and reflects the fact that multiple Gaussian configurations may produce nearly identical image projections.

No ground-truth 3D representation is required. Training uses images, pointmaps or cameras, and photometric reconstruction. The encoder operates in two modes:

$E_\theta$2

for dense-view target representations, and

$E_\theta$3

for sparse conditioning representations. Training uses variable numbers of input images.

4. Occupancy-aware masked autoregressive completion

SPAR3S separates the scene volume into observed, empty, and unobserved positions:

• $E_\theta$4: occupied and observed positions containing latent tokens;
• $E_\theta$5: positions observed to be empty;
• $E_\theta$6: unobserved positions whose state is unknown.

Thus,

$E_\theta$7

A random observed subset $E_\theta$8 is selected from target tokens, and the prediction set is

$E_\theta$9

The training mask is constrained so that

$Z_{\mathrm{3D}}=E_\theta(\mathcal I).$0

Genuinely unobserved regions are therefore always prediction targets, but their losses are masked because their true occupancy and latent values are unknown. This allows the model to learn scene completion from partially observed examples.

The sparse conditioning representation $Z_{\mathrm{3D}}=E_\theta(\mathcal I).$1 is supplied alongside visible target tokens. A masked autoregressive model $Z_{\mathrm{3D}}=E_\theta(\mathcal I).$2 predicts occupancy and latent content:

$Z_{\mathrm{3D}}=E_\theta(\mathcal I).$3

where $Z_{\mathrm{3D}}=E_\theta(\mathcal I).$4 denotes the voxel positions to be predicted.

Architecture

The autoregressive model contains two bidirectional attention blocks,

$Z_{\mathrm{3D}}=E_\theta(\mathcal I).$5

followed by voxel-wise occupancy and latent-token heads, denoted by $Z_{\mathrm{3D}}=E_\theta(\mathcal I).$6 and $Z_{\mathrm{3D}}=E_\theta(\mathcal I).$7.

The first block processes conditioning tokens and visible target tokens. The second receives aggregated context together with masked voxel embeddings. Masked positions are initialized with a learnable mask token.

Each attention block contains five self-attention layers with LayerNorm, multilayer perceptrons, and multi-head self-attention. The internal dimension is $Z_{\mathrm{3D}}=E_\theta(\mathcal I).$8. Separate projections and LayerNorms are used for conditioning or unmasked tokens and masked tokens.

SPAR3S uses both absolute and relative 3D positional information:

• sinusoidal absolute embeddings encode normalized voxel coordinates;
• three-dimensional rotary positional encoding uses voxel coordinates within self-attention.

The occupancy head is a three-layer MLP with GELU activations and one output logit. It is described in the main text with masked binary cross-entropy, while the implementation uses focal loss to address occupied-versus-empty imbalance.

The latent head is a lightweight denoising diffusion network with two layers of width $Z_{\mathrm{3D}}=E_\theta(\mathcal I).$9. Conditioning is injected through adaptive LayerNorm modulation.

Occupancy prediction

The occupancy objective is

$D_\phi$0

where $D_\phi$1 is the target occupancy and $D_\phi$2 is the predicted occupancy. Unknown positions in $D_\phi$3 are excluded from the occupancy loss.

Latent-token diffusion

Continuous latent values are generated with a token-wise denoising objective. Let $D_\phi$4 denote masked target latent tokens, $D_\phi$5 Gaussian corruption noise, $D_\phi$6 a diffusion timestep, and $D_\phi$7 the contextual features from the second transformer block. The diffusion loss is

$D_\phi$8

The denoiser predicts the injected noise, enabling sampling of plausible latent tokens conditioned on visible scene content. SNR weighting is used in the implementation to avoid overemphasizing highly noisy timesteps.

A second diffusion head refines conditioning tokens using decoded information. It is trained with the same diffusion objective, and a stop-gradient operation prevents this auxiliary task from destabilizing the primary model.

5. Autoregressive spatial ordering and inference

A conventional autoregressive model factorizes a sequence as

$D_\phi$9

SPAR3S generalizes this idea to groups of spatially related voxels. The volume is divided into disjoint subsets,

$G=D_\phi(Z_{\mathrm{3D}}).$0

and each group is predicted jointly conditioned on previously completed groups:

$G=D_\phi(Z_{\mathrm{3D}}).$1

Breadth-first region growing

SPAR3S constructs a sparse symmetric $G=D_\phi(Z_{\mathrm{3D}}).$2-nearest-neighbor graph over voxels. Observed voxels serve as seeds. Breadth-first search assigns each voxel a depth,

$G=D_\phi(Z_{\mathrm{3D}}).$3

Candidate voxels are grouped by BFS depth, so generation proceeds outward from observed regions. Nearby regions are completed before more distant regions, encouraging local spatial continuity.

Masked autoregression permits arbitrary ordering during training, but BFS region-growing order is used during inference because it improves spatial consistency. Relative positional encoding and coordinate-based rotary positional encoding prevent the model from relying solely on sequence position.

Completion procedure

Given sparse images $G=D_\phi(Z_{\mathrm{3D}}).$4, inference proceeds by:

1. estimating pointmaps and cameras;
2. encoding the sparse images;
3. identifying observed occupied voxels;
4. constructing the BFS region-growing order;
5. predicting occupancy and latent tokens for successive voxel groups;
6. retaining latent tokens whose occupancy exceeds a threshold;
7. adding accepted tokens to the conditioning set;
8. decoding the completed sparse latent grid into Gaussian splats;
9. rendering arbitrary novel views.

At iteration $G=D_\phi(Z_{\mathrm{3D}}).$5,

$G=D_\phi(Z_{\mathrm{3D}}).$6

If

$G=D_\phi(Z_{\mathrm{3D}}).$7

the latent token is accepted:

$G=D_\phi(Z_{\mathrm{3D}}).$8

Otherwise, the voxel is discarded as empty. The process continues until candidate regions have been exhausted.

The completed latent grid is decoded as

$G=D_\phi(Z_{\mathrm{3D}}).$9

and rendered from a novel camera $C_i$0 as

$C_i$1

Coarse-to-fine occupancy

SPAR3S optionally uses two occupancy stages. A coarse stage predicts occupancy densely in a compact latent volume using a high-recall threshold but does not predict latent values. A fine stage operates only on coarse-selected voxels, refines occupancy, and predicts latent tokens for occupied positions.

Random masking ratios between $C_i$2 and $C_i$3 reduce memory and computation during occupancy training. The efficiency claim is structural rather than a formal complexity bound: sparse occupied-token processing, latent compression, grouped autoregression, lightweight token-wise diffusion, and coarse-to-fine occupancy avoid dense full-resolution 3D generation.

6. Training, evaluation, and limitations

Training configuration

The scene encoder-decoder is trained with $C_i$4 images, four conditioning views, and twelve target views randomly sampled during training. The reported configuration uses approximately $C_i$5 million iterations, batch size $C_i$6, Adam optimization, cosine learning-rate decay from $C_i$7, and Xavier initialization. Training time is reported as approximately one week on one NVIDIA A100 in supplementary details.

The autoregressive latent model uses the pretrained encoder to supply conditioning and target latent tokens. It is trained for approximately $C_i$8 million iterations with batch size $C_i$9, Adam optimization, cosine decay from $\mathcal R$0, and latent dimension $32. The supplied manuscript gives inconsistent hardware and training-time descriptions, reporting both approximately four days on one NVIDIA A100 and training on one V100; these figures are therefore approximate.

Datasets

SPAR3S is evaluated on synthetic indoor scenes from 3DFront and real-estate imagery from RealEstate10k.

3DFront contains more than $\mathcal R$1 indoor scenes, including bedrooms, living rooms, dining rooms, and libraries. Cameras are randomly distributed, and subsets maximizing scene coverage are selected. Ground-truth camera poses are used, and pointmaps are obtained by backprojecting rendered depth maps. Evaluation includes wide-baseline and low-overlap configurations.

RealEstate10k contains more than $\mathcal R$2 real-estate video clips, predominantly depicting indoor environments. Camera poses, intrinsics, and pointmaps are estimated using MASt3R-SfM. Its smooth camera trajectories and substantial overlap make it easier than the wide-baseline synthetic setting, while testing generalization to real images and estimated geometry.

The principal metrics are PSNR, SSIM, LPIPS, FID, and KID. Higher PSNR and SSIM are preferred; lower LPIPS, FID, and KID are preferred.

Quantitative results

For two-view conditioning at $\mathcal R$3, the reported results are:

Method	3DFront FID	3DFront PSNR	3DFront SSIM	3DFront LPIPS	RealEstate10k FID	RealEstate10k PSNR	RealEstate10k SSIM	RealEstate10k LPIPS
3DGS	260	8.65	.14	.79	271	7.92	.12	.79
PixelSplat	165	9.07	.18	.67	155	13.73	.41	.50
DepthSplat	110	13.76	.49	.55	82	13.25	.46	.41
DiffusioNeRF	229	12.80	.19	.73	—	—	—	—
MVSplat360	111	9.72	.35	.70	66	15.40	.49	.40
LatentSplat	180	13.92	.33	.61	53	16.09	.48	.39
SPAR3S	59	15.18	.62	.50	41	16.72	.57	.36

The results indicate stronger performance in wide-baseline, low-overlap scene completion than the compared methods. Feed-forward methods such as PixelSplat and 3DGS remain limited in unobserved regions. DepthSplat handles viewpoint changes more effectively but remains primarily constrained to observed geometry. LatentSplat and MVSplat360 provide generative capabilities but are less robust under unconstrained camera layouts.

Ablation findings

The reported four-conditioning-view ablation on 3DFront is:

Variant	PSNR	SSIM	LPIPS
Full SPAR3S	15.18	.62	.50
Without diffusion head	13.90	.53	.54
Without 3D relative positional encoding	13.74	.49	.58
Without BFS ordering	14.81	.59	.53
Without occupancy refinement	14.48	.56	.56
Ground-truth occupancy oracle	17.90	.68	.45

The results attribute measurable contributions to diffusion-based latent prediction, 3D positional encoding, BFS ordering, and occupancy refinement. The ground-truth occupancy oracle demonstrates that occupancy errors remain a major bottleneck.

Increasing the number of conditioning views from $\mathcal R$4 to $\mathcal R$5 generally improves performance. Lower occupancy thresholds are more useful with sparse inputs because missing a scene region is more damaging than adding some spurious occupied voxels. Increasing the number of Gaussian splats per voxel initially improves reconstruction, with diminishing returns; the reported configuration uses $\mathcal R$6 splats per voxel. A latent dimension below $\mathcal R$7 significantly reduces quality, while a Kullback–Leibler coefficient stronger than $\mathcal R$8 degrades reconstruction. The selected configuration is

$\mathcal R$9

Limitations

SPAR3S depends on sufficiently accurate pointmaps and camera information. In RealEstate10k, these are estimated with MASt3R-SfM, so errors in structure-from-motion or pointmap alignment can affect voxelization and completion.

Occupancy prediction is a principal failure mode. If an occupied voxel inside a wall or object is incorrectly classified as empty, that region is removed from the sparse representation and can produce a visually salient hole. The ground-truth occupancy ablation confirms the importance of this problem.

Other limitations include:

• voxel resolution limits geometric detail;
• a fixed number of Gaussians per voxel can produce localized blur;
• generated content is plausible but not guaranteed to correspond to the true unseen scene;
• dense occupancy prediction remains computationally costly;
• incorrect pointmaps can corrupt geometry-guided attention and latent alignment;
• the method may struggle outside the training distribution;
• no formal asymptotic complexity analysis or exact inference-time benchmark is reported;
• complete Gaussian parameterization, all generative loss weights, voxel dimensions, and some implementation details are not specified in the supplied paper description.

SPAR3S is therefore best characterized as a native 3D generative completion framework in which sparse voxel-aligned latent tokens provide the interface between multi-view image evidence and probabilistic scene generation. Its principal distinction from deterministic sparse-view reconstruction is the explicit prediction of both spatial support and latent scene content beyond the observed regions, followed by decoding into a persistent 3D Gaussian representation suitable for arbitrary novel-view rendering.