Leveling3D: Closed-Loop 3D Reconstruction

• Leveling3D is a feed-forward 3D reconstruction pipeline that fuses 3D Gaussian Splatting with diffusion-based repair to address sparse-view artifacts.
• It employs a closed-loop process where initial coarse renders are refined in masked regions and reinjected to enhance geometric consistency and view synthesis.
• Empirical evaluations show significant improvements in PSNR, SSIM, and depth metrics, underlining its effectiveness in overcoming underconstrained 3D reconstruction challenges.

Searching arXiv for the Leveling3D paper and closely related work. Leveling3D is a feed-forward 3D reconstruction pipeline that couples 3D Gaussian Splatting with geometrically consistent diffusion-based generation in order to improve sparse-view novel-view synthesis and depth estimation (Huang et al., 17 Mar 2026). It targets the underconstrained regime in which feed-forward 3DGS produces incomplete geometry, texture corruption, and blank or low-opacity regions in extrapolated views. Its defining mechanism is a closed-loop reconstruction–generation–reconstruction process: an initial feed-forward 3DGS model renders coarse extrapolated views, a geometry-aware diffusion module repairs only the artifact regions of those views, and the repaired views are then reused as inputs to a second feed-forward 3DGS pass, thereby “leveling up” the reconstructed representation (Huang et al., 17 Mar 2026).

1. Problem formulation and research setting

Leveling3D is formulated for sparse, unposed RGB observations of a static scene. The input is a small set of images $\{I_i\}_{i=1}^N$, and the objective is to obtain a 3D Gaussian representation that supports high-quality novel-view synthesis, including extrapolated views whose frusta lie largely outside the convex hull of the inputs. The paper situates this problem within feed-forward 3D reconstruction, where models such as AnySplat, VolSplat, and PF3Plat infer a 3D representation in a single forward pass rather than through slow per-scene optimization (Huang et al., 17 Mar 2026).

The central failure mode is geometric underconstraint. Under sparse input, feed-forward 3DGS leaves unobserved regions weakly determined, so extrapolated renders exhibit incomplete geometry, stretched or smeared texture, and blank regions where few or no Gaussians contribute. Previous “fixing” methods based on diffusion are described as largely geometry-agnostic: they improve local appearance but do not explicitly incorporate strong 3D priors, often remain multi-view inconsistent, and typically do not feed repaired views back into the underlying 3D representation. Leveling3D is therefore designed not as a pure 2D post-processor but as a system in which image generation is conditioned on multi-view geometry priors and then reinjected into feed-forward 3D reconstruction (Huang et al., 17 Mar 2026).

This places Leveling3D within a broader class of pipelines that use generated or repaired views to support 3D inference. A related but methodologically different example is “2L3: Lifting Imperfect Generated 2D Images into Accurate 3D,” which also treats imperfect generated views as supervision for 3D reconstruction, but does so through a NeuS-style SDF representation, intrinsic decomposition guidance, monocular normal priors, and semantic view augmentation rather than through 3D Gaussian Splatting and diffusion inpainting (Chen et al., 2024).

2. System architecture and reconstruction loop

The pipeline uses AnySplat as the feed-forward 3DGS backbone. Given $N$ input images, the AnySplat network $f_\theta$ predicts a set of Gaussians and camera poses,

$$f_{\boldsymbol{\theta}}:\{I_i\}_{i=1}^N \mapsto \Bigl\{ \boldsymbol{\mu}, \sigma, \boldsymbol{r}, \boldsymbol{s}, \boldsymbol{c} \Bigr\}^{\mathcal G} \cup \{p_i\}_{i=1}^N,$$

where each Gaussian is parameterized by a center $\boldsymbol{\mu}\in\mathbb{R}^3$, scaling $\boldsymbol{s}\in\mathbb{R}^3$, rotation $\boldsymbol{r}$, opacity $\sigma$, and color $\boldsymbol{c}\in\mathbb{R}^3$ (Huang et al., 17 Mar 2026).

AnySplat is described as comprising a geometry encoder distilled from VGGT together with three heads: a camera head $F_C$, a depth head $N$0, and a Gaussian head $N$1. The geometry encoder aggregates multi-view tokens and produces geometry tokens $N$2. The camera head predicts camera poses $N$3, the depth head predicts depth maps $N$4 and confidence maps $N$5, and the Gaussian head regresses the Gaussian primitives using depth, pose, and aggregated geometry tokens. Rendering follows standard front-to-back alpha compositing. For a pixel $N$6, the rendered color is

$N$7

and the accumulated opacity is

$N$8

These opacity maps are later used to locate underconstrained regions (Huang et al., 17 Mar 2026).

The complete Leveling3D loop has five stages. First, AnySplat reconstructs an initial Gaussian set $N$9, estimates camera poses, and exposes the geometry tokens $f_\theta$0. Second, the system samples extrapolated camera poses $f_\theta$1 and renders coarse extrapolated images $f_\theta$2 together with their opacity maps. Third, artifact masks are derived from the opacity maps and then refined. Fourth, Stable Diffusion 2 inpainting, conditioned through the geometry-aware leveling adapter, repairs only the masked regions and produces refined views $f_\theta$3. Fifth, the refined views are concatenated with the original inputs,

$f_\theta$4

and AnySplat is run a second time to produce the final refined 3DGS $f_\theta$5 (Huang et al., 17 Mar 2026).

The importance of this loop is architectural rather than merely cosmetic. The generated images are not an endpoint; they become additional pseudo-observations that densify the geometric constraints seen by the feed-forward reconstructor. The method does this without per-scene optimization of the Gaussian representation.

3. Geometry-aware leveling adapter

The geometry-aware leveling adapter is the mechanism that aligns the diffusion model with feed-forward 3D geometry. It is described as a lightweight technique, conceptually similar to a T2I-Adapter, but augmented with cross-attention to geometry tokens extracted from AnySplat. Stable Diffusion 2 remains frozen, as does AnySplat; only the adapter and its cross-attention pathway are trained (Huang et al., 17 Mar 2026).

The adapter operates on a geometry-aware conditioned image $f_\theta$6. Let $f_\theta$7 be a reference input image. After patchification, the reference image yields tokens

$f_\theta$8

and the geometry encoder provides geometry tokens $f_\theta$9. Cross-attention is then formed as

$$f_{\boldsymbol{\theta}}:\{I_i\}_{i=1}^N \mapsto \Bigl\{ \boldsymbol{\mu}, \sigma, \boldsymbol{r}, \boldsymbol{s}, \boldsymbol{c} \Bigr\}^{\mathcal G} \cup \{p_i\}_{i=1}^N,$$0

$$f_{\boldsymbol{\theta}}:\{I_i\}_{i=1}^N \mapsto \Bigl\{ \boldsymbol{\mu}, \sigma, \boldsymbol{r}, \boldsymbol{s}, \boldsymbol{c} \Bigr\}^{\mathcal G} \cup \{p_i\}_{i=1}^N,$$1

After unpatching the attention result to an image-like tensor $$f_{\boldsymbol{\theta}}:\{I_i\}_{i=1}^N \mapsto \Bigl\{ \boldsymbol{\mu}, \sigma, \boldsymbol{r}, \boldsymbol{s}, \boldsymbol{c} \Bigr\}^{\mathcal G} \cup \{p_i\}_{i=1}^N,$$2, the final condition becomes

$$f_{\boldsymbol{\theta}}:\{I_i\}_{i=1}^N \mapsto \Bigl\{ \boldsymbol{\mu}, \sigma, \boldsymbol{r}, \boldsymbol{s}, \boldsymbol{c} \Bigr\}^{\mathcal G} \cup \{p_i\}_{i=1}^N,$$3

The resulting multi-scale features are injected into the UNet encoder by

$$f_{\boldsymbol{\theta}}:\{I_i\}_{i=1}^N \mapsto \Bigl\{ \boldsymbol{\mu}, \sigma, \boldsymbol{r}, \boldsymbol{s}, \boldsymbol{c} \Bigr\}^{\mathcal G} \cup \{p_i\}_{i=1}^N,$$4

with $$f_{\boldsymbol{\theta}}:\{I_i\}_{i=1}^N \mapsto \Bigl\{ \boldsymbol{\mu}, \sigma, \boldsymbol{r}, \boldsymbol{s}, \boldsymbol{c} \Bigr\}^{\mathcal G} \cup \{p_i\}_{i=1}^N,$$5 (Huang et al., 17 Mar 2026).

Architecturally, the adapter begins with an $$f_{\boldsymbol{\theta}}:\{I_i\}_{i=1}^N \mapsto \Bigl\{ \boldsymbol{\mu}, \sigma, \boldsymbol{r}, \boldsymbol{s}, \boldsymbol{c} \Bigr\}^{\mathcal G} \cup \{p_i\}_{i=1}^N,$$6 convolution with stride $$f_{\boldsymbol{\theta}}:\{I_i\}_{i=1}^N \mapsto \Bigl\{ \boldsymbol{\mu}, \sigma, \boldsymbol{r}, \boldsymbol{s}, \boldsymbol{c} \Bigr\}^{\mathcal G} \cup \{p_i\}_{i=1}^N,$$7 that maps a $$f_{\boldsymbol{\theta}}:\{I_i\}_{i=1}^N \mapsto \Bigl\{ \boldsymbol{\mu}, \sigma, \boldsymbol{r}, \boldsymbol{s}, \boldsymbol{c} \Bigr\}^{\mathcal G} \cup \{p_i\}_{i=1}^N,$$8 condition image to a $$f_{\boldsymbol{\theta}}:\{I_i\}_{i=1}^N \mapsto \Bigl\{ \boldsymbol{\mu}, \sigma, \boldsymbol{r}, \boldsymbol{s}, \boldsymbol{c} \Bigr\}^{\mathcal G} \cup \{p_i\}_{i=1}^N,$$9 feature map. It then applies four feature extraction blocks, each composed of a convolution layer and two ConvNeXt blocks, followed by three average-pooling downsampling blocks that produce a multi-scale feature pyramid. The adapter contains approximately $\boldsymbol{\mu}\in\mathbb{R}^3$0 million trainable parameters. Reported inference time is about $\boldsymbol{\mu}\in\mathbb{R}^3$1 seconds per image, which is described as comparable to T2I-Adapter and faster than ControlNet (Huang et al., 17 Mar 2026).

This design gives the diffusion model direct access to geometry priors that originate in the reconstruction backbone rather than in externally estimated depth or normals. A plausible implication is that Leveling3D treats geometry tokens as a scene-level control signal: multiple extrapolated views share the same underlying geometric representation, so the adapter can promote cross-view consistency without explicit multi-view optimization inside the diffusion model.

4. Artifact-region generation, palette filtering, and training protocol

Artifact localization begins with the 3DGS opacity map. A coarse mask is formed by thresholding:

$\boldsymbol{\mu}\in\mathbb{R}^3$2

Pixels with low opacity correspond to areas where geometry is missing or weakly supported. Stable Diffusion 2 is then run in inpainting mode so that only masked regions are synthesized while unmasked regions remain anchored to the coarse render. The UNet input latent is

$\boldsymbol{\mu}\in\mathbb{R}^3$3

combining the noisy latent, the VAE latent of the coarse image, and the refined mask (Huang et al., 17 Mar 2026).

Test-time masking refinement is performed with morphological operations. The coarse mask is first processed by morphological closing with a $\boldsymbol{\mu}\in\mathbb{R}^3$4 kernel and then dilated with a $\boldsymbol{\mu}\in\mathbb{R}^3$5 kernel. The paper reports that the $\boldsymbol{\mu}\in\mathbb{R}^3$6 setting yields the best trade-off: smaller dilation leaves residual artifacts at boundaries, whereas larger dilation invades already reliable regions and causes the diffusion model to overwrite them (Huang et al., 17 Mar 2026).

Training uses a palette filtering strategy that selects masked regions according to the distribution of ground-truth intensities inside the mask. For masked pixels $\boldsymbol{\mu}\in\mathbb{R}^3$7 with intensity $\boldsymbol{\mu}\in\mathbb{R}^3$8, let $\boldsymbol{\mu}\in\mathbb{R}^3$9 and $\boldsymbol{s}\in\mathbb{R}^3$0 be the mean and standard deviation over the masked region. The palette score is

$\boldsymbol{s}\in\mathbb{R}^3$1

A sample is retained if $\boldsymbol{s}\in\mathbb{R}^3$2 with $\boldsymbol{s}\in\mathbb{R}^3$3. The paper states that this stabilizes training and improves diverse distributed generation by filtering the noisy-clean pairs used for adapter tuning (Huang et al., 17 Mar 2026).

The adapter is trained with perceptual supervision. Over a sequence of extrapolated views sharing the same geometry tokens, the loss is

$\boldsymbol{s}\in\mathbb{R}^3$4

where $\boldsymbol{s}\in\mathbb{R}^3$5 is the ground-truth novel view and $\boldsymbol{s}\in\mathbb{R}^3$6 is the refined output. The internal LPIPS form is

$\boldsymbol{s}\in\mathbb{R}^3$7

The paper notes that there is no explicit depth or normal loss on the diffusion side and no explicit regularization of Gaussian parameters introduced by Leveling3D itself (Huang et al., 17 Mar 2026).

The reported training setup uses roughly $\boldsymbol{s}\in\mathbb{R}^3$8k noisy-clean image pairs generated from DL3DV-10K and ScanNet++, $\boldsymbol{s}\in\mathbb{R}^3$9 epochs, AdamW with learning rate $\boldsymbol{r}$0 and weight decay $\boldsymbol{r}$1, eight A6000 GPUs, and batch size $\boldsymbol{r}$2 per GPU. For each sequence, gradients are accumulated across extrapolated views sharing the same geometry tokens so that the adapter learns a multi-view-consistent control signal (Huang et al., 17 Mar 2026).

5. Empirical performance and ablation evidence

The method is evaluated on novel-view synthesis and depth estimation under sparse two-view input, with additional tests over wider view counts. The principal NVS benchmarks are MipNeRF360 and VRNeRF; depth is evaluated on TartanAir and ScanNet. Reported metrics include PSNR, SSIM, LPIPS, and FID for rendering quality, and Abs Rel, RMSE, $\boldsymbol{r}$3, RMSE$\boldsymbol{r}$4, and Met3R for depth and multi-view consistency (Huang et al., 17 Mar 2026).

Method	MipNeRF360	VRNeRF
AnySplat	15.60 / 0.318 / 0.314	15.89 / 0.532 / 0.347
Difix3D+	15.68 / 0.334 / 0.312	16.22 / 0.540 / 0.363
GSFix3D	14.96 / 0.302 / 0.354	13.47 / 0.412 / 0.530
GSFixer	15.69 / 0.332 / 0.348	15.86 / 0.554 / 0.356
ViewExtrapolator	14.85 / 0.324 / 0.606	16.72 / 0.591 / 0.518
Leveling3D	16.76 / 0.352 / 0.306	18.35 / 0.610 / 0.316

In the table, each benchmark column is reported as PSNR / SSIM / LPIPS. Relative to AnySplat, Leveling3D improves PSNR by $\boldsymbol{r}$5 on MipNeRF360 and by $\boldsymbol{r}$6 on VRNeRF. The paper also reports favorable speed relative to video-based baselines: Leveling3D requires $\boldsymbol{r}$7 seconds per image, compared with $\boldsymbol{r}$8 seconds for GSFixer and $\boldsymbol{r}$9 seconds for ViewExtrapolator (Huang et al., 17 Mar 2026).

Method	TartanAir	ScanNet
AnySplat	0.915 / 20.75 / 0.311 / 0.0683	0.372 / 58.83 / 0.708 / 0.0472
Difix3D+	0.893 / 20.30 / 0.321 / 0.0625	0.286 / 0.358* / 0.787 / 0.0452
GSFix3D	0.905 / 20.49 / 0.315 / 0.0675	0.290 / 33.40 / 0.791 / 0.0453
GSFixer	0.892 / 20.37 / 0.315 / 0.0665	0.296 / 31.30 / 0.784 / 0.0462
ViewExtrapolator	3.791 / 139.39 / 0.102 / 0.0633	3.794 / 120.67 / 0.323 / 0.0646
Leveling3D	0.853 / 19.54 / 0.351 / 0.0614	0.252 / 24.68 / 0.826 / 0.0376

Here each dataset column is reported as Abs Rel / RMSE / $\sigma$0 / Met3R. The gains are not only photometric. Leveling3D also achieves the lowest Met3R on both datasets, which the paper interprets as evidence that geometry-consistent extrapolated views improve the reconstructed 3D structure itself rather than merely the rendered appearance (Huang et al., 17 Mar 2026).

Ablation studies identify three critical components. First, geometry-token fusion improves consistency and perceptual quality beyond naive diffusion and generic control mechanisms. On ScanNet, Leveling3D reaches PSNR $\sigma$1, SSIM $\sigma$2, LPIPS $\sigma$3, and FID $\sigma$4, outperforming AnySplat, naive SD2, ControlNet, and T2I-Adapter. Second, palette filtering reduces collapse and improves LPIPS and FID. Third, mask refinement produces the largest SSIM increase and cleaner transition boundaries. A further robustness study on ScanNet with $\sigma$5 to $\sigma$6 input views shows Leveling3D consistently outperforming AnySplat, from PSNR $\sigma$7 versus $\sigma$8 at two views to $\sigma$9 versus $\boldsymbol{c}\in\mathbb{R}^3$0 at ten views (Huang et al., 17 Mar 2026).

6. Limitations, interpretation, and relation to adjacent lines of work

The paper explicitly reports several limitations. If the initial AnySplat reconstruction fails severely, Leveling3D cannot fully recover, because its diffusion stage is restricted to the masked artifact regions and does not repair errors outside those regions. Small objects or subtle structures near mask boundaries may be over-extended or terminated incorrectly. Extremely aggressive extrapolation can still induce hallucinations. The method is also designed for static scenes and does not address temporal coherence for dynamic content (Huang et al., 17 Mar 2026).

Conceptually, the main contribution is the integration of feed-forward 3D reconstruction and geometrically consistent generation into a single closed loop. Earlier work on hierarchical 3D processing focused on multi-level spatial representations, such as coarse and fine voxel hierarchies processed by coupled 3D CNNs (Ghadai et al., 2018). Other work used “level” in the procedural sense, treating 3D environments as objects of controllable generation in Minecraft or database-driven multi-floor game design (Jiang et al., 2022, Xu et al., 25 Aug 2025). Leveling3D is different in scope: its “leveling” refers neither to hierarchical voxel discretization nor to game-level synthesis, but to improving a feed-forward 3D representation by iterating between extrapolated rendering, geometry-aware image repair, and renewed reconstruction. This suggests a terminological overlap across subfields rather than a shared technical program.

Within 2D-to-3D lifting, Leveling3D also differs from neural-field approaches that use generated or imperfect views as supervision. “2L3” addresses inconsistent lighting, misaligned geometry, and sparse viewpoints by combining intrinsic decomposition, transient monocular priors, and semantic view augmentation in a NeuS-style SDF pipeline (Chen et al., 2024). Leveling3D addresses an adjacent problem—artifact-prone feed-forward 3DGS under sparse views—but solves it through geometry-token-conditioned inpainting and a second-pass feed-forward 3DGS reconstruction. A plausible implication is that both methods instantiate the same broader research direction: generated or repaired views become intermediate supervisory objects that improve the final 3D representation, but the representation class and conditioning machinery remain fundamentally different.

In that sense, Leveling3D is best understood as a reconstruction-centric refinement framework for sparse-view 3D Gaussian Splatting. Its distinctive claim is not simply better inpainted images, but a mechanism by which those images become new geometric evidence for feed-forward 3D reconstruction, yielding state-of-the-art results on public datasets for both novel-view synthesis and depth estimation (Huang et al., 17 Mar 2026).