LagerNVS: Feed-forward Novel View Synthesis

• LagerNVS is a fully neural, feed-forward novel view synthesis method that integrates explicit 3D reconstruction with 2D rendering using a transformer-based encoder.
• It employs a two-stage encoder–decoder architecture where a pre-trained 3D-supervised transformer extracts spatially organized latent tokens for efficient view generation.
• The system extends to generative synthesis via a diffusion-based decoder, achieving real-time performance and state-of-the-art metrics on diverse multi-view datasets.

LagerNVS is a fully neural, real-time, feed-forward approach to Novel View Synthesis (NVS) that bridges explicit 3D reconstruction and purely 2D rendering methods. By leveraging strong 3D inductive biases through an encoder initialized from a 3D-supervised transformer backbone, LagerNVS achieves state-of-the-art deterministic feed-forward NVS across diverse datasets, strong real-time render speeds, and extensibility to generative synthesis using a diffusion-based decoder (Szymanowicz et al., 20 Mar 2026).

1. Network Architecture

LagerNVS employs a two-stage encoder–decoder architecture. The encoder, $f_{\text{enc}}$, accepts $V$ source RGB images $\{I_1,\dots,I_V\}$ and corresponding (optional) camera parameters $\{g_1,\dots,g_V\}$ to produce a set of latent feature tokens:

$$z = f_\text{enc}(I_1,g_1,\dots,I_V,g_V) \in \mathbb{R}^{\sum_i P_i \times C},$$

where $z_i \in \mathbb{R}^{P_i \times C}$ is derived from the transformer backbone of a pre-trained 3D reconstruction model (VGGT). These tokens encode 3D-aware, spatially organized scene information.

The decoder, $f_{\text{dec}}$, utilizes these aggregated latents $z$ in conjunction with a novel target camera $g$ to render the synthesized image:

$$\hat{I} = f_\text{dec}(z \,;\, g) \in \mathbb{R}^{3 \times H \times W}.$$

The decoder encodes the target view as a dense Plücker-ray map $R \in \mathbb{R}^{6 \times H \times W}$, pools it into camera tokens $s$, and deploys a lightweight Vision Transformer (ViT) wherein these camera tokens $s$ attend (cross- and/or full-attention) to the scene tokens $z$. The final tokens are reshaped to yield $H \times W$ RGB pixels, delivering the synthesized novel view in a fully feed-forward pass.

2. 3D Inductive Bias and Encoder Pre-training

The encoder integrates 3D inductive bias by initialization from a pre-trained VGGT (Volumetric Geometry-guided Transformer) network. VGGT is trained under explicit 3D supervision on multi-view data, where each image $I_i$ is paired with ground-truth depth $D_i$ and, when available, point clouds or multi-view correspondences. The pre-training objective combines depth and point cloud losses:

$$L_\text{3D} = \sum_{i=1}^V \| D_i^{\text{pred}} - D_i^{\text{gt}} \|_2^2 + \lambda_\text{pc} \sum_{p \in \mathcal{P}} \| \mathbf{x}_p^{\text{pred}} - \mathbf{x}_p^{\text{gt}} \|^2,$$

where $\lambda_\text{pc}$ balances the point-cloud term. After pre-training, VGGT’s explicit output heads are discarded, and only its transformer backbone is retained to extract $z_i$ tokens, exploiting the latent 3D structural awareness imparted by the pre-training phase.

3. Fine-tuning Regimen and Loss Design

The end-to-end LagerNVS model is fine-tuned on image tuples $$(I_\text{src},g_\text{src}; I_\text{tgt},g_\text{tgt})$$ drawn from a heterogeneous mixture of 13 multi-view datasets, such as RealEstate10k, DL3DV, and WildRGBD. Unlike freezing the encoder backbone, which degrades view synthesis and geometric cues, fine-tuning all weights is found essential for optimal performance.

The composite training loss consists of an $L_2$ photometric term and a perceptual feature consistency term:

$$L_\text{photo}(I_\text{render}, I_\text{target}) = \lambda_2 \| I_\text{render} - I_\text{target} \|_2^2 + \lambda_p \| \phi(I_\text{render}) - \phi(I_\text{target}) \|_2^2,$$

where $\phi(\cdot)$ is a frozen VGG-based perceptual embedding and $\lambda_2, \lambda_p$ are scalars. The optimizer is AdamW, with cosine learning-rate decay, linear warmup, QK-norm attention, gradient clipping, and checkpointing. Augmentation involves randomizing the number of source views (1–10), dropping camera tokens, and varying input aspect ratios for robustness.

4. Real-time NVS Performance and Benchmarks

After 250,000 iterations of training at $512 \times 512$ resolution (batch size 512), LagerNVS achieves state-of-the-art deterministic feed-forward NVS metrics. On the RealEstate10k dataset with two source views, it attains:

• PSNR = 31.39 dB
• SSIM = 0.928
• LPIPS = 0.078

This performance surpasses LVSM by +1.7 dB and reconstruction-based 3D Gaussian splatting methods by larger margins (e.g., on DL3DV, DepthSplat yields PSNR = 23.47 dB versus LagerNVS's 29.45 dB). Rendering is real time: On a single NVIDIA H100 GPU, LagerNVS synthesizes $512 \times 512$ images at 30+ FPS with up to nine source images, enabling interactive or production-grade NVS workflows.

5. Diffusion-based Generative Extension

Addressing ambiguity in occluded or unobserved regions, LagerNVS supports an extension to generative synthesis by adapting the decoder as a conditional denoising-diffusion model (“LagerNVS–Diffusion”). The encoder is frozen, while the decoder is fine-tuned using:

• An additional input channel for the noisy target image $x_t$,
• Timestep-conditioned AdaLN-zero layers,
• The standard diffusion denoising objective:

$$L_\text{diff} = \mathbb{E}_{t, x_0, \epsilon} \| \epsilon - \epsilon_\theta(x_t, t, z) \|_2^2,$$

where $$x_t = \sqrt{\alpha_t} x_0 + \sqrt{1 - \alpha_t} \epsilon$$.

After 60,000 additional iterations, the model accurately hallucinates texture and geometry in unobserved regions and enables free-form scene extrapolation, generalizing the system’s output in ambiguous settings where deterministic models “regress to the mean.”

6. Generalization, Application Range, and Limitations

LagerNVS demonstrates robust generalization, performing novel view synthesis on:

• Internet photos with unknown camera poses (utilizing null-vector camera tokens)
• Ego-centric robot-mounted imagery
• 360° panoramas
• Non-square aspect ratios
• Single-view synthesis with narrow viewpoint change

The primary deterministic limitation is a tendency toward mean regression in ambiguous settings, resulting in blurry outputs for occluded regions, which motivates the diffusion-based extension for generative completion.

Planned improvements include integrating a rendering-aware pre-training head to preserve appearance cues in VGGT features, enhancing spatial–temporal consistency for dynamic content, and scaling the diffusion decoder for higher resolutions and reduced inference overhead. LagerNVS closes the method gap between explicit 3D reconstruction and 2D feed-forward methods, providing high-fidelity, real-time, and generative novel view synthesis (Szymanowicz et al., 20 Mar 2026).