LagerNVS: Latent Geometry for Fully Neural Real-time Novel View Synthesis

Abstract: Recent work has shown that neural networks can perform 3D tasks such as Novel View Synthesis (NVS) without explicit 3D reconstruction. Even so, we argue that strong 3D inductive biases are still helpful in the design of such networks. We show this point by introducing LagerNVS, an encoder-decoder neural network for NVS that builds on `3D-aware' latent features. The encoder is initialized from a 3D reconstruction network pre-trained using explicit 3D supervision. This is paired with a lightweight decoder, and trained end-to-end with photometric losses. LagerNVS achieves state-of-the-art deterministic feed-forward Novel View Synthesis (including 31.4 PSNR on Re10k), with and without known cameras, renders in real time, generalizes to in-the-wild data, and can be paired with a diffusion decoder for generative extrapolation.

• The paper introduces a highway encoder-decoder that uses pre-trained 3D-aware features to achieve deterministic, real-time novel view synthesis.
• It achieves over 30 FPS at 512×512 resolution, outperforming state-of-the-art methods in PSNR, SSIM, and LPIPS metrics.
• The method generalizes to unconstrained inputs, supporting diverse views including egocentric, 360°, and unposed images, with potential for diffusion-based rendering.

Latent Geometry for Fast, Generalizable Neural Novel View Synthesis

Introduction and Motivation

Novel View Synthesis (NVS) aims to generate new images of a scene from previously observed views. The canonical pipeline optimizes an explicit 3D representation (e.g., NeRF or Gaussian Splatting) before rendering, but this is computationally inefficient and data-hungry, especially under sparse, unconstrained input. Recent advances have shown that deep networks can regress novel views directly, bypassing explicit 3D reconstruction. However, such feed-forward methods largely forgo strong 3D inductive biases, which can be critical for geometry-aware rendering and generalization.

This work introduces LagerNVS, an encoder-decoder architecture that leverages pre-trained 3D-aware features for real-time, fully neural NVS across unconstrained settings. Crucially, the encoder is initialized using a backbone trained for 3D reconstruction with explicit geometric supervision, yielding latent features with strong but implicit 3D bias. This architecture achieves deterministic, feed-forward, single-GPU NVS at $>$30 FPS at $512 \times 512$ resolution and outperforms existing approaches including LVSM, 3DGS variants, and both with and without access to camera intrinsics.

Figure 1: Method overview—inputs are any number of images (and optionally cameras), processed into 3D-aware latent features that enable efficient decoder rendering of novel views.

Architecture: Encoder-Decoder with Implicit 3D Bias

LagerNVS employs a highway encoder-decoder structure: the encoder is built on VGGT—a ViT-based multi-view 3D reconstruction model pre-trained with explicit 3D supervision. Rather than using explicit outputs (e.g., depths or reconstructed geometry), the last-layer tokens of the transformer backbone (excluding "camera tokens") are extracted and projected to modulate the decoder.

The decoder, a transformer block, tokenizes the target camera via dense Plücker ray maps, allowing it to be conditionally traversed from arbitrary viewpoints. Camera tokens can be optionally omitted at inference. The combination of dense information flow (highway) and an expressive 3D-aware latent space distinguishes LagerNVS from bottlenecked and decoder-only baselines.

Figure 2: Architectural comparison—highway encoder-decoder (left) maximizes per-view token flow; bottleneck encoder-decoder (middle) compresses all views into fixed tokens; decoder-only (right) requires re-evaluation for every novel view.

Generalization Across Scenarios

A significant contribution is the ability to generalize to scenes and conditions far outside standard NVS datasets: egocentric, 360°, non-square, unposed images, and in-the-wild internet photos. This is enabled by multi-dataset pre-training and the flexibility of the network structure to handle arbitrary numbers of input images (with or without cameras).

Figure 3: Generalization ability—rendering from input modalities such as internet photos, robot egocentric views, 360° images, and non-canonical aspect ratios.

Single-view NVS is also supported under restricted camera motion, a scenario poorly handled by most explicit 3D pipelines.

Figure 4: Single-view NVS is supported for small viewpoint changes using LagerNVS.

Empirical Performance and Comparisons

LagerNVS rigorously surpasses prior latent-rendering SoTA architectures (notably LVSM) in both accuracy and qualitative geometry estimation. With batch size 512 on RealEstate10k, LagerNVS delivers a +1.7 dB PSNR margin over LVSM and exhibits superior image similarity (SSIM, LPIPS). Qualitative results show robustness in regions with poor 2D texture correspondence, benefiting from 3D monocular cues during rendering.

Figure 5: Comparison to LVSM—LagerNVS outperforms in difficult 2D matching regions and leverages monocular depth cues.

Importantly, LagerNVS also surpasses feed-forward 3DGS-based methods (e.g., DepthSplat, AnySplat, FLARE) in both scenarios with and without known camera poses. The method exhibits clear improvements in reconstructing thin structures, occlusions, and reflective surfaces where 3DGS alignment is challenging.

Figure 6: Against feed-forward 3DGS methods, LagerNVS better reconstructs thin, reflective, and occluded scene elements.

Ablation Studies and Analysis

Architectural ablation studies validate several factors:

• 3D-aware latent initialization from a pre-trained reconstruction model is essential. Random or 2D-pre-trained initializations perform markedly worse.
• End-to-end encoder fine-tuning is required. Freezing the encoder or partial fine-tuning leads to diminished rendering quality.
• The highway encoder-decoder avoids information bottlenecks imposed by fixed token compressions, resulting in higher-fidelity renderings without significant computational overhead during inference.

  Figure 7: Ablation results—expressive architecture, 3D pre-training, and comprehensive fine-tuning are all critical for optimal performance.

Generative Extension via Diffusion

The deterministic nature of regression architectures leads to averaging artifacts when rendering highly ambiguous or unobserved regions. By reusing the LagerNVS decoder in a denoising diffusion pipeline (with minimal architectural modification), the model can hallucinate plausible completions for occluded or extrapolated views, which is not achievable with vanilla regression NVS networks.

Figure 8: The decoder, after fine-tuning with a diffusion objective, can generate plausible content (hallucination) for unseen scene regions.

Occlusion handling is also evaluated: LagerNVS can often produce reasonable completions for simple occlusions and retains reasonable output even in challenging cases.

Figure 9: Occlusion handling performance—realistic completions for moderately occluded targets, degraded gracefully as ambiguity increases.

Practical and Theoretical Implications

Practical Impact

• Real-time performance ($>$30 FPS at $512\times512$ resolution) with up to 9 input views, on a single H100 GPU, without custom rendering code, explicit 3D representation, or JIT compilation.
• Scalable to unordered, uncalibrated, and internet-scale multi-view datasets.
• Fully neural, with unified inference for both posed and unposed camera scenarios.

Theoretical Insights

• Implicit 3D inductive bias in feature extractors—obtained via pre-training on explicit 3D reconstruction—regularizes novel-view rendering, improving geometric consistency even without explicit geometry in the inference loop.
• Architectural choice (highway vs bottleneck vs decoder-only) profoundly affects information flow and renderer quality, especially under limited decoder capacity budgets.
• Pre-training on mixtures of datasets is essential for wide generalization in NVS, analogous to advances in large-scale vision-LLMs.

Future Prospects

LagerNVS indicates a promising path toward fully neural, real-time NVS by leveraging implicit geometric knowledge. Future work may entail:

• Integrating latent diffusion objectives to unify the generative and regressive paradigms for open-world, physically plausible rendering under ambiguity.
• Scaling highway encoder-decoders for more complex, possibly dynamic 3D scenes.
• Synthesizing explicit 3D representations as latent byproducts for tasks beyond view synthesis, such as geometry-aware manipulation and embodied perception.

Conclusion

LagerNVS demonstrates that strong implicit 3D geometric inductive bias—incorporated via pre-trained 3D reconstruction features and a highway encoder-decoder architecture—substantially advances the regime of real-time, generalizable novel view synthesis. The method sets a new deterministic SoTA for feed-forward NVS on major datasets, generalizes robustly to diverse settings, and provides a foundation for future hybrid generative-reconstructive rendering paradigms.

---

Refer to (2603.20176) for implementation, code, checkpoints, and detailed empirical analysis.