Sparse-Input Novel View Synthesis

• Sparse-input novel view synthesis is a technique that generates photorealistic and geometrically consistent views from minimal images, overcoming severe 3D ambiguity challenges.
• It leverages advanced architectures such as data-centric Transformers, explicit 3D regularization, and pretrained diffusion models to infer scene geometry without camera poses.
• The methods balance model generalization with per-instance fidelity by employing self-supervised losses, epipolar constraints, and plug-in generative priors for robust image synthesis.

Sparse-input novel view synthesis (NVS) refers to the generation of photorealistic and geometrically consistent images from novel viewpoints, given only a handful (as few as two to five) of observed images—often without access to camera pose, depth, or other 3D supervisory signals. This problem represents an extreme challenge due to the severe under-determination of 3D structure from incomplete 2D information. Recent advancements address limitations of classical multi-view 3D representations (e.g., NeRF, 3D Gaussian Splatting) by leveraging architectures, optimization tactics, and learning regimes suitable for sparse, even unposed, data, and increasingly by leveraging large-scale data or powerful pretrained models.

1. Problem Formulation and Challenges

Sparse-input NVS considers as input a set $$\mathcal I = \{ I_i \in \mathbb{R}^{H\times W\times 3} \}_{i=1}^N$$ of $N$ (typically $2\leq N \leq 5$) unposed images of a scene. The task is to learn a mapping

$f_\theta : \{I_i\}_{i=1}^N \longmapsto \hat I_t$

producing a photorealistic novel view $\hat I_t$ at a desired, often unobserved, target viewpoint, for which neither pose nor geometry is known. In the strictly unposed setting, methods must infer or learn scene geometry, occlusion order, and view consistency directly from raw images, with no access to explicit camera or 3D scene parameters (Wang et al., 11 Jun 2025).

Challenges include:

• Extreme information scarcity and geometric ambiguity.
• Lack of geometric supervision (poses, depth, or 3D reconstruction may be unavailable or unreliable).
• Tendency of conventional 3D representations to overfit or produce artifacts ("holes," "floaters," or degenerate geometry).
• The need for models to reason about implicit 3D structure from raw 2D observations.
• Balancing model generalization (across many scenes) with per-instance fidelity.

2. Architectural Paradigms for Sparse-Input NVS

Sparse-input NVS approaches fall broadly into the following architectural categories:

1. Data-Centric Transformer and Latent Embedding Models: Remove explicit 3D bias, relying on massively overparameterized Transformer architectures and large-scale data to learn implicit 3D awareness directly from images (Wang et al., 11 Jun 2025). In UP-LVSM, images are encoded via DINOv2, aggregated by a multi-layer Transformer, and rendered via cross-attention using a learned latent "Plücker code" representing a pose manifold.
2. Regularized Explicit 3D Representations: Impose strong regularization on 3D Gaussian Splatting (3DGS) or voxel-based radiance fields, adding structured per-pixel parameterizations, monocular/stereo depth priors, or self-supervised constraints (e.g., binocular consistency (Han et al., 24 Oct 2024), flow-based, or total variation losses (Paliwal et al., 28 Mar 2024)) to prevent overfitting and enforce coherence.
3. Pretrained Diffusion and Generative Priors: Employ frozen or plug-and-play diffusion models to inject visual realism, fill in gaps, or hallucinate plausible geometry where direct 3D evidence is missing (Li et al., 25 Mar 2025, Kani et al., 2023, Zhang et al., 17 Nov 2025, Bao et al., 31 Mar 2025). Diffusion models are conditioned using control signals derived from sparse views, point clouds, or cross-view attention.
4. Hybrid Construct-Optimize Pipelines: Construct a coarse 3D model by back-projecting monocular or pairwise depth into a (possibly unposed) sparse 3D Gaussian cloud, then refine jointly via differentiable correspondences and pose/depth adjustment without explicit camera initialization (Jiang et al., 6 May 2024, He et al., 21 Aug 2025).
5. Epipolar Geometry and Flow-Based Point Cloud Densification: Recent approaches use optical flow and epipolar constraints to densify the 3D point cloud and deliver accurate, scale-consistent initializations for 3DGS, reducing the need for ad-hoc regularizers and enabling high-fidelity NVS under extreme sparsity (Zheng et al., 24 Mar 2025).

3. Learning Strategies and Implicit 3D Reasoning

Novel view synthesis under sparse input must induce implicit 3D awareness in the absence of direct geometric supervision. Key mechanisms include:

• Implicit Camera Embeddings: The UP-LVSM framework predicts a 7D latent "camera" embedding $c_t=(\mathbf{x}, \mathbf{q})$ for the novel view, expanded to per-pixel Plücker ray codes $Z_t \in \mathbb{R}^{H \times W \times 6}$ via an MLP, permitting direct rendering with only 2D supervision (Wang et al., 11 Jun 2025).
• Cross-View Attention: Transformers aggregate tokens across all views, and cross-attend with pose-agnostic rendering queries, enabling 3D-consistent scene understanding even in the unposed regime.
• Self-Supervision and Priors: Binocular-guided methods employ stereo consistency via differentiable warping and self-supervised losses (Han et al., 24 Oct 2024), while others leverage flow-consistency constraints to regularize per-pixel/geometric assignments (Paliwal et al., 28 Mar 2024, Zheng et al., 24 Mar 2025).
• Plug-in Diffusion Guidance: Diffusion models can serve as strong implicit priors for plausible appearance and geometry, being supervised by spatial signals derived from 3D features, warped images, or inpainted occlusion regions (Li et al., 25 Mar 2025, Kani et al., 2023, Bao et al., 31 Mar 2025).

Empirical evidence indicates that as the scale of training data increases, architectures with minimal geometric prior (i.e., less explicit 3D inductive bias) learn implicit spatial reasoning that can match or exceed the performance of approaches constrained by strong 3D structure, making data-centric methods particularly powerful in the large-scale, unconstrained setting (Wang et al., 11 Jun 2025).

4. Regularization, Priors, and Optimization Tactics

Sparse-input NVS approaches address the risk of overfitting and degeneracy via:

• Total Variation and Multi-view TV: Regularization terms penalize high-frequency variation in depth or color among neighboring Gaussians or rays (Paliwal et al., 28 Mar 2024).
• Flow and Stereo-Based Consistency: Dense correspondence fields (optical flow) are exploited to enforce 3D consistency between views, particularly in constructing and selecting per-pixel depths (Zheng et al., 24 Mar 2025, Paliwal et al., 28 Mar 2024).
• Opacity Decay and Pruning: Mechanisms such as Gaussian opacity decay shrink and prune stray Gaussians far from true surfaces, yielding more compact and robust representations (Han et al., 24 Oct 2024, Miao et al., 5 Nov 2025).
• Pseudo-supervision via Synthetic or Enhanced Views: Uncertainty-aware, diffusion-generated or inpainted pseudo-views are used as supervisory signals, leveraging generative models to densify input coverage and mitigate gaps (Bao et al., 31 Mar 2025, Xu et al., 22 Nov 2025).
• Layered and Segmented Representations: For challenging unbounded/360° settings, layered Gaussian splatting or mask-based segmentation helps disambiguate disparate scene partitions and improve surface integrity (Bao et al., 31 Mar 2025).

Recent methods further optimize through data-driven selection (epipolar sensitivity in depth blending (Zheng et al., 24 Mar 2025)), adaptive loss weighting (e.g., pixel-uncertainty masks (Xu et al., 22 Nov 2025)), and plug-and-play modules for occlusion removal or style adaptation (Li et al., 25 Mar 2025).

5. Quantitative Results and Evaluation

State-of-the-art approaches consistently improve on standard sparse-input NVS benchmarks under minimal input regimes. Representative results from leading methods:

Method	Input Pose	PSNR↑	SSIM↑	LPIPS↓
PixelNeRF	posed	20.33	0.572	0.549
MVSplat	posed	26.45	0.874	0.123
LVSM	posed	27.60	0.874	0.117
UP-LVSM	unposed	28.82	0.891	0.104

UP-LVSM (Wang et al., 11 Jun 2025), which uses no explicit pose or 3D knowledge, outperforms all prior pose-dependent baselines on the RealEstate10K dataset when trained on 66K scenes. On LLFF (3 views), binocular-guided 3DGS yields PSNR 21.44/SSIM 0.751/LPIPS 0.168 (Han et al., 24 Oct 2024), and advanced point-cloud or flow-based methods such as NexusGS, COGS, and DentalSplat show even higher fidelity reconstructions on both object-centric and challenging real-world data under 2–4-view settings (Zheng et al., 24 Mar 2025, Jiang et al., 6 May 2024, Miao et al., 5 Nov 2025).

Experiments further show:

• Data-centric, low-bias architectures scale more favorably with dataset size, exhibiting larger absolute PSNR/SSIM gains as the number of training scenes increases (Wang et al., 11 Jun 2025).
• Densification and generative pseudo-supervision via diffusion models or synthetic hemisphere-sampled views dramatically reduce overfitting and geometric distortion when only 2–4 input images are available (Chen et al., 25 May 2025, Bao et al., 31 Mar 2025, Xu et al., 22 Nov 2025).
• Explicit regularization—flow, TV, masked self-supervision—remains essential for convergence and quality under extreme input sparsity.

6. Emergent Properties, Limitations, and Evolving Directions

Sparse-input NVS methods exhibit several salient properties:

• Implicit 3D awareness can arise solely from 2D training when sufficient data and network capacity are available, as evidenced by UP-LVSM's emergent 3D-consistent attention maps (Wang et al., 11 Jun 2025).
• Flexible, plug-and-play pipeline design enables new capabilities (occlusion removal, style transfer, in-the-wild adaptation) via modular diffusion or segmentation components (Li et al., 25 Mar 2025).
• Composability and generalizability: Data-centric and generative approaches can generalize across category, scene, or domain, with robust quantitative and qualitative performance even under novel or OOD distributions (Kani et al., 2023).
• Sub-linear scaling of regularizers: As data scales, the need for strong 3D or geometric priors diminishes; regularization becomes more about stabilizing, rather than dictating, the inductive bias.

Limitations include:

• Persistent ambiguity and potential for hallucination in highly occluded or unseen regions.
• Degraded performance under severe pose error or total lack of viewpoint overlap, unless explicit 2D–3D correspondences are enforced (Jiang et al., 6 May 2024).
• Dependence on the quality and diversity of pretrained models (DINOv2, diffusion) for appearance generalization.
• Need for further reduction in memory and compute for real-time or resource-constrained scenarios, although explicit 3D/voxel models are already much more efficient than MLP-based NeRFs (Sun et al., 2023).

Continued research explores tighter fusion of generative and geometric modules, self-supervised or epipolar-guided pose estimation, multi-layered/segmented representations for unbounded scenes, and dynamic or temporal (video) NVS from sparse observations.

7. Significance and Outlook

The evolution of sparse-input novel view synthesis illustrates a broader trend from explicit, geometry-driven reasoning to data-centric, self-supervised, and generative paradigms. As demonstrated, the reduction or elimination of 3D inductive bias—when combined with Transformers, plug-in diffusion, and large-scale datasets—can yield models whose performance not only scales with data but often surpasses that of conventional 3D-reliant techniques (Wang et al., 11 Jun 2025, Bao et al., 31 Mar 2025, Kani et al., 2023). Practical impact encompasses real-world capture with minimal input (e.g., casual phone photography, telemedicine imaging (Miao et al., 5 Nov 2025), AR/VR 360° reconstructions (Chen et al., 25 May 2025)), democratizing high-fidelity free-viewpoint imaging for non-expert users under realistic—often severely constrained—capture conditions.

Sparse-input NVS continues to be a fertile area for investigating the interaction between geometric consistency, high-level priors, and scale, with differentiated trajectories in fully unposed, dynamic, and unconstrained photographic regimes. The field moves toward increasingly robust, generalizable, and data-driven solutions for photorealistic 3D scene and object synthesis from minimal observations.