Generalizable Novel View Synthesis

• The paper introduces a method where a single shared model synthesizes novel views without per-scene tuning from limited inputs.
• It leverages diverse architectures—including feature-conditioned radiance fields, explicit 3D structures, and diffusion-based models—for robust view synthesis.
• It balances explicit geometric cues with generative priors to effectively address challenges like sparse views, occlusions, and ambiguous scene structures.

Generalizable novel view synthesis (NVS) is the setting in which a single model is trained across many scenes and then synthesizes images from new viewpoints for previously unseen scenes in a feed-forward manner, without per-scene optimization or test-time fitting (Lee et al., 2024, Jin et al., 2024, Nair et al., 8 Sep 2025). In contrast to per-scene NeRF- or 3D Gaussian Splatting-style pipelines that optimize an explicit scene representation for each new capture, generalizable NVS aims to amortize multi-view reasoning, appearance inference, and camera-conditioned rendering into a scene-agnostic network that can operate from sparse observations, often one to a few views, and often under substantial geometric ambiguity (Jin et al., 2024, Elata et al., 2024). Across recent work, the field spans feature-conditioned radiance fields, cost-volume and stereo-guided pipelines, explicit 3D Gaussian representations, transformer models with minimal 3D inductive bias, and diffusion or video-diffusion systems that synthesize views directly in image or latent space (Lee et al., 2024, Min et al., 2024, Jin et al., 2024, Elata et al., 2024, You et al., 2024).

1. Problem definition and conceptual scope

Generalizable NVS is typically formulated as: given one or more source images with known or inferable camera parameters, synthesize an image at a new camera pose using a model trained once across many scenes (Jin et al., 2024, Elata et al., 2024). The shared property across this literature is that generalization refers to transferring a learned view-synthesis function to unseen scenes without scene-specific retraining, in contrast to classical NeRF-style optimization that fits a separate coordinate MLP, voxel grid, hash grid, or Gaussian set per scene (Lee et al., 2024, Nair et al., 8 Sep 2025).

Within that common formulation, the field now includes several distinct operational regimes. One regime is scene-level sparse-view NVS, exemplified by RealEstate10K, ACID, DL3DV, and related benchmarks, where models infer scene structure and synthesize target views from very few inputs, often two views or even a single view (Nair et al., 8 Sep 2025, Elata et al., 2024, Jin et al., 2024). A second regime is object-centric NVS, where cameras orbit a single object and the model must complete unseen surfaces and back views from one or a few images (Liang et al., 20 Mar 2026, Zheng et al., 2023, Jeong et al., 2023). A third regime extends the problem to challenging acquisition conditions, including stereo-camera rigs (Lee et al., 2024), motion-blurred inputs (Shi et al., 31 May 2026), appearance transfer across weather or lighting conditions (Bengtson et al., 2023), and long-range camera trajectories in generative video settings (Chen et al., 23 Feb 2026).

A recurring technical difficulty is that sparse-view NVS is fundamentally ill-posed. With only one or two inputs, models must infer depth, occlusions, disocclusions, view-dependent effects, and the appearance of regions not directly observed (Nair et al., 8 Sep 2025, Elata et al., 2024). This suggests that the modern generalizable NVS literature is organized less by a single architecture than by different choices about where to place prior knowledge: explicit geometry, learned stereo or MVS priors, camera-ray embeddings, semantic priors, or large generative priors from diffusion and video models (Lee et al., 2024, Jin et al., 2024, Chen et al., 23 Feb 2026, You et al., 2024).

2. Representation choices and architectural families

One major family consists of feature-conditioned radiance-field methods. In these systems, the radiance field is not represented by a per-scene MLP, but by networks trained across many scenes that condition density and color prediction on source-view features sampled at projected 3D points (Lee et al., 2024, Bengtson et al., 2023, Zhou et al., 2022). This family includes transformer-based generalizable NeRFs such as GNT, which decomposes view synthesis into a view transformer over per-view features and a ray transformer over points sampled along target rays, and then predicts colors with a color reconstruction loss over rays (Bengtson et al., 2023). StereoNeRF extends this line by explicitly integrating stereo matching into a NeRF-based generalizable view synthesis pipeline, using a stereo feature extractor, depth-guided plane sweeping, and a stereo depth loss (Lee et al., 2024).

A second family uses explicit intermediate 3D structure. ProbNVS builds multi-scale MVS cost volumes, obtains depth probability distributions, samples only a small set of points for rendering under learned probability-guided sampling, and then performs neural volume rendering with a confidence-aware refinement module (Zhou et al., 2022). eFreeSplat takes a different explicit route by predicting a feed-forward 3D Gaussian Splatting representation for unseen scenes from two views, with one Gaussian per pixel per view and standard 3DGS rendering after an epipolar-free feature extractor and Iterative Cross-view Gaussians Alignment (Min et al., 2024). DeblurNVS does not use explicit volumes or Gaussian primitives, but treats Depth Anything 3 latent space as an intermediate geometric representation and restores sharp, geometry-consistent latents before target-view synthesis (Shi et al., 31 May 2026).

A third family is defined by minimal 3D inductive bias. LVSM removes explicit 3D representations, explicit rendering equations, plane sweeps, cost volumes, epipolar projections, and specialized attention, exposing only camera geometry encoded as per-pixel Plücker rays to a transformer (Jin et al., 2024). Its decoder-only variant directly maps input image tokens and target ray tokens to target image patches, and the only 3D cue is the ray embedding (Jin et al., 2024). This design was later extended by token-disentangled transformer models that distinguish source and target tokens using modulation, explicitly separating the role of source observations from the role of target reconstruction (Nair et al., 8 Sep 2025).

A fourth family is generative diffusion-based NVS. Pixel-space diffusion models such as VIVID frame single-image novel view synthesis as conditional diffusion in pixel space and argue that improved generative backbones matter more than increasingly elaborate geometry encodings (Elata et al., 2024). NVS-Solver instead treats a pre-trained video diffusion model as a zero-shot NVS engine, adaptively modulating the diffusion score with warped input views according to camera pose and diffusion step (You et al., 2024). OrbitNVS reformulates object-centric NVS as orbit video generation with camera adapters, a normal-map branch, and pixel-space supervision (Liang et al., 20 Mar 2026). SemanticNVS augments camera-conditioned multi-view diffusion with warped and iterative DINOv2 features to preserve semantics over long trajectories (Chen et al., 23 Feb 2026). NVComposer goes further by eliminating explicit external alignment and jointly generating target novel views and condition camera poses from multiple sparse and unposed images (Li et al., 2024).

This diversity of representations suggests a central structural split in the field: some methods externalize geometry into cost volumes, depth maps, Gaussians, or latent geometric fields, while others ask a large transformer or diffusion model to infer geometry implicitly from camera-conditioned image tokens (Lee et al., 2024, Min et al., 2024, Jin et al., 2024, Nair et al., 8 Sep 2025).

3. Geometry, correspondence, and conditioning mechanisms

A persistent theme in generalizable NVS is how to represent camera geometry and cross-view correspondence. A classical route is to build cost volumes or plane sweeps. StereoNeRF uses stereo depths from a frozen UniMatch network to define local depth windows around likely surfaces, then constructs multi-view cost volumes by depth-guided plane sweeping rather than globally sampling a fixed depth range (Lee et al., 2024). This shrinks the search space, reduces outliers, and improves depth accuracy especially in textureless regions and thin structures (Lee et al., 2024). ProbNVS similarly relies on learned MVS priors, but uses discrete depth probability distributions to perform coarse-to-fine inverse-transform sampling, reducing rendering to only eight samples per ray at the final stage (Zhou et al., 2022).

Another route is to weaken or remove explicit epipolar constraints. eFreeSplat argues that epipolar priors can be unreliable in complex real scenes, particularly in non-overlapping and occluded regions, and instead uses a self-supervised ViT with CroCo-style cross-view completion pre-training to learn epipolar-free feature matching and encoding (Min et al., 2024). It then applies Iterative Cross-view Gaussians Alignment to enforce depth-scale consistency across views before constructing a Gaussian representation (Min et al., 2024). This is conceptually distinct from purely geometry-free methods, because it still outputs explicit Gaussians, but the matching prior is learned through cross-view completion rather than explicit line or plane geometry (Min et al., 2024).

A different line encodes geometry by camera rays rather than reconstructed depth. LVSM uses per-pixel Plücker ray embeddings as the sole 3D cue in a full transformer, and VIVID shows that a simple pose embedding already captures most of the geometric benefit on a strong diffusion backbone, with epipolar bias, monocular depth, or coordinate warping giving only marginal gains on top (Jin et al., 2024, Elata et al., 2024). Free3D pushes ray conditioning further with Ray Conditioning Normalization, where per-pixel Plücker ray embeddings modulate layer-normalized UNet features at every layer, replacing weak global camera tokens with distributed pixel-level viewing-direction conditioning (Zheng et al., 2023). NVS-Adapter uses Plücker-ray embeddings inside view-consistency cross-attention, allowing target and reference features to align through learned ray-aware correspondences while retaining a frozen Stable Diffusion backbone (Jeong et al., 2023).

Several recent generative models supplement geometric conditioning with semantic priors. SemanticNVS argues that ray maps and warped RGB are insufficient for long-range camera motion and therefore injects warped DINOv2 features and iterative DINO features extracted from intermediate denoised samples into a SEVA-style multi-view diffusion model (Chen et al., 23 Feb 2026). NVComposer distills geometric priors from DUSt3R pointmaps into encoder features via a geometry-aware alignment loss, but removes the external stereo model at inference time, allowing the model to infer latent camera poses and spatial relationships internally (Li et al., 2024). This suggests a broader trend toward hybrid conditioning: geometry alone is often insufficient in weak-overlap regimes, while semantics alone is too unconstrained.

4. Data regimes, benchmarks, and evaluation practice

The progress of generalizable NVS is tightly coupled to dataset construction. Scene-level training and evaluation frequently rely on RealEstate10K, ACID, and DL3DV, sometimes augmented by synthetic data or specialized domain-specific datasets (Nair et al., 8 Sep 2025, Elata et al., 2024, Jin et al., 2024). LVSM trains on RealEstate10K for scene-level NVS and Objaverse for object-level NVS, emphasizing that the same architecture can generalize across object-level and scene-level settings (Jin et al., 2024). Transformer scaling work highlights that public multi-view scene datasets such as Re10K, ACID, and DL3DV cover fewer than 100k scenes, which can bottleneck large transformer models, motivating synthetic multi-view data from diffusion models like CAT3D (Nair et al., 8 Sep 2025).

Several works contribute new datasets tailored to underexplored settings. StereoNeRF introduces StereoNVS, described as the first multi-view dataset of stereo-camera images, with 53 real scenes and 50 synthetic scenes, camera intrinsics and poses, rectified binocular pairs, and synthetic depth maps for geometry evaluation (Lee et al., 2024). DeblurNVS constructs DL3DV-10K-Blur, a motion-blurred NVS dataset obtained by frame interpolation and temporal averaging, containing around 10k scenes and around 5M sharp–blur pairs (Shi et al., 31 May 2026). Aug3D addresses large-scale outdoor generalizable NVS on UrbanScene3D by clustering scenes with Structure-from-Motion shared points and augmenting them through reconstructed-scene view synthesis, arguing that limited view overlap is a major bottleneck in feed-forward outdoor NVS (Rauniyar et al., 11 Jan 2025).

Evaluation practices remain heterogeneous but converge around a core metric set. Most image-quality evaluations report PSNR, SSIM, and LPIPS (Lee et al., 2024, Nair et al., 8 Sep 2025, Jin et al., 2024, Elata et al., 2024, Min et al., 2024). Geometry quality is often evaluated with ABS or absolute depth error where ground-truth or proxy depth is available, as in StereoNVS (Lee et al., 2024), or with DISTS and FID in deblurring settings where perceptual fidelity is especially relevant (Shi et al., 31 May 2026). Generative systems with long trajectories report distribution and temporal-consistency metrics such as FID, FVD, KVD, ImQ, Drift, and MEt3R (Chen et al., 23 Feb 2026, Li et al., 2024). For appearance-transfer NVS, the literature also uses tOF and tLP to measure temporal and multi-view consistency under edited conditions (Bengtson et al., 2023).

Across these benchmarks, a recurring empirical pattern is that stronger geometry or semantic priors improve difficult cases such as textureless regions, thin structures, sparse overlap, motion blur, and long-range camera motion (Lee et al., 2024, Shi et al., 31 May 2026, Chen et al., 23 Feb 2026). A plausible implication is that dataset diversity alone is not enough; the field increasingly rewards architectures that know how to exploit that diversity through specialized conditioning.

5. Generalization strategies and empirical scaling laws

Generalizable NVS systems now draw generalization from at least four distinct sources: scene-agnostic parameter sharing, geometric priors, semantic priors, and synthetic-data scaling.

A first strategy is to strengthen scene-agnostic geometry pathways. StereoNeRF improves generalization by combining shared CNNs, stereo attention modules, depth-guided plane sweeping, and a stereo depth loss that supervises stereo, MVS, and NeRF-rendered depths using pseudo-ground-truth from a frozen stereo model (Lee et al., 2024). ProbNVS improves efficiency and retains strong generalization by coupling learned MVS depth probabilities to rendering, rather than sampling the full depth range densely (Zhou et al., 2022). DeblurNVS decouples latent restoration from target-view synthesis so that geometric latents are first restored from motion-blurred inputs before novel-view generation (Shi et al., 31 May 2026).

A second strategy is to scale transformer capacity and data. LVSM shows monotonic gains as decoder-only depth increases from 6 to 24 layers on RealEstate10K and GSO, and emphasizes that the decoder-only architecture scales more gracefully with additional views than an encoder–decoder latent bottleneck (Jin et al., 2024). Token-disentangled transformer models report that naively adding synthetic diffusion-generated multi-view data can hurt LVSM, whereas differentiating source and target tokens allows synthetic data to improve Re10K, ACID, and DL3DV performance and cross-dataset generalization (Nair et al., 8 Sep 2025). This suggests that scaling data successfully may require architectural changes that explicitly protect the target reconstruction pathway from source-side artifacts.

A third strategy is to expand the training distribution with single-view or synthetic augmentation. VIVID uses homography-based camera rotation augmentation on single-view datasets such as OpenImages, mixing a small percentage of synthetic rotation pairs into RealEstate10K training. The reported result is that 10% synthetic single-view augmentation improves out-of-domain FID and PSNR on LLFF, MipNeRF-360, and Ref-NeRF without degrading in-domain RealEstate10K performance, whereas 25% synthetic begins to overemphasize rotations (Elata et al., 2024). Aug3D, in a different domain, uses SfM-based clustering and scene augmentation on a large outdoor dataset and reports that reducing cluster size from 20 to 10 improves PSNR by 10%, with Aug3D further improving PixelNeRF’s ability to predict novel views in outdoor scenes (Rauniyar et al., 11 Jan 2025).

A fourth strategy is to exploit foundation-model priors. NVS-Adapter keeps the entire Stable Diffusion backbone frozen and trains only view-consistency cross-attention and global semantic conditioning modules, preserving the original T2I model’s broad generalization while adding geometry-aware NVS behavior (Jeong et al., 2023). OrbitNVS adapts a large pre-trained video generation model with camera adapters, a normal-map branch, and pixel-space supervision, achieving large PSNR margins in the single-view object-centric regime (Liang et al., 20 Mar 2026). NVS-Solver goes even further by dispensing with any NVS-specific training and using test-time adaptive modulation of a pre-trained video diffusion model as a zero-shot solver (You et al., 2024). These developments suggest that in some parts of the field, generalization is now as much about leveraging pre-trained generative priors as about constructing better explicit geometry modules.

6. Limitations, controversies, and open directions

Despite rapid progress, several limitations recur across the literature. Extremely sparse input remains a core challenge. StereoNeRF explicitly notes that with very few views it can still produce blurry images and inaccurate geometry due to inherent ill-posedness (Lee et al., 2024). Pixel-space diffusion NVS observes that large viewpoint changes and strong occlusions require hallucination of plausible unseen regions, and there is no explicit guarantee of multi-view 3D consistency when the model is trained on single target views (Elata et al., 2024). SemanticNVS shows that long-range camera motion causes severe degradation in existing diffusion models, motivating explicit semantic conditioning (Chen et al., 23 Feb 2026).

A second limitation is dependence on specific acquisition assumptions. StereoNeRF requires rectified stereo pairs with a fixed baseline and is not directly applicable to arbitrary monocular-only datasets (Lee et al., 2024). Free3D and OrbitNVS operate in object-centric settings and rely on known camera poses, normalized scale, and static objects (Zheng et al., 2023, Liang et al., 20 Mar 2026). DeblurNVS assumes camera poses are available via DA3’s camera module and remains slower than pure feed-forward encoders because inference still includes diffusion steps (Shi et al., 31 May 2026). NVComposer removes external alignment at inference time, but still relies on DUSt3R supervision during training and on a fixed bundle structure during diffusion (Li et al., 2024).

A third limitation concerns efficiency versus structure. Explicit geometry methods can be strong on consistency but expensive or brittle. ProbNVS shows that learned probability-guided sampling can make NeRF-style rendering 15 to 40 times faster than prior baselines, yet its confidence-aware refinement still cannot exactly guarantee cross-view rendering consistency (Zhou et al., 2022). eFreeSplat is efficient and explicit, but depends heavily on CroCo pre-training and uses one Gaussian per pixel per view, which can be memory intensive at higher resolutions (Min et al., 2024). Diffusion systems such as NVS-Solver and SemanticNVS achieve strong quality in difficult regimes but incur substantial sampling cost and repeated feature extraction (You et al., 2024, Chen et al., 23 Feb 2026).

A fourth issue is the tension between geometry-aware structure and pure data-driven scaling. LVSM argues that explicit 3D representations, plane sweeps, and epipolar modules can become bottlenecks as model and data scale (Jin et al., 2024). VIVID reports that on a strong pixel-space diffusion backbone, pose embedding captures most of the geometric benefit and that geometry encodings beyond pose provide only minor incremental gains (Elata et al., 2024). By contrast, StereoNeRF, eFreeSplat, ProbNVS, and DeblurNVS each show substantial gains from stronger geometric inductive biases in their respective settings (Lee et al., 2024, Min et al., 2024, Zhou et al., 2022, Shi et al., 31 May 2026). This suggests an unresolved divide in the field: whether future generalizable NVS should become more explicitly geometric or more strongly generative with only lightweight camera conditioning.

The forward-looking directions stated across these works are correspondingly diverse. Suggested directions include integrating diffusion-based or other learned priors for extremely sparse inputs (Lee et al., 2024); improving synthetic multi-view generation and artifact handling (Nair et al., 8 Sep 2025); extending stereo methods to monocular-only inputs via stereo hallucination and to dynamic scenes (Lee et al., 2024); scaling semantic conditioning more efficiently and applying it to dynamic or longer trajectories (Chen et al., 23 Feb 2026); using hierarchical transformers, linear attention, or state-space models to reduce memory (Nair et al., 8 Sep 2025); and combining fast generalizable models with explicit 3D techniques such as hash encodings, Gaussian splatting, or downstream scene reconstruction (Lee et al., 2024, Jin et al., 2024). A plausible implication is that the next phase of generalizable NVS will be hybrid: not a single dominant representation, but systems that selectively combine camera-ray conditioning, geometric priors, semantic understanding, and large generative backbones according to the data regime and failure mode.