Georeferenced Novel View Synthesis

• The paper presents a novel approach to synthesize photorealistic views by integrating georeferenced imagery with advanced geometric priors and precise camera pose estimation.
• It exploits transformer-based models, diffusion frameworks, and pointmap conditioning to maintain 3D structural integrity even in sparse data scenarios.
• Empirical evaluations show that fusing ground-level, LiDAR, and satellite data significantly enhances rendering fidelity and spatial alignment in large-scale environments.

Georeferenced image-based novel view synthesis refers to the computational problem and suite of methodologies for synthesizing previously unseen (novel) views of a scene using sets of input images with known or inferrable camera pose and geodetic metadata. The overarching technical challenge is to leverage spatially referenced appearance and geometry to render photorealistic and geometrically consistent images from arbitrary query viewpoints, frequently in large-scale and data-sparse scenarios. This area spans generative diffusion frameworks, transformer-based models with geometric attention biases, and physically grounded scene representations, all unified by the ability to reason about camera geometry, 3D structure, and spatial context.

1. Problem Statement and Motivation

Georeferenced novel view synthesis (NVS) is defined as generating color images and, in some cases, 3D geometry for viewpoints that have not been directly observed, conditioned on one or more input images and their associated spatial information including camera intrinsics, extrinsics, and external geolocation (e.g., GPS). Unlike traditional multi-view NVS, which generally assumes dense, regular camera poses and local consistency, the georeferenced setting must contend with diverse reference poses, scale disparities (terrestrial, aerial, satellite), and often sparse or noisy geometric data. Applications range from automated urban mapping and street-level localization to remote sensing and virtual city-scale exploration (Nguyen et al., 6 Jan 2025, Turkulainen et al., 19 May 2026).

The synthesis task is underpinned by the fundamental need to model not just image-to-image correspondence, but also 3D spatial relationships, visibility, and occlusion. Input modalities include RGB images, depth or point clouds from LiDAR/DEM sources, and orthorectified satellite images. Recent works demonstrate that accurate synthesis benefits from explicit georeferencing, either via learned geometric priors, explicit warping and inpainting, or attention mechanisms informed by 3D spatial proximity (Kwak et al., 13 Jun 2025, Venkat et al., 2023).

2. Geometric Priors and Conditioning Representations

Methodologies in georeferenced NVS are broadly differentiated by how geometry is represented and injected into the synthesis pipeline. A principal axis of distinction lies between models that are geometry-free (set-latent) and those that are geometry-biased or geometry-conditioned.

• Geometry-biased transformer frameworks use 3D geometric cues (camera poses, rays, or pointmaps) to modulate attention weights, enforcing inductive bias for spatial consistency. An example is the Geometry-biased Transformer (GBT), which augments standard attention with a learnable penalty on 3D ray-to-ray distances, derived from Plücker embeddings or equivalent representations (Venkat et al., 2023). This bias enables multi-view transformers to focus attention on spatially proximate tokens, improving geometric fidelity in synthesized views.
• Pointmap-based diffusion approaches create dense grids where each pixel stores a 3D position, typically constructed by transforming per-pixel (latitude, longitude, altitude) into a target camera frame (via ECEF and SE(3)), and then passing these as ControlNet conditions and/or cross-attention keys (Nguyen et al., 6 Jan 2025). Optionally, normalization and Fourier-style positional encodings are used to capture high-frequency spatial variations.
• Proximity-based mesh conditioning involves reconstructing a triangulated mesh (e.g., ball-pivoting algorithms) from sparse or noisy point clouds, projecting it from the target viewpoint, and filtering by normal orientation to favor geometrically plausible conditioning (Kwak et al., 13 Jun 2025). Depth and surface normal maps derived from the mesh may be appended to other conditioning signals, and care is taken to avoid propagating erroneous geometry.
• Gaussian splats as scene parameterization are leveraged in hybrid feed-forward architectures, with ground-level and satellite imagery fused via aligned feature spaces and cross-view attention. Each splat is parameterized by mean position, covariance, spherical harmonic color, and opacity; their collection forms a compact and renderable representation for arbitrary camera poses (Turkulainen et al., 19 May 2026).

The table below summarizes key geometric conditioning strategies in representative works:

Representation	Conditioning Modality	Notable Work
Plücker ray embeddings + distance bias	Transformer attention	GBT (Venkat et al., 2023)
Rasterized pointmaps (ℰ ∈ ℝ^{H×W×3})	ControlNet/attention	PointmapDiffusion (Nguyen et al., 6 Jan 2025)
Ball-pivoting mesh projections	U-Net input features	MoAI (Kwak et al., 13 Jun 2025)
Gaussian splats (μ_j, Σ_j, c_j, o_j)	Splat rasterization	Cross-View Splatter (Turkulainen et al., 19 May 2026)

All approaches are fundamentally reliant on accurate camera calibration and geospatial metadata, with quantitative robustness tied to the noise and coverage properties of these geometric cues.

3. Model Architectures and Inference Mechanisms

The core architectural paradigms for georeferenced NVS can be divided into diffusion-based and feed-forward renderer classes.

Diffusion-based Frameworks

• Dual-branch diffusion models (MoAI) deploy two parallel denoising U-Nets: one for color image synthesis, one for geometry prediction (e.g., pointmaps). Each is independently conditioned on warped geometric features and attention is aggregated from multi-view references (Kwak et al., 13 Jun 2025). Novel-view synthesis is posed as an inpainting problem: known pixels (back-projected from references) are treated as observations, and unknown (occluded or unobserved) pixels are synthesized through DDPM diffusion steps.
• Pointmap-conditioned diffusion (PointmapDiffusion) leverages a frozen pre-trained U-Net (e.g., Stable Diffusion v1.5) plus architectural augmentations. Two small ControlNets inject spatially warped reference/target pointmaps into the mid- and up-sampling blocks of the U-Net. A reference-guided cross-view attention block in the decoder allows dynamic copying of appearance from reference pixels whose 3D positions are nearby in the target frame (Nguyen et al., 6 Jan 2025).

Feed-forward Approaches

• Geometry-biased transformer models (GBT) encode all input images into patch tokens fused with camera-aware ray embeddings, pass through a geometry-biased transformer encoder/decoder stack, and decode query rays for a novel pose into RGB via MLPs. The distance bias in attention logits is learnable per layer, directly favoring tokens that are 3D-proximate to the query (Venkat et al., 2023).
• Cross-View Splatter accepts both ground-level images (with GPS/heading) and satellite (BEV) images. Each branch independently encodes images using ViT backbones, predicts camera pose, dense depth, and 3D Gaussian splats. Extensive cross-view (meta) attention fuses ground and satellite features, enabling learning of joint scene geometry. At inference, the full Gaussian splat collection is rasterized to arbitrary view via 3D splatting, facilitating photorealistic and globally aligned rendering at large scale (Turkulainen et al., 19 May 2026).

4. Loss Functions, Optimization, and Supervision

Supervision strategies vary by architectural class but are unified in the explicit weighting of image reconstruction, geometry alignment, and, where applicable, inpainting losses:

• Diffusion denoising loss (DDPM): Standard $\ell_2$ norm between predicted and real noise added at each step, on both image and geometry branches: e.g., $$L_{\text{diff}}^{\text{img}} = E_{I_t, \epsilon, t} [ \|\epsilon - \epsilon_{\theta}(I_t, t; c^t, c^r)\|_2^2 ]$$.
• Cross-modal attention distillation: Optional $\ell_1$ penalty for divergence between spatial attention maps of image and geometry branches, but often replaced by hard substitution of attention statistics (Kwak et al., 13 Jun 2025).
• Inpainting reconstruction loss: For pixels unobserved in warped reference pointmaps, reconstruction of RGB and geometry is enforced only on these regions, e.g., $$L_{\text{rec}}^{\text{img}} = \| (1-M_t) \odot (\hat{Y}^{\text{img}} - I_{\text{gt}}) \|_2^2$$.
• Geometry consistency: Pointmap-based methods may apply an explicit geometry-consistency loss by warping predicted target images back to reference frames and penalizing pixel-wise disparities (Nguyen et al., 6 Jan 2025).
• Color and pose supervision: Feed-forward methods often employ $\ell_2$ or perceptual (LPIPS) loss on rendered images from Gaussian splats, and explicit losses on predicted camera pose, depth, and splat shapes (Turkulainen et al., 19 May 2026).

Hyper-parameters governing these loss terms are selected for balance, with ablations indicating individual contributions in the range of 0.1–0.3 dB PSNR (Kwak et al., 13 Jun 2025).

5. Empirical Evaluation and Benchmarking

Benchmarks for georeferenced NVS encompass sparse-view, extrapolative, and interpolative scenarios on datasets such as DTU, Co3D, RealEstate10K, Tanks & Temples, and large-scale urban/satellite collections.

• Diffusion models (MoAI, PointmapDiffusion): On DTU, two-view extrapolation yields PSNR ≈ 15.58, SSIM ≈ 0.615, LPIPS ≈ 0.184, outperforming PixelSplat, and similar trends hold on RealEstate10K and in 1-shot extrapolation (Kwak et al., 13 Jun 2025). PointmapDiffusion achieves competitive image and geometry alignment, with robustness to sparse (10–50%) LiDAR input, and geolocation error measured by projected 3D keypoint RMS (Nguyen et al., 6 Jan 2025).
• Feed-forward models (Cross-View Splatter): Combines ground and satellite imagery to achieve 11.33–12.61 dB PSNR (Tanks & Temples, 1–3 views), a gain of ≈2 dB over ground-only approaches. DL3DV results similarly favor combined branch, with greatest gains at low context-target scene overlap, reflective of improved global coverage due to BEV cues (Turkulainen et al., 19 May 2026).
• Geometry-biased attention models (GBT): On CO3D, GBT achieves PSNR ≈ 22.56 dB versus 19.24 dB (ViewFormer) and 20.37 dB (pixelNeRF), with sharper details and improved hallucination of unseen geometry; ablations confirm the necessity of geometric attention bias for spatial fidelity (Venkat et al., 2023).

Qualitative inspection shows accurate hallucination of hidden surfaces (e.g., sofa legs, table undersides), globally aligned colored point clouds, and realistic rendering in large-scale urban or natural environments. Failure cases are mainly associated with unobserved geometry in both ground and BEV channels, temporal decorrelation between input modalities, or drift in geospatial alignment (particularly in DEM/LiDAR or GPS signals).

6. Current Limitations and Future Research Directions

Current frameworks manifest several limitations. Accuracy and global spatial consistency are sensitive to the precision of input georeferences: performance degrades in the presence of GPS noise, DEM misalignment, or calibration error. Real-time inference remains out of reach for most diffusion-based models due to the computational overhead of sequential denoising; faster feed-forward methods partially address this constraint, but may sacrifice flexibility in handling unobserved or highly extrapolative views (Nguyen et al., 6 Jan 2025, Turkulainen et al., 19 May 2026).

Scene coverage is ultimately limited by input data: no method reliably synthesizes geometry absent from both ground and BEV (e.g., interiors or deep shadowed regions), and satellite data recency or occlusion may artifactually constrain render quality. In high-overlap, small-baseline cases, additional georeferenced input yields diminishing returns; conversely, in low-overlap, broad-baseline scenarios, explicit BEV or global geometric priors provide substantial benefits (Turkulainen et al., 19 May 2026).

Active research fronts include:

• Multi-temporal and multi-modal fusion, enabling models to reason across temporally offset satellite and ground captures, or across SAR/multispectral bands (Nguyen et al., 6 Jan 2025, Turkulainen et al., 19 May 2026).
• Learned inpainting of globally unobserved regions via diffusion priors constrained by geometric uncertainty.
• Integration of large-scale reconstruction (e.g., GS-LRM, Bolt3D), full 6 DoF satellite imagery, and new attention architectures for increased spatial and appearance consistency (Turkulainen et al., 19 May 2026).
• Real-time or lightweight variants targeted for on-device inference in robotics, AR, or UAV navigation.

7. Comparative Summary of Representative Techniques

Approach	Geometric Conditioning	Image Synthesis Paradigm	Key Results
MoAI (Kwak et al., 13 Jun 2025)	Warped pointmaps + mesh	DDPM dual-branch inpainting	PSNR 17.41 (RealEstate10K), SOTA in extrapolation
GBT (Venkat et al., 2023)	Ray distance bias (Plücker)	Transformer set-latent	PSNR 22.56 (CO3D), improved detail
PointmapDiffusion (Nguyen et al., 6 Jan 2025)	ECEF->cam pointmap ControlNet	Diffusion+ControlNet	Robust to 10% LiDAR, <2 m geolocation error
Cross-View Splatter (Turkulainen et al., 19 May 2026)	Gaussian splats, BEV fusion	Feed-forward ViT + splat	PSNR 12.61 (Tanks & Temples, combined)

These results illustrate the critical role of explicit geometric priors and cross-modal feature fusion in overcoming the data and coverage limitations pervasive in large-scale georeferenced NVS tasks. The integration of globally referenced appearance and geometry, either via direct input or learned attention biases, remains the central technical axis along which progress is measured.