Sat3DGen: Satellite-to-Street 3D Generation

• Sat3DGen is a feed-forward system that generates photorealistic street-level 3D scenes from a single overhead satellite image using a tri-plane NeRF representation.
• It introduces geometric constraints and a mixed satellite-to-panorama-to-perspective training strategy to significantly reduce RMSE and FID compared to prior methods.
• The system supports multiple applications including semantic-map-to-3D synthesis, multi-camera video generation, large-scale meshing, and unsupervised DSM estimation.

Searching arXiv for the specified paper and related work. Sat3DGen is a feed-forward system for generating photorealistic, street-level 3D scenes from a single overhead satellite image. It is presented as a geometry-first framework that addresses a central trade-off in satellite-to-street synthesis: geometry-colorization models achieve high geometric fidelity but are typically building-focused and lack semantic diversity, whereas proxy-based feed-forward image-to-3D models generate richer holistic scenes but with coarse and unstable geometry. Sat3DGen retains a feed-forward tri-plane NeRF representation while introducing geometric constraints and a mixed satellite-to-panorama-to-perspective training strategy intended to counter the viewpoint gap and sparse, inconsistent supervision in satellite-to-street data. On a new benchmark pairing VIGOR-OOD with high-resolution DSM data, it improves geometric RMSE from $6.76$ m to $5.20$ m and reduces FID from approximately $40$ to $19$ relative to Sat2Density++, while also supporting downstream applications including semantic-map-to-3D synthesis, multi-camera video generation, large-scale meshing, and unsupervised single-image DSM estimation (Qian et al., 14 May 2026).

1. Problem formulation and motivation

The task addressed by Sat3DGen is the generation of a true 3D street-view scene from one overhead satellite patch $I_{\mathrm{sat}}$. In the formulation given for the method, the target scene includes buildings, roads, trees, road markings, and related street-level content. The motivation is practical as well as methodological: such a representation is described as appealing for mapping, simulation, and robotics (Qian et al., 14 May 2026).

Two bottlenecks structure the problem. The first is the viewpoint gap: training uses one top-down view and only a handful of street-view panoramas per tile, so the change from orthographic satellite to perspective street-view leaves large portions of the volume underconstrained, especially roofs and facades. The second is sparse, inconsistent supervision: satellite images provide no direct depth, panoramas observe only a few vantage points, and roof surfaces receive almost no multi-view signal. Within this setting, prior geometry-colorization pipelines based on a height-map, mesh, and texture sequence are described as missing nonbuilding semantics such as crosswalks, medians, and trees, while naive feed-forward NeRF or tri-plane pipelines are described as yielding rich content but unstable geometry, including bubbly roofs, floating artifacts, and torn edges (Qian et al., 14 May 2026).

Sat3DGen is framed explicitly as a response to these failure modes. Its core premise is to preserve the efficiency and semantic breadth of a feed-forward volume representation while introducing constraints that bias the learned field toward plausible outdoor geometry. A plausible implication is that the method treats the geometry problem not as an auxiliary regularization issue, but as the organizing principle of the entire training procedure.

2. Feed-forward pipeline and scene representation

At the architectural level, Sat3DGen follows a single-pass pipeline summarized as image $\rightarrow$ 3D volume $\rightarrow$ render or mesh. The input is a $256 \times 256$ satellite image. A frozen DINO-v3 ViT encoder $E_{\mathrm{sat}}$ maps this image into a $16 \times 16 \times 1024$ token grid,

$5.20$0

To accommodate boundary scene content, the token grid is padded at the periphery with $5.20$1 learnable spatial tokens. A small VAE-style decoder then upsamples the padded tokens into three orthogonal feature planes,

$5.20$2

with $5.20$3 (Qian et al., 14 May 2026).

The volumetric representation is a tri-plane NeRF. A shallow MLP receives features $5.20$4 sampled by bilinear interpolation from the tri-planes at any 3D point $5.20$5, and predicts density $5.20$6 and radiance $5.20$7. Both satellite rays and street-view rays are marched and composited to produce rendered color images. The same learned density field is also used for geometry extraction: Marching Cubes is applied at a fixed isovalue $5.20$8 to produce a watertight mesh (Qian et al., 14 May 2026).

This organization is important because it unifies rendering and meshing under one learned volume. The representation is therefore not limited to image synthesis; it also supports explicit 3D asset production. In the paper’s own summary, the output is a fully renderable NeRF and a watertight mesh, which directly enables the downstream uses described later.

3. Geometry-first constraints

The geometry-first methodology adds three geometric losses and supplements standard satellite and panorama supervision with perspective-view training. The first added term is the gravity-based density variation loss. The stated intuition is that real outdoor matter tends to accumulate downwards, so the density field $5.20$9 is encouraged to be non-increasing with altitude. For a sampled point $40$0 and a vertically offset point $40$1, the loss penalizes cases in which the density at the higher point exceeds the lower point by more than a slack $40$2: $40$3 The slack is fixed at $40$4, specifically to allow genuine overhangs such as tree canopies and bridges (Qian et al., 14 May 2026).

The second term is a monocular relative-depth prior in satellite view. To resolve roof ambiguity, Sat3DGen uses pseudo-labels $40$5 from Depth Anything v2 and applies a scale-and-shift-invariant depth loss to the rendered depth $40$6. Solving for optimal per-image $40$7, the loss is

$40$8

The construction is designed to impose shape information without assuming direct metric satellite-view depth supervision during training (Qian et al., 14 May 2026).

The third component is perspective-view training. In addition to orthographic satellite supervision and panorama supervision, the method samples narrow-field-of-view perspective crops from each ground-truth panorama and renders them from the same volume. With $40$9 denoting the standard perspective projection,

$19$0

the inclusion of perspective views is described as effectively multiplying the number of camera views and enforcing consistent geometry under strong viewpoint changes. The method also adds a depth smoothness regularizer,

$19$1

to suppress noise in predicted depth maps (Qian et al., 14 May 2026).

Taken together, these constraints define the paper’s central claim: geometric failures in satellite-to-street feed-forward generation can be materially reduced by modifying the inductive bias and supervision geometry of the model rather than abandoning the feed-forward tri-plane formulation.

4. Photometric supervision and combined objective

Sat3DGen does not optimize geometry in isolation. For each rendered view $19$2, whether satellite, panorama, or perspective, the model uses a photometric reconstruction and perceptual loss against the corresponding ground-truth image $19$3: $19$4 This term anchors appearance fidelity at the image level while the geometry-focused terms shape the underlying density field (Qian et al., 14 May 2026).

To encourage sharper and more realistic textures, particularly on outdoor facades, the framework includes a small Patch-GAN discriminator trained with hinge loss $19$5. On panoramas only, Sat3DGen additionally predicts residual transmittance $19$6 for sky versus volume and applies two sky-specific losses: a BCE sky-mask loss and an $19$7 color loss restricted to sky pixels. These terms are specialized to the fact that sky behavior differs from volumetric urban content and can otherwise distort panorama supervision (Qian et al., 14 May 2026).

The full objective sums all terms: $19$8 The structure of this objective makes the geometry-first designation precise. Geometry is not an emergent by-product of photometric fitting alone; it is directly optimized through explicit priors alongside appearance realism (Qian et al., 14 May 2026).

5. Benchmark construction and empirical results

For evaluation, Sat3DGen introduces a new benchmark by pairing the VIGOR-OOD held-out city, Seattle, with $19$9 m-resolution LiDAR-derived DSM tiles from King County, reprojected and aligned per satellite image. This benchmark is intended to quantify 3D accuracy rather than only image quality. The principal geometric metric is RMSE in meters,

$I_{\mathrm{sat}}$0

and photorealism is evaluated using FID,

$I_{\mathrm{sat}}$1

The paper also reports KID, DINO-based semantic similarity, PSNR, SSIM, and LPIPS for completeness (Qian et al., 14 May 2026).

On the VIGOR-OOD test set, the comparison highlighted in the paper is against Sat2Density++. Sat3DGen improves RMSE from $I_{\mathrm{sat}}$2 m to $I_{\mathrm{sat}}$3 m and reduces FID from approximately $I_{\mathrm{sat}}$4 to $I_{\mathrm{sat}}$5. The abstract emphasizes that the FID improvement is achieved without extra tailored image-quality modules. The reported interpretation is that the geometric leap also boosts photorealism, which is a notable result because image realism and geometric fidelity are often treated as competing objectives in satellite-to-street generation (Qian et al., 14 May 2026).

Ablation results are used to support the contribution of each geometric module. Removing the gravity loss raises FID to approximately $I_{\mathrm{sat}}$6; dropping the depth prior increases RMSE to $I_{\mathrm{sat}}$7 m; and disabling perspective training eliminates most gains. Qualitatively, Sat3DGen is reported to produce planar roofs, straight facades, no floaters, clear road markings, and semantically faithful tree belts, while Sat2Density++ yields bubbly surfaces and torn edges. Panorama and multi-camera video sequences are described as smooth and artifact-free (Qian et al., 14 May 2026).

6. Applications, scope, and distinction from similarly named work

Because the method outputs both a renderable NeRF and a watertight mesh, Sat3DGen is demonstrated in several downstream scenarios. In semantic-map-to-3D synthesis, an OpenStreetMap road and building footprint map is converted into color-coded satellite input via a diffusion ControlNet plus SDXL pipeline, after which Sat3DGen generates the 3D scene. In surround-view multi-camera video generation, a single satellite frame is used to generate four synchronized street-view streams for simulation or VR. In large-scale meshing, a $I_{\mathrm{sat}}$8 or $I_{\mathrm{sat}}$9 satellite mosaic is processed in a sliding-window fashion and per-tile meshes are fused into a contiguous 3D city block. In unsupervised DSM estimation, the method directly renders metric satellite-view depth with learned scale despite the absence of ground-truth depth in training (Qian et al., 14 May 2026).

These applications clarify the scope of the framework. Sat3DGen is not only a view synthesis model; it is also a 3D asset generator intended for meshing, simulation, and depth-related tasks. The paper further states that the code has been released, which positions the system as an implementable research artifact rather than solely a proof of concept (Qian et al., 14 May 2026).

A recurrent source of confusion is the similarity between the names Sat3DGen and SAT3D. The latter refers to “Image-driven Semantic Attribute Transfer in 3D,” a distinct method that edits semantic attributes from a reference image by manipulating style-code channels in a pre-trained StyleGAN-based 2D or 3D-aware generator such as EG3D. SAT3D focuses on attribute transfer tasks such as beard, smile, or hairstyle editing, using a Meta Attribute Mask Matrix and CLIP-based quantitative guidance, whereas Sat3DGen addresses comprehensive street-level 3D scene generation from a single satellite image (Zhai et al., 2024). The shared naming therefore does not indicate a shared task definition or architectural objective.