Skyfall-GS: Synthesizing Immersive 3D Urban Scenes from Satellite Imagery (2510.15869v1)

Abstract: Synthesizing large-scale, explorable, and geometrically accurate 3D urban scenes is a challenging yet valuable task in providing immersive and embodied applications. The challenges lie in the lack of large-scale and high-quality real-world 3D scans for training generalizable generative models. In this paper, we take an alternative route to create large-scale 3D scenes by synergizing the readily available satellite imagery that supplies realistic coarse geometry and the open-domain diffusion model for creating high-quality close-up appearances. We propose \textbf{Skyfall-GS}, the first city-block scale 3D scene creation framework without costly 3D annotations, also featuring real-time, immersive 3D exploration. We tailor a curriculum-driven iterative refinement strategy to progressively enhance geometric completeness and photorealistic textures. Extensive experiments demonstrate that Skyfall-GS provides improved cross-view consistent geometry and more realistic textures compared to state-of-the-art approaches. Project page: https://skyfall-gs.jayinnn.dev/

• The paper’s main contribution is a scalable two-stage pipeline that synthesizes immersive 3D urban scenes solely from multi-view satellite imagery without needing street-level data.
• It combines 3D Gaussian Splatting with curriculum-based iterative refinement using diffusion models to enhance both geometric fidelity and texture realism.
• Quantitative evaluations show significant improvements over baselines with real-time rendering speeds and high perceptual quality, as confirmed by extensive user studies.

Skyfall-GS: Synthesizing Immersive 3D Urban Scenes from Satellite Imagery

Introduction and Motivation

Skyfall-GS introduces a scalable framework for synthesizing city-block scale, immersive 3D urban scenes using only multi-view satellite imagery as input. The method addresses the limitations of prior approaches, which either require costly 3D ground-truth data or rely on domain-specific semantic maps and height fields, resulting in oversimplified geometry and unrealistic textures. By leveraging the ubiquity and geographic coverage of satellite imagery, Skyfall-GS circumvents the need for street-level or 3D scan data, enabling the generation of realistic, navigable urban environments suitable for applications in simulation, robotics, and entertainment.

Figure 1: Skyfall-GS synthesizes high-fidelity 3D urban scenes from multi-view satellite imagery, enabling realistic drone-view navigation without additional 3D or street-level training data.

Methodology

Two-Stage Pipeline

Skyfall-GS employs a two-stage pipeline:

1. Reconstruction Stage: Initial 3D scene reconstruction is performed using 3D Gaussian Splatting (3DGS), enhanced with appearance modeling and regularization to address the challenges of multi-date satellite imagery and limited parallax. Camera parameters are approximated from RPC models using SatelliteSfM, and per-image/per-Gaussian embeddings are used to model illumination and appearance variations. Opacity regularization and pseudo-camera depth supervision (using MoGe) mitigate floating artifacts and improve geometric fidelity, especially in texture-less regions.
2. Synthesis Stage: The curriculum-based Iterative Dataset Update (IDU) refines the initial 3DGS by progressively introducing lower-elevation viewpoints. Rendered views are enhanced using prompt-to-prompt editing with a pre-trained text-to-image diffusion model (FLUX.1 via FlowEdit), which hallucinates plausible textures and geometry in occluded regions. Multiple diffusion samples per view are synthesized to promote 3D consistency and mitigate view-dependent artifacts.

   Figure 2: Overview of the Skyfall-GS pipeline: (a) 3DGS reconstruction with appearance modeling and depth supervision; (b) curriculum-based IDU refinement using diffusion models.

Curriculum Learning and Diffusion-Based Refinement

The curriculum learning strategy is motivated by the observation that 3DGS reconstructions from satellite imagery degrade at lower elevation angles due to occlusions and limited parallax. By gradually lowering the viewpoint across IDU episodes, the method incrementally exposes and refines previously occluded regions, improving both geometric completeness and texture realism.

Figure 3: Curriculum strategy motivation—render quality degrades with decreasing elevation angle in initial 3DGS reconstructions.

Prompt-to-prompt editing via FlowEdit enables targeted enhancement of degraded regions, guided by source and target prompts that describe the desired improvements. Multiple diffusion samples per view are used during optimization to ensure that the final 3DGS representation achieves a consensus across diverse denoising trajectories, promoting cross-view consistency.

Experimental Results

Quantitative and Qualitative Evaluation

Skyfall-GS is evaluated on the DFC2019 (WorldView-3) and GoogleEarth datasets, comparing against Sat-NeRF, EOGS, Mip-Splatting, CoR-GS, CityDreamer, and GaussianCity. Metrics include FID_CLIP, CMMD, PSNR, SSIM, and LPIPS, with distribution-based metrics prioritized due to the generative nature of the task.

Skyfall-GS consistently achieves the best scores across all metrics, with FID_CLIP reduced by a factor of 3–9 compared to baselines. On DFC2019, Skyfall-GS attains FID_CLIP of 27.35 (vs. 88.36 for Sat-NeRF), and on GoogleEarth, 9.91 (vs. 36.52 for CityDreamer). User studies with 89 participants confirm dominant winrates (≈97% on DFC2019, ≈90% on GoogleEarth) for geometric accuracy, spatial alignment, and perceptual quality.

Figure 4: Qualitative comparison—Skyfall-GS outperforms baselines in geometric accuracy and texture quality, especially in low-altitude novel views.

Ablation Studies

Ablations demonstrate the necessity of each component. Appearance modeling is essential for convergence with multi-date imagery. Opacity regularization and pseudo-camera depth supervision reduce floating artifacts and improve planar region geometry. In the synthesis stage, multiple diffusion samples and curriculum learning both significantly improve rendering quality and geometric coherence.

Figure 5: Ablation—opacity regularization, depth supervision, and multi-sample diffusion enhance density, geometry, and texture consistency.

Rendering Efficiency

Skyfall-GS achieves real-time rendering (11 FPS on NVIDIA T4, 40 FPS on MacBook Air M2), outperforming CityDreamer (0.18 FPS on A100) and matching GaussianCity (10.72 FPS on A100), demonstrating practical deployability on consumer hardware.

Implementation Details

• 3DGS Reconstruction: 30,000 iterations, densification between 1,000–21,000, scaling learning rate 0.001, densification gradient threshold 0.001, Gaussian pruning for covariance >20, loss weights $\lambda_{\text{D-SSIM}=0.2$, $\lambda_{\text{op}=10$, $\lambda_{\text{depth}=0.5$.
• Appearance Modeling: Per-image embedding (dim 32), per-Gaussian embedding (dim 24), MLP (2 layers, 128 neurons), learning rates 0.001–0.005.
• Pseudo-Camera Depth Supervision: 24 views every 10 iterations, elevations 80°–45°, radii 300–250 units, MoGe for scale-invariant depth.
• IDU Refinement: 5 episodes × 10,000 iterations, 9 look-at points (DFC2019), 16 (GoogleEarth), 6 cameras/point, 2 diffusion samples/view, elevations 85°–45°, FlowEdit noise 4–10, prompt engineering for artifact removal and texture enhancement.
• Training Time: 1 hour for reconstruction, 6 hours for synthesis on RTX A6000.

  Figure 6: Pseudo-camera sampling strategy for depth supervision—240 points sampled for robust geometry regularization.

Limitations and Future Directions

Skyfall-GS requires substantial computational resources for iterative refinement and produces over-smoothed textures at extreme street-level perspectives. Future work should focus on reducing computational overhead, improving street-level detail, and integrating robust geometric validation. Extension to dynamic scenes and larger-scale environments is a promising direction.

Implications and Outlook

Skyfall-GS demonstrates that high-fidelity, navigable 3D urban scenes can be synthesized from satellite imagery alone, without reliance on 3D ground-truth or semantic maps. This approach enables scalable virtual city creation for simulation, robotics, and entertainment, with strong generalization across diverse urban typologies and geographic regions. The integration of open-domain diffusion models for appearance refinement sets a precedent for leveraging foundation models in 3D scene synthesis, suggesting future research into multi-modal generative priors and more advanced curriculum strategies for occlusion recovery.

Figure 7: Multi-date satellite imagery—substantial appearance shifts (illumination, cloud cover, surface changes) challenge consistent 3D reconstruction.

Conclusion

Skyfall-GS establishes a robust framework for synthesizing immersive, real-time 3D urban scenes from multi-view satellite imagery, combining 3D Gaussian Splatting with curriculum-driven iterative refinement via diffusion models. The method achieves superior geometric and perceptual fidelity compared to state-of-the-art baselines, validated by both quantitative metrics and user studies. Its scalability, rendering efficiency, and independence from 3D ground-truth data position it as a practical solution for large-scale virtual city generation. Future research should address computational efficiency, street-level realism, and dynamic scene modeling to further advance the capabilities of satellite-based 3D synthesis.