Sat2City: 3D Urban Asset Generation

• Sat2City is a framework that produces explicit 3D city models from a single satellite image using cascaded latent diffusion and structured latent priors.
• It employs methods like triplet-bottleneck VAEs, sparse voxel grids, inverse sampling, and flow matching to convert overhead observations into detailed 3D meshes with PBR textures.
• The approach is validated through geometry and appearance benchmarks, achieving notable improvements in reconstruction accuracy and asset realism on both synthetic and real-world data.

Searching arXiv for Sat2City-related papers to ground the article in current literature. Sat2City denotes both a specific framework and, more broadly, an asset-centric branch of satellite-to-city generation research concerned with producing explicit 3D urban content from overhead observations. In the narrow sense, "Sat2City: 3D City Generation from A Single Satellite Image with Cascaded Latent Diffusion" formulates the task as generating a full 3D city from a single satellite-derived observation using sparse voxel grids, triplet-bottleneck VAEs, and cascaded latent diffusion (Hua et al., 6 Jul 2025). "Sat2City v2: Native 3D City Asset Generation from a Single Satellite Image" redefines the practical regime around real satellite-image-conditioned textured mesh generation by adapting a pretrained native structured-latent 3D foundation model to weakly aligned satellite–mesh pairs (Hua et al., 23 Jun 2026).

1. Definition and problem setting

Sat2City addresses the problem of inferring a reusable 3D urban asset from overhead input rather than merely synthesizing a few plausible rendered views. The original framework takes a single satellite-derived height image or elevated satellite observation and produces an explicit 3D city model with both geometry and appearance; the v2 system takes one orthorectified satellite crop and outputs an explicit textured 3D city mesh asset with geometry and baked PBR texture maps (Hua et al., 6 Jul 2025, Hua et al., 23 Jun 2026).

The task is intrinsically ill-posed. A top-down observation compresses a highly complex 3D urban scene into limited 2D evidence. Roof silhouettes do not uniquely determine façades, overhangs, or side-wall geometry; fine urban structures are underconstrained; and appearance cues are ambiguous because satellite imagery mainly observes roofs and top surfaces while the desired output includes facades, roads, vegetation, and other lateral or partially occluded elements. Sat2City v2 further emphasizes that real meshes derived from photogrammetry introduce fragmented surfaces, temporal mismatch, incomplete geometry, and reconstruction artifacts, so the problem is not only underdetermined but also noisy under realistic supervision (Hua et al., 23 Jun 2026).

A central conceptual distinction is that Sat2City is not framed as satellite-to-street-view synthesis, and it is not framed as a rendering proxy optimized only for image generation. Its target is an explicit 3D-native asset that can be meshed, exported, edited, and reused in downstream workflows such as digital twins, simulation, urban planning, and geospatial intelligence (Hua et al., 23 Jun 2026). This separates it from cross-view image synthesis, video synthesis, and rendering-oriented scene representations.

2. Original Sat2City: cascaded sparse-voxel latent diffusion

The original Sat2City pipeline begins from a satellite-view height map represented as a point-cloud-like structural prior $P_h$. During training, this conditioning signal is paired with a colorized point cloud $P_C \in \mathbb{R}^{N \times 6}$ sampled from artist-created 3D city meshes. The target city is voxelized into a sparse voxel grid $G$, with normals $A_N$ stored as geometric attributes and color supervised indirectly from $P_C$ (Hua et al., 6 Jul 2025).

Its architecture is organized around two major components: triplet-bottleneck VAEs and cascaded latent diffusion models. The latent decomposition is

$\mathcal{X}_0 = \{X_D, X_S, \{X_{Ck}\}_{k=0}^n\},$

where $X_D$ is a dense geometry latent, $X_S$ is a sparse geometry latent, and $\{X_{Ck}\}_{k=0}^n$ are multi-level appearance latents produced through Re-Hash (Hua et al., 6 Jul 2025). The cascade is explicitly staged. Dense geometry diffusion predicts $X_D$ from the satellite-derived condition and recovers global occupancy and layout. Sparse geometry diffusion predicts $P_C \in \mathbb{R}^{N \times 6}$0 conditioned on decoded geometry from the dense stage, refining occupied surfaces and recording voxel-pruning structure. Appearance diffusion then predicts hierarchical appearance latents conditioned on the geometric pruning structure so that color generation is aligned with the final sparse geometry (Hua et al., 6 Jul 2025).

The VAE design is not monolithic. Sat2City uses a dense neck for occupancy-aware geometry diffusion, a sparse neck for refined structure, and a Re-Hash neck for hierarchical appearance encoding. The reported dense-geometry VAE loss combines binary cross-entropy on occupancy, $P_C \in \mathbb{R}^{N \times 6}$1 on normals, and a KL term; the sparse-geometry-plus-appearance VAE adds an $P_C \in \mathbb{R}^{N \times 6}$2 loss on point colors. The reported weights are

$P_C \in \mathbb{R}^{N \times 6}$3

The paper states that the KL term is not applied to the appearance hierarchy because appearance is more variable than geometry and KL regularization there would overconstrain material learning (Hua et al., 6 Jul 2025).

Two mechanisms are central to the original framework. The first is Re-Hash, a hierarchical coarsening operation in the appearance bottleneck. Starting from the finest appearance latent $P_C \in \mathbb{R}^{N \times 6}$4, it constructs coarser sparse grids with

$P_C \in \mathbb{R}^{N \times 6}$5

and transfers features via trilinear interpolation,

$P_C \in \mathbb{R}^{N \times 6}$6

This provides coarse contextual information for smoother appearance transitions and more stable optimization at scene scale (Hua et al., 6 Jul 2025).

The second is inverse sampling, introduced because direct color supervision at voxel vertices is unstable. Instead of assigning colors directly to grid vertices, Sat2City decodes multilevel appearance features and samples them back at original point-cloud locations: $P_C \in \mathbb{R}^{N \times 6}$7 At inference, the same mechanism is applied at generated geometry vertices to obtain $P_C \in \mathbb{R}^{N \times 6}$8. This makes appearance supervision implicit and spatially smooth rather than a hard per-vertex labeling problem (Hua et al., 6 Jul 2025).

The diffusion formulation itself is standard latent diffusion with a forward process

$P_C \in \mathbb{R}^{N \times 6}$9

a reverse process parameterized by denoising U-Nets, and v-parameterization. The paper states that all three bottleneck grids use the same diffusion training structure, based on the 3D variant of the Dhariwal-Nichol/XCube backbone, with DDIM sampling at inference (Hua et al., 6 Jul 2025).

3. Synthetic supervision, benchmarks, and empirical profile of the original framework

A major component of the original Sat2City system is its synthetic paired training corpus. Artists create large 3D city meshes in Blender, and the authors sample 100 million points from mesh surfaces using CloudCompare. A synthetic height field is rendered in Blender by mapping elevation to grayscale under a vertically aligned orthographic camera. The reported rendering characteristics are an image resolution of $G$0 pixels, covered land area $G$1, and ground sampling distance of 0.93 $G$2 per pixel as written in the paper. The aligned city data are cropped into $G$3 pixel segments while preserving point-cloud/height-map correspondence (Hua et al., 6 Jul 2025).

The resulting dataset contains 3110 instances in total, with 10% reserved for testing and validation. Each cropped point cloud contains between 174,000 and 5.9 million points. The rationale for this synthetic route is explicit: Sat2City requires direct 3D supervision for both geometry and appearance at city scale, whereas prior urban generation datasets generally do not offer this combination (Hua et al., 6 Jul 2025).

Evaluation is split between geometry metrics and perceptual assessment. For geometry, the paper reports MMD and COV, each with Chamfer Distance (CD) and Earth Mover’s Distance (EMD), sampling 10,000 points from generated meshes and reference point clouds. Sat2City obtains MMD-CD $G$4, MMD-EMD $G$5, COV-CD $G$6, and COV-EMD $G$7. The paper highlights a 98.1% improvement in COV-CD over the previous SOTA, a 7.8% improvement in COV-EMD, a 49.4% reduction in MMD-CD relative to BlockFusion, and a 37.9% reduction in MMD-EMD (Hua et al., 6 Jul 2025).

For appearance and structural completeness, the reported user study involves 60 participants and four scores: TPQ, TSC, GPQ, and GSC. Sat2City achieves TPQ 7.35, TSC 8.03, GPQ 6.27, and GSC 7.02, outperforming both the reconstructed 3D outputs of CityDreamer and the retrained Sat2Scene baseline in the reported comparison (Hua et al., 6 Jul 2025).

The ablations align with the method’s design claims. The bottleneck study indicates that a unified geometry+appearance latent causes severe gradient conflicts, that dual-stage training improves optimization, and that Re-Hash further stabilizes and accelerates convergence. The inverse-sampling ablation shows visible artifacts when direct color assignment or splatting is used instead. The cascade ablation reports chaotic generation without the dense stage and inconsistent appearance decoding without the sparse stage. The comparison to Sat2Scene is used to argue that sparse latent voxels scale better than dense point-based color generation under the paper’s city-scale conditions, especially because successful Sat2Scene generation is described as needing roughly 400 points/m², whereas the Sat2City dataset has only around 14 points/m² (Hua et al., 6 Jul 2025).

4. Sat2City v2: native structured-latent 3D generation

Sat2City v2 is presented as both an extension of and a substantial redesign of the original framework. The core shift is from a synthetic, height-map-conditioned, task-specific sparse-voxel latent diffusion system to a real-world, satellite-image-conditioned framework built by adapting a pretrained native structured-latent 3D foundation model, instantiated with TRELLIS.2 (Hua et al., 23 Jun 2026).

The key design argument is that weakly aligned real satellite–mesh data are too noisy to support learning a good 3D representation from scratch. Instead of training a task-specific VAE directly on noisy city meshes, Sat2City v2 encodes each mesh into a pretrained native 3D latent space containing decoupled latent spaces for shape and material, Sparse Compression VAEs, and O-Voxels for local dual-grid geometry and surface attributes (Hua et al., 23 Jun 2026). This changes the learning problem from “discover the full latent manifold from noisy supervision” to “adapt satellite conditioning into a pretrained asset manifold.”

The v2 pipeline is explicitly factorized into geometry adaptation and geometry-aware appearance synthesis: $G$8 and

$G$9

Here $A_N$0 is the input satellite image, $A_N$1 is a frozen image encoder, $A_N$2 are satellite conditioning tokens, $A_N$3 is a frozen sparse-structure generator producing active sparse coordinates $A_N$4, $A_N$5 is the fine-tuned geometry flow, $A_N$6 and $A_N$7 are frozen geometry decoder and encoder, and $A_N$8 and $A_N$9 are frozen appearance flow and decoder (Hua et al., 23 Jun 2026).

Only $P_C$0 is adapted on the Sat2City v2 dataset. This is a decisive methodological difference from the original Sat2City, where the representation itself was learned task-specifically. The satellite crop is resized to $P_C$1, and a frozen DINOv3-L encoder extracts patch-token features $P_C$2. A frozen sparse-structure generator predicts a binary coarse occupancy grid whose active cells define $P_C$3, the support for shape-latent tokens (Hua et al., 23 Jun 2026).

Geometry generation uses conditional rectified flow matching rather than diffusion. For each training mesh, the frozen geometry encoder produces a raw shape latent $P_C$4, normalized by pretrained latent statistics: $P_C$5 A noisy intermediate latent is constructed as

$P_C$6

with $P_C$7 and $P_C$8. The target velocity is

$P_C$9

and the reported geometry loss is

$\mathcal{X}_0 = \{X_D, X_S, \{X_{Ck}\}_{k=0}^n\},$0

Cross-attention injects satellite tokens into the geometry flow, but unlike the v1.5 diagnostic route described in the paper, v2 does not pseudo-project image features into voxel columns (Hua et al., 23 Jun 2026).

After geometry decoding, the generated mesh is re-encoded and used as an anchor for appearance generation. The evolving material latent is concatenated channel-wise with encoded shape latent, while the same satellite tokens condition the frozen material flow through cross-attention. The material decoder outputs sparse PBR fields—base color, metallic, roughness, and alpha—which are baked into texture maps. The paper states that the appearance stage re-encodes the generated mesh at resolution 1024 and exports $\mathcal{X}_0 = \{X_D, X_S, \{X_{Ck}\}_{k=0}^n\},$1 texture maps (Hua et al., 23 Jun 2026).

Implementation details are unusually concrete. The geometry flow has 1.3B parameters, latent resolution is derived from a $\mathcal{X}_0 = \{X_D, X_S, \{X_{Ck}\}_{k=0}^n\},$2 O-Voxel grid, training runs for 30,000 steps with AdamW, learning rate $\mathcal{X}_0 = \{X_D, X_S, \{X_{Ck}\}_{k=0}^n\},$3, weight decay 0.01, bfloat16, adaptive gradient clipping, EMA 0.9999, $\mathcal{X}_0 = \{X_D, X_S, \{X_{Ck}\}_{k=0}^n\},$4, classifier-free guidance condition drop probability 0.1, per-GPU batch size 4 split into 2 microbatches, and hardware of 4 NVIDIA A800-SXM4-80GB GPUs. At most 8192 geometry-latent tokens per scene are allowed, and 7 oversized samples are excluded for this reason (Hua et al., 23 Jun 2026).

5. Real-world paired data and benchmark results in Sat2City v2

A central contribution of Sat2City v2 is a real-world dataset of geographically matched satellite-image–textured-mesh pairs. Each sample is defined as

$\mathcal{X}_0 = \{X_D, X_S, \{X_{Ck}\}_{k=0}^n\},$5

where $\mathcal{X}_0 = \{X_D, X_S, \{X_{Ck}\}_{k=0}^n\},$6 is an orthorectified satellite image, $\mathcal{X}_0 = \{X_D, X_S, \{X_{Ck}\}_{k=0}^n\},$7 is a textured 3D city mesh, and $\mathcal{X}_0 = \{X_D, X_S, \{X_{Ck}\}_{k=0}^n\},$8 is a shared latitude-longitude bounding box (Hua et al., 23 Jun 2026). The alignment is geographic but weak: the modalities correspond to the same geographic crop but are not pixel-perfectly aligned and may differ in time, geometry, and reconstructed detail.

The reported scale is 16,241 total satellite–mesh pairs across 24 regions in 9 cities. Of these, 14,658 are paired training samples after validity checks, 14,651 are usable geometry-flow training pairs after preprocessing and token filters, and 1,590 are test pairs used exclusively for testing. The test split includes New York, Philadelphia, and an unseen-city subset from Atlanta (Hua et al., 23 Jun 2026).

Collection uses Blender 4.2.1, Blender Python API, and the Blosm add-on. The stated procedure is to define a regional latitude-longitude search window, partition it into non-overlapping city-block crops using latitude-dependent km-to-degree conversion, import both Google 3D Tiles mesh and satellite overlay under the same coordinates, standardize materials, and export a scene package. Each retained package contains the satellite-textured-mesh pair, derived point-cloud and height-map representations, and rendered perspective views from virtual cameras. The camera set includes 50 views with $\mathcal{X}_0 = \{X_D, X_S, \{X_{Ck}\}_{k=0}^n\},$9 poses along a hemispherical trajectory around the mesh (Hua et al., 23 Jun 2026).

Evaluation is divided into three benchmarks. The first is metric-scale DSM reconstruction, following Sat3DGen on 2,882 satellite–DSM pairs from the unseen Seattle split of VIGOR. Sat2City v2 obtains MAE 3.36, RMSE 4.74, 70.67% of pixels with error $X_D$0 m, and 89.05% with error $X_D$1 m, ranking first on all four reported metrics against Sat2Density++, Sat3DGen, and TRELLIS.2 (Hua et al., 23 Jun 2026).

The second is a generated-mesh geometry benchmark on 1,590 held-out SatCity scenes, including unseen Atlanta. The reported metrics are MMD (CD), MMD (EMD), COV (CD), COV (EMD), Normal PSNR, and Normal LPIPS. Sat2City v2 achieves 0.0038 / 0.0117 / 97.4 / 84.3 / 14.84 / 0.1186. It is best on all reported geometry metrics except Normal LPIPS, where TRELLIS.2 is slightly better at 0.1099 versus 0.1186 (Hua et al., 23 Jun 2026).

The third is a satellite-conditioned appearance benchmark on the same 1,590 held-out scenes, using OpenAI CLIP ViT-L/14 over RGB and normal renders from four fixed orbit views. Sat2City v2 reaches CLIP 0.7342 and CLIP-N 0.6849, again leading the compared baselines (Hua et al., 23 Jun 2026). The paper attributes these gains to satellite-domain adaptation over a pretrained native 3D prior rather than to learning a new latent representation from noisy city meshes.

6. Position within related satellite-to-city research

Sat2City occupies one branch within a broader literature that includes cross-view image synthesis, trajectory-conditioned video synthesis, renderable 3D scene generation, and building-centric urban content generation. The differences are not terminological only; they concern representation, supervision, and intended output.

"Sat2Vid: Street-view Panoramic Video Synthesis from a Single Satellite Image" synthesizes temporally and geometrically consistent street-view panoramic video from a single satellite image and a camera trajectory by explicitly creating a 3D point-cloud representation and maintaining dense 3D–2D correspondences across frames (Li et al., 2020). The output is video, not an explicit reusable city asset. "Sat2Scene: 3D Urban Scene Generation from Satellite Images with Diffusion" moves closer to 3D scene generation by taking fixed geometry, generating point-wise RGB colors with a 3D sparse diffusion model, converting the result into per-point neural features, and rendering arbitrary views with a Point-NeRF-style renderer (Li et al., 2024). This produces a feed-forward renderable scene representation, but geometry is not generated from scratch; it is assumed to be inferred from satellite imagery beforehand.

"Sat2RealCity: Geometry-Aware and Appearance-Controllable 3D Urban Generation from Satellite Imagery" adopts a building-centric decomposition rather than a monolithic city block generator. It uses OSM-based spatial priors, a dual-pathway appearance-control mechanism, and an MLLM-powered semantic-guided generation pipeline. Buildings are generated one by one and assembled back into a city using original OSM coordinates, and the underlying generator can decode into NeRF, 3D Gaussian Splatting, and mesh (Kang et al., 14 Nov 2025). Relative to Sat2City, this suggests a different resolution of the city-scale problem: entity-wise composition instead of city-scale sparse-voxel or native-latent asset generation.

Earlier work on satellite-to-street synthesis for geo-localization occupies yet another point in the design space. "Coming Down to Earth: Satellite-to-Street View Synthesis for Geo-Localization" jointly learns a conditional GAN-based synthesis module and a retrieval branch so that the generator encoder features are directly reused for cross-view retrieval (Toker et al., 2021). That work is highly relevant to the broader satellite-to-city theme, but its goal is panoramic street-view synthesis and localization rather than explicit 3D asset construction.

A common misconception is therefore to treat all satellite-to-ground or satellite-to-city methods as instances of a single problem. The literature in the supplied sources separates at least four targets: synthesized street panoramas, panoramic videos along a trajectory, renderable 3D scene representations for arbitrary views, and explicit reusable 3D city assets. Sat2City is most precisely associated with the last of these (Hua et al., 6 Jul 2025, Hua et al., 23 Jun 2026).

7. Limitations, misconceptions, and likely directions

The original Sat2City explicitly acknowledges the lack of full evaluation on real-world satellite imagery paired with real 3D city models. Its training domain is synthetic, its conditioning signal is height-map-like rather than native satellite RGB, and the paper identifies finite output resolution and compute cost as practical constraints (Hua et al., 6 Jul 2025). Sat2City v2 addresses several of these points but does not eliminate all of them.

The stated limitations of Sat2City v2 are a weak alignment ceiling, dataset bias and limited coverage, temporal mismatch and reconstruction artifacts, and view-level photorealism that remains limited because the material prior is frozen and satellite-to-surface alignment is weak (Hua et al., 23 Jun 2026). The paper also notes that stronger metric accuracy would require mapping assets with imagery, camera geometry, and 3D reconstruction maintained in one coordinate system, which are difficult to release at scale.

Another misconception is to read Sat2City as a pure reconstruction system. Both versions are generative. The original model learns explicit geometry and appearance priors from synthetic 3D supervision; Sat2City v2 maps satellite evidence into a pretrained 3D asset manifold and then performs geometry-conditioned texturing. In both cases, the output is intended to be plausible and satellite-consistent rather than survey-grade exact (Hua et al., 6 Jul 2025, Hua et al., 23 Jun 2026). This suggests that Sat2City is best understood as a controllable 3D-native generation framework with geospatial conditioning, not as deterministic inversion of overhead imagery.

The trajectory from the original framework to v2 also clarifies an architectural lesson. The first system used task-specific sparse-voxel latent hierarchies, Re-Hash, and inverse sampling to make city-scale 3D generation feasible under synthetic supervision. The second system preserves the geometry-to-appearance cascade but replaces the learned-from-scratch latent space with a pretrained native structured-latent prior, arguing that real weakly aligned data are better used to adapt conditioning than to define the representation itself (Hua et al., 23 Jun 2026). A plausible implication is that future Sat2City-style systems will continue to combine explicit asset generation with stronger geospatial priors, better-aligned real datasets, and hybrid representations that join mesh-native outputs with rendering-oriented layers.