WildCity: A Real-World City-Scale Testbed for Rendering, Simulation, and Spatial Intelligence

Abstract: Humans can navigate an unfamiliar city and gradually form a coherent spatial mental map spanning tens of square kilometers. Can AI build spatial representations at a comparable scale? Although recent foundation models have advanced scene reconstruction and embodied intelligence, scaling to entire cities remains an open challenge, primarily due to the lack of city-scale data. To bridge the gap, we introduce WildCity, a real-world multimodal dataset collected by autonomous fleets traversing complex urban environments. Our dataset includes 18 trajectories, each averaging 83.7 kilometers in length, and preserves the core challenges of in-the-wild perception, e.g., dynamic objects, lighting variations, and imperfect camera poses. We further establish an urban-tailored reconstruction baseline and convert the reconstructed environments into a closed-loop simulator. Beyond the dataset and baseline, we systematically analyze the key challenges on the path to simulation-ready urban digital twins: scalability, extrapolation, and uncertainty. Ultimately, WildCity aims to catalyze progress not only in city-scale rendering, but more broadly in the pursuit of AI that can perceive, remember, and reason across space at a scale comparable to human cognition. Project page: https://han-xiangyu.github.io/Wild-City/

• The paper introduces WildCity, a city-scale testbed offering over 1,500 km of continuous, multimodal urban data for advanced spatial AI benchmarking.
• It employs an urban-adapted 3D Gaussian Splatting method with rig pose optimization, sky modeling, and ground regularization to reduce geometric errors by up to 40%.
• The dataset supports closed-loop simulation and agent-based reasoning, facilitating research in long-horizon planning and realistic digital twin reconstruction.

WildCity: A Real-World City-Scale Testbed for Rendering, Simulation, and Spatial Intelligence

Motivation and Positioning

WildCity establishes a new urban-scale benchmark addressing critical gaps in current spatial AI research: the lack of high-fidelity, continuous real-world data at the true scale, and the corresponding limitations of current neural rendering and embodied intelligence methods outside toy or synthetic domains. Prior datasets either lack multimodal coverage, are geographically constrained, or are limited to short, disconnected scenes, failing to capture the complexity, uncertainty, and spatial drift challenges inherent in real city-scale operations.

WildCity responds by introducing a dataset collected by autonomous fleets traversing six major U.S. cities, providing 1,500+ km of continuous, multimodal logs (synchronized surround-view RGB, LiDAR, GPS, IMU) over eighteen long-horizon trajectories. This scale and diversity support research on spatial representation, urban rendering, simulation, and long-term planning, moving closer to the demands of practical robotic and AI systems operating in real cities.

Figure 1: Atlanta coverage illustrating the complexity and scale of WildCity's dataset.

Dataset Composition and Annotation

WildCity's sensor platform includes six surround cameras, LiDAR, IMU, and GPS, with rigorous cross-modal calibration and motion compensation to maximize geometric fidelity. Data is collected under diverse operational and environmental conditions—different cities, days, lighting, weather, and traffic—capturing the irreducible uncertainty of open-world driving. Trajectories average 83.7 km each, and the dataset comprises over 3 million keyframes, enabling evaluation at both local (meters) and global (dozens of kilometers) spatial scales.

A distinguishing feature is the provision of semantic masks to support region-aware reconstruction and downstream tasks. Using SAM3 with text prompts and 3D tracking cuboids, the pipeline generates ground, sky, and dynamic-object masks capable of distinguishing truly dynamic objects from stationary but movable categories. This enables more robust ground/surface modeling and dynamic-object filtering, achieving an overall 91.6% mIoU against human-labeled ground truth.

Figure 2: Procedure for generating semantic masks, including dynamic/static object filtering via 3D tracking cuboids.

Figure 3: Qualitative agreement between manual and automatic semantic masks for dynamic objects, sky, and ground.

Methodological Contributions: Urban-Scale 3DGS

The baseline built on 3D Gaussian Splatting (3DGS) is specifically adapted to city-scale, long-range, noisy, and unbounded urban environments. Urban-tailored design choices include:

• Rig pose optimization: Pose refinement exploits the rigid body structure of multi-camera vehicles, jointly optimizing ego-pose and all camera extrinsics relative to the vehicle frame. This corrects local and global misalignment that naively independent camera pose refinement cannot address.
• Dedicated sky modeling: A lightweight view-dependent MLP predicts the infinite background (sky), decoupling its appearance from the city geometry and preventing far-field floaters—a critical failure mode under miscalibrated or incomplete 3D representations.
• Ground regularization: Gaussian components identified as ground are regularized for vertical alignment, slice-wise height stability, and opacity, reducing underconstrained geometry in planar road regions with weak texture or sparse overlap.
• Robust extrapolation: View synthesis beyond the reference trajectory is stabilized by progressive render-repair-augment cycles using the Difix3D+ diffusion model, substantially mitigating floating artifacts and hallucinations.
• Distributed optimization: Training leverages multi-GPU Gaussian sharding and synchronized gradient updates for scalability to tens of millions of 3D primitives, essential for billion-pixel, kilometer-scale scenes.

City-Scale Simulation and Embodied Reasoning

WildCity's reconstructed digital twins are integrated with a closed-loop simulator, supporting agent-based embodied reasoning—navigation, perception, and planning—at urban scale. Vision-Language-Action models such as Alpamayo-1 demonstrate interactive decision making along extended routes, with continuous visual feedback, enabling the study and benchmarking of long-horizon planning, spatial memory, localization, and generalization in photorealistic environments.

Figure 4: Example of closed-loop simulation with vision-language-action agents navigating the reconstructed city environment.

Empirical Results

Comparative Performance

Against established baselines—3DGS, H-3DGS, CityGaussianV2, and VGGT-Long—WildCity's baseline achieves quantitatively superior geometric fidelity and competitive or superior 2D metrics, especially at scale. Notably, previous methods maintain high peak PSNR/SSIM on short trajectories but suffer catastrophic geometric drift (Depth L1 > 17m); WildCity's approach reduces geometric error by up to 40% (e.g., Depth L1 = 11.8m on short, 6.6m on long trajectories), demonstrating that current partitioning and hierarchical strategies alone are insufficient for global alignment and simulation readiness.

Qualitative Analysis

On-trajectory rendering shows sharper textures and more coherent edge structures, especially for surfaces that are underconstrained or composed of thin geometry. Extrapolated/off-trajectory synthesis, a requirement for interactive agent simulation, degrades significantly less than all baselines, confirming improved robustness in unobserved regions.

Figure 5: Qualitative rendering comparison for in-trajectory and extrapolated views. Diffusion-based post-repair further improves off-trajectory appearance.

Figure 6: Visualizations of off-trajectory extrapolation at 1m, 3m, and 5m offsets, comparing WildCity with leading baselines.

Scalability, Extrapolation, and Uncertainty

Quantitative analysis across scales reveals that rendering quality degrades as trajectory length increases, with more severe impact in real urban data relative to synthetic benchmarks, primarily due to compounding pose drift, appearance variation, and unmodeled uncertainty (e.g., dynamic objects, unstructured lighting, sensor noise). Ablation studies show that structural constraints (rigid pose, ground regularization, sky separation) are essential, with nontrivial trade-offs in local image metrics but marked improvements in 3D geometric stability.

Figure 7: Scalability analysis: real-world WildCity exhibits steep degradation in geometric and photometric metrics with increasing data scale, compared to synthetic scenes.

Figure 8: Ablation results: each module—ground regularization, sky model, rig pose optimization—plays a distinct role in mitigating real-world uncertainty and geometric drift.

Implications, Limitations, and Future Directions

WildCity sets a new standard for city-scale, real-world 3D dataset benchmarks, enabling direct study of the core obstacles in simulation-ready digital twin construction: geometric scalability, robust view extrapolation, and modeling in the presence of data uncertainty. The dataset's depth and diversity will facilitate advances in learned SLAM, neural rendering architectures with strong geometric priors, data-driven uncertainty quantification, and generalizable agent-based reasoning.

Key limitations remain: semantic masks, while highly accurate, are auto-generated and imperfect at boundaries/rare categories; absolute pose accuracy is GPS-limited, with sub-centimeter horizontal and centimeter vertical drift per kilometer; the Gaussian budget and VRAM demands increase linearly with trajectory length. Advances in pose supervision, efficiency, and scale will be required before high-fidelity, interactive simulation of entire metropolitan regions becomes routine.

WildCity is positioned to spur research beyond explicit rendering and reconstruction: spatial memory, persistent mapping, lifelong navigation, and long-term planning in embodied settings. Its release closes a critical gap between synthetic and real-world city-scale intelligence, and is expected to catalyze breakthroughs in generalizable spatial AI and robust, simulation-enabled embodied learning.

Conclusion

WildCity delivers a comprehensive, multimodal, city-scale dataset and testbed for the evaluation and development of neural rendering, simulation, and spatial reasoning algorithms at scales matching real urban environments. By combining methodological innovation, empirical benchmarking, and agent-centered simulation, it provides the necessary foundation for robust, generalizable AI systems that perceive, remember, and act across the true spatial and semantic complexity of cities (2607.06838).