Urban Foundation Models (UFMs)

• Urban Foundation Models (UFMs) are large-scale, multimodal AI systems that integrate heterogeneous urban data for robust prediction, planning, and decision-making.
• They employ dedicated encoders and fusion layers to align text, images, graphs, and spatio-temporal signals, ensuring effective data fusion and representation.
• UFMs support zero- and few-shot adaptations, enabling real-time urban analytics and simulation with minimal fine-tuning and enhanced generalization.

Urban Foundation Models (UFMs) are large-scale, multimodal artificial intelligence models explicitly designed to ingest, encode, and reason with the heterogeneity, spatiality, and temporality of urban data. UFMs generalize the “pre-train once, adapt everywhere” paradigm, widely established in natural language and vision foundation models, to the structure and complexity of urban systems. Unlike task-specific models, UFMs are architected to support a broad array of urban prediction, reasoning, simulation, and decision-support tasks—ranging from climate resilience and urban planning to operational monitoring and generative design—via unified, transferable representations and parameter sharing across modalities, tasks, and geographies (Mai et al., 2023, Yuan et al., 2024, Zhang et al., 2024, Huang et al., 9 Nov 2025).

1. Conceptual Foundations and Motivations

The core motivation for Urban Foundation Models is the recognition that highly fragmented, scenario-specific models in urban computing are insufficient for the scale, diversity, and fusion needs encountered in smart cities. Urban systems inherently generate data streams in multiple granularities and modalities: text (planning documents, POI descriptions), tabular/graph (census, infrastructure), imagery (street-level, aerial, remote sensing), spatio-temporal time series (traffic, environmental sensors), geometric primitives (buildings, road networks), and explicit geocoordinates. UFMs aim to learn universal representations and cross-domain correlations across these data types, enabling robust zero-shot and few-shot generalization, real-time adaptation, and fair and privacy-conscious urban intelligence (Zhang et al., 2024, Mühlematter et al., 15 Oct 2025, Tan et al., 2023).

UFMs are rooted in three converging principles:

1. Multimodality-first design: Jointly process text, images, graphs, sensors, geometry, and coordinates.
2. Spatio-temporal reasoning: Encode spatial layouts, temporal dynamics, and hierarchical geographies using attention-based or graph-based mechanisms.
3. Unified pre-training and adaptation: Pre-train on heterogeneous, large-scale urban data, then adapt rapidly to new cities or tasks via fine-tuning, prompt learning, or minimal supervision (Yuan et al., 2024, Kreismann, 20 Sep 2025, Fleckenstein et al., 21 Oct 2025).

2. Data Modalities, Encoding Strategies, and Present Taxonomies

UFMs are classified along a data-centric taxonomy capturing five principal sources (Zhang et al., 2024):

Modality	Typical Data	Model Classes
Language	Geo-text, POI descriptions, social media	Masked LM, generative LLMs
Vision	Street view, remote sensing, maps, 3D models	Masked image models, contrastive encoders
Trajectory	GPS traces, road networks, check-ins	Sequence models, graph-based transformers
Time Series	Traffic, AQI, energy, sensor grids	Temporal transformers, contrastive/self-supervised
Multimodal	Joint text-image-graph-sensor combinations	Cross-modal contrastive, fusion transformers

For modality fusion, state-of-the-art models deploy independent encoders per modality—e.g., transformers (T5, PaLM) for text, ViT/CNN for imagery, MLP/GNN for tabular/graph, small MLPs on Fourier-transformed coordinates for geospatial inputs—and align their embeddings using fusion transformers, cross-modal attention, or memory-based prompt architectures (Mai et al., 2023, Mühlematter et al., 15 Oct 2025).

A key innovation is the use of stochastic multimodal masking and contrastive cross-modal objectives (e.g., CLIP-style losses) to ensure robust representational alignment and enable inference with arbitrary subsets of available modalities (Mühlematter et al., 15 Oct 2025).

3. Representative Model Architectures

Canonical UFM designs are characterized by the following structural elements:

• Multi-encoder backbone: Independent encoders for text ($E_{\text{text}}$), vision ($E_{\text{vis}}$), tabular/graph ($E_{\text{poi}}$), and geospatial ($E_{\text{geo}}$) data, each mapping inputs to a common latent space ($$z_{\text{text}},z_{\text{vis}},z_{\text{poi}},z_{\text{geo}}$$).
• Fusion layer: Lightweight transformer or cross-modal attention network to combine modality-specific representations; may use contrastive or shared memory retrieval mechanisms (Mai et al., 2023, Yuan et al., 2024, Mühlematter et al., 15 Oct 2025).
• Spatio-temporal module: Spatial and temporal positional encodings, often via rotary or sinusoidal schemes; spatial clustering for graph compression ($k$-means, MiniST units); or explicit attention alternation over spatial and temporal token groups (Yuan et al., 2024, Chen et al., 24 Feb 2026).
• Task heads: Universal prediction, generative, or regression heads for multi-task forecasting or reasoning.

Example loss terms include unimodal task loss, cross-modal contrastive loss, and spatial/geospatial alignment penalties enforcing proximity preservation in the embedding space (Mai et al., 2023). Generative diffusion or autoencoding objectives further regularize long-term, open-world transfer (Yuan et al., 2024, Imanov et al., 5 Feb 2026).

4. Urban Applications and Benchmarks

UFMs underpin a diverse range of tasks:

• Physical environment modeling: Urban heat island detection and mitigation simulation with minimal fine-tuning, outperforming baseline CNNs and demonstrating high spatial generalization and sub-1 °C MAE on land surface temperature (Fleckenstein et al., 21 Oct 2025, Kreismann, 20 Sep 2025).
• Urban resilience and risk forecasting: Diffusion–transformer architectures such as Skjold-DiT jointly predict flood/heat/structural vulnerabilities and transportation access, supporting counterfactual “what-if” interventions and uncertainty quantification (Imanov et al., 5 Feb 2026).
• Urban sensing: Automated waterlogging detection and assessment in real-time CCTV imagery using a mixture of vision and vision-language foundation models, with chain-of-thought prompting for structured report generation (Zhang et al., 21 Oct 2025).
• Spatio-temporal flow forecasting: Models such as UniFlow and UrbanDiT unify grid and graph-based representations to predict city-wide flows, achieving state-of-the-art results on both traffic and crowd datasets, robust few-/zero-shot transfer, and strong noise resilience (Yuan et al., 2024, Yuan et al., 2024, Chen et al., 24 Feb 2026).
• 3D urban modeling: BuildingWorld provides a planet-scale, diverse 3D building dataset with standardized evaluation (corner/edge metrics, Chamfer Distance, global IoU) for benchmarking foundation-driven digital twin and reconstruction pipelines (Huang et al., 9 Nov 2025).

Performance benchmarks are evaluated with standardized error metrics: macro-F1, accuracy, RMSE/MAE for regression, coefficient of determination ($R^2$), and fairness or bias metrics such as group-wise rank correlation (Wang et al., 18 Oct 2025).

5. Adaptation Strategies and Generalization

UFMs support multiple adaptation and deployment paradigms:

• Zero-shot inference: Models are directly prompted or queried without any urban task-specific fine-tuning, leveraging universal representations inherited from large-scale pre-training (Mai et al., 2023, Tan et al., 2023).
• Few-shot adaptation: Lightweight adapters, low-rank modules (LoRA), or meta-initialization (MAML) are added, with fine-tuning on 1–5 samples per class recovering up to 80–90% of the accuracy deficit with negligible parameter update cost (Mai et al., 2023).
• Full supervised fine-tuning: End-to-end training on downstream dataset, typically reserved for high-value or high-accuracy use cases.
• Data-efficient transfer: Minimal data requirements for robust cross-city generalization across climate, culture, and infrastructure (e.g., <0.6 °C MAE variation in urban temperature mapping) (Fleckenstein et al., 21 Oct 2025, Yuan et al., 2024).

Group-Relative Policy Optimization (GRPO) and fairness-aware reward design (combining accuracy with regional parity) are used to mitigate geo-bias and ensure equitable performance and trustworthiness across underrepresented geographies (Wang et al., 18 Oct 2025).

6. Challenges, Risks, and Best Practices

Key open challenges in UFM design include:

• Multimodality alignment: Difficulties in aligning heterogeneous spatial, visual, and semantic modalities, particularly in presence of spatial misregistration, variable temporal granularity, or data sparsity (Mai et al., 2023, Mühlematter et al., 15 Oct 2025).
• Scale and heterogeneity: Spatial and temporal scale mismatches and marginal distribution shifts, especially when transferring between cities, climates, or infrastructure types (Chen et al., 24 Feb 2026).
• Bias and fairness: Overrepresentation of affluent urban cores, dataset-induced skew, and the need for balanced, group-aware optimization (Wang et al., 18 Oct 2025).
• Privacy and governance: Risks of re-identification, especially when integrating mobility or street-view data; best practices require differential privacy, secure aggregation, and open dataset documentation.
• Operational robustness and evaluation: Rigorous spatial cross-validation (by geographic cluster, not random splits), robust noise augmentation, and the use of planet-scale, open benchmarks (e.g., BuildingWorld, POI100, EvalST) are recommended for trustworthy assessment (Huang et al., 9 Nov 2025, Fleckenstein et al., 21 Oct 2025, Chen et al., 24 Feb 2026).

Best practices include modular backbone design, spatially aware validation, explicit reporting of model biases, and privacy-by-design protocols (Mai et al., 2023, Zhang et al., 2024).

7. Prospects and Future Directions

Emerging research highlights the following directions for Urban Foundation Models:

• Hierarchical and continuous learning: Dynamic memory architectures and super-resolution heads for hierarchical urban analytics (e.g., sub-10 m microclimate) (Fleckenstein et al., 21 Oct 2025, Yuan et al., 2024).
• Agentic integration: Coupling UFMs to domain tools or APIs (e.g., GIS systems, traffic simulators) for agentic task orchestration and interactive planning (Zhang et al., 2024).
• Federated and privacy-preserving training: Federated and prompt-efficient learning under decentralized data and regulatory constraints (Zhang et al., 2024).
• Open-world and real-time deployment: Streaming pretraining and map-reduce strategies for low-latency, city-wide generative AI augmentation (Campo et al., 4 May 2025).
• Standardization of benchmarks and evaluation: Global, task-diverse datasets (EvalST, BuildingWorld, BCUR) and standard metrics for transparent comparison.

The ongoing development and deployment of UFMs is expected to underpin a new generation of data-driven, intelligent, and adaptive urban systems, supporting the transition toward Urban General Intelligence (Zhang et al., 2024, Wang et al., 18 Oct 2025, Huang et al., 9 Nov 2025).