Urban In-Context Learning
- Urban In-Context Learning is a paradigm that uses inference-time urban context (masks, prompts, multimodal metadata) to drive predictive behavior.
- It unifies approaches like masked diffusion in urban profiling and prompt-conditioned spatio-temporal learning to handle heterogeneous data across cities.
- The framework reduces the need for city-specific retraining and enhances fairness, robustness, and transferability in urban reasoning systems.
Searching arXiv for the specified topic and papers to ground the article in current literature. Urban In-Context Learning denotes a family of urban machine-learning paradigms in which adaptation is achieved from inference-time context rather than from task- or city-specific retraining. In current work, that context may be observed region values and masks in urban profiling, prompt tokens summarizing dataset- and task-specific structure in spatio-temporal foundation models, multimodal metadata such as satellite imagery, coordinates, address strings, nearby POIs, and street-view images in urban reasoning systems, or scene context and goals in urban driving (Zhang et al., 5 Aug 2025, Yuan et al., 2024, Feng et al., 29 Jun 2025, Guo et al., 2023). The term therefore spans several related but non-identical formulations: masked autoencoding over urban regions, prompt-conditioned diffusion transformers, instruction-tuned multimodal LLMs, retrieval-augmented and tool-augmented urban assistants, and context-conditioned control policies.
1. Historical emergence and conceptual scope
Early urban uses of context-rich adaptation appeared before the phrase itself was formalized. In urban region profiling, UrbanCLIP introduced an image-language pipeline in which each satellite tile is paired with an automatically generated textual description, and a CLIP-like model is trained with contrastive loss and language modeling loss so that language becomes part of the representation used for downstream urban tasks (Yan et al., 2023). In urban driving, CCIL proposed a context-conditioned imitation-learning formulation in which the policy maps context state into the ego vehicle’s future trajectory and explicitly avoids dependence on the ego vehicle’s own past state, framing robust driving as behavior inferred from scene context, map structure, other agents, and goal (Guo et al., 2023).
A more explicit urban foundation-model formulation appeared in UrbanDiT, which described a single generative model steered by prompts and context—rather than by task- or city-specific fine-tuning—to perform many urban spatio-temporal tasks across heterogeneous data sources (Yuan et al., 2024). The phrase was then formalized most directly in a one-stage urban profiling framework that unified pretraining and inference through masked autoencoding and diffusion over urban regions, with observed regions acting as in-context examples (Zhang et al., 5 Aug 2025). In parallel, instruction-tuned urban MLLMs such as UrbanLLaVA extended the idea to structured geospatial data, trajectories, satellite images, and street-view imagery, while Urban-R1 emphasized context-sensitive multimodal reasoning and geo-bias mitigation through reinforcement-learning-based post-training (Feng et al., 29 Jun 2025, Wang et al., 18 Oct 2025).
The literature therefore does not use the phrase in a single narrow sense. One line of work treats Urban In-Context Learning as a one-stage masked-diffusion predictor over regions (Zhang et al., 5 Aug 2025). Another treats it as prompt-based conditioning for open-world urban spatio-temporal learning (Yuan et al., 2024). A third treats it as instruction-following multimodal urban reasoning supported by prompts, retrieved knowledge, and examples (Li et al., 2 Apr 2025, Feng et al., 29 Jun 2025). A plausible implication is that the phrase now functions as an umbrella concept for urban systems whose behavior is shaped primarily by context tokens, masks, multimodal evidence, or retrieved knowledge supplied at inference time.
| System | Context carrier | Urban domain |
|---|---|---|
| UrbanCLIP | image-text pairs and generated textual descriptions | urban region profiling |
| UrbanDiT | data-driven prompts, task-specific prompts, and masks | urban spatio-temporal learning |
| UIC with UMDT | observed region values and mask vector | urban profiling |
| UrbanLLaVA | instruction prompts over text and images | urban intelligence |
| Urban-R1 | multimodal urban prompts and GRPO rollouts | urban reasoning across regions |
| CCIL | context state and goal-oriented scene representation | urban driving |
2. Formal definitions and theoretical structure
The clearest formal definition is given in the one-stage urban profiling framework of Urban In-Context Learning. Instead of the conventional two-stage pipeline—first learning region embeddings and then training a separate linear probe or shallow predictor—the framework uses a single pretrained model with fixed parameters to directly predict unknown region values from observed regions and their values, without parameter updates (Zhang et al., 5 Aug 2025). Its inference interface is
where the model learns from observed regions in context. Pretraining and inference share the same masked autoencoding form: some regions are masked, unmasked regions provide the conditioning signal, and the model reconstructs the unknown values (Zhang et al., 5 Aug 2025).
That paper is explicit about its analogy to GPT-style in-context learning. GPT unifies pretraining and inference through next-token prediction; Urban In-Context Learning instead unifies them through masked autoencoding over regions because urban data are not natural sequences and region values are continuous rather than discrete tokens (Zhang et al., 5 Aug 2025). This formulation is central because it reframes urban profiling from “representation learning plus downstream regressor” into a direct conditional prediction problem.
A complementary theoretical vocabulary comes from work on in-context environment learning in world models. That framework distinguishes Environment Recognition (ER), where context is used to identify which known environment is active and then apply an environment-specific model, from Environment Learning (EL), where context acts as a dataset from which the model infers the local transition law directly (Wang et al., 26 Sep 2025). Although that analysis is not urban-specific, it maps naturally onto urban settings in which an environment may correspond to a city, neighborhood, or regime such as rush hour, weekend demand, or a policy configuration (Wang et al., 26 Sep 2025). This suggests a useful distinction within urban research: some systems behave as urban environment recognizers, while others behave as urban environment learners.
The world-model analysis also defines in-context adaptation as monotonic improvement with longer context: where is the context sequence and is a distributional error metric (Wang et al., 26 Sep 2025). A plausible implication is that urban in-context systems should be judged not only by zero-shot performance, but also by whether prediction quality improves as more city-specific evidence is placed into context.
3. Masked diffusion and prompt-conditioned spatio-temporal learning
The strongest non-language realization of Urban In-Context Learning is the masked-diffusion urban profiling framework. Its core model, the Urban Masked Diffusion Transformer, treats an urban profile as a set of region-wise values and uses a mask vector to distinguish observed from unknown regions (Zhang et al., 5 Aug 2025). Unmasked regions carry clean values, masked regions carry noisy values, and the initial embedding for region is
with 0 a region embedding and 1 a global value vector (Zhang et al., 5 Aug 2025). Reverse diffusion then reconstructs masked regions conditioned on observed ones, and inference fixes observed regions to their true values while unknown ones are initialized from Gaussian noise (Zhang et al., 5 Aug 2025).
A distinctive addition is the Urban Representation Alignment Mechanism, which aligns the model’s intermediate features with reference embeddings from a classical urban profiling method such as UrbanVLP. The alignment loss is a cosine-similarity objective,
2
and is used because diffusion training on urban datasets with only a few hundred regions can be unstable (Zhang et al., 5 Aug 2025). Empirically, the one-stage method outperformed two-stage baselines across Manhattan and Chicago; for Manhattan house-price prediction, the best baseline MAE was 3 while the proposed method achieved 4, and PCC increased from 5 to 6 (Zhang et al., 5 Aug 2025). The same study reported that removing diffusion or alignment degraded performance, indicating that both distributional prediction and alignment contributed materially (Zhang et al., 5 Aug 2025).
UrbanDiT generalizes the context-conditioning idea from region-level profiling to open-world spatio-temporal learning across heterogeneous datasets. It converts both grid-based and graph-based data into a common sequential token space, then prepends prompt tokens that summarize dataset-specific and task-specific structure (Yuan et al., 2024). The model supports five tasks—forward prediction, backward prediction, temporal interpolation, spatial extrapolation, and spatio-temporal imputation—and unifies them as partially observed spatio-temporal fields distinguished by different masking patterns (Yuan et al., 2024).
Its data-driven prompts are retrieved from three learnable memory pools corresponding to time-domain, frequency-domain, and spatial patterns, while the task-specific prompt is derived from the mask tensor itself (Yuan et al., 2024). The core conditional input is
7
so observed locations are fixed to clean values and unobserved locations remain noisy for denoising (Yuan et al., 2024). Prompt tokens are concatenated with data tokens: 8 This architecture made a concrete empirical case for prompt-based urban adaptation. The model’s zero-shot performance on PopSH and TaxiBJ surpassed most baselines that were fully trained on the target data, and prompt ablations showed that removing any prompt increased RMSE, with the frequency-domain prompt producing the largest degradation (Yuan et al., 2024). Representative gains include TaxiBJ forward-prediction MAE 9 versus 0 for UniST and 1 for CSDI, and spatial extrapolation TaxiBJ MAE 2 versus 3 for CSDI and 4 for ImputeFormer (Yuan et al., 2024).
UrbanDiT and masked-diffusion urban profiling therefore represent two related but different non-language forms of Urban In-Context Learning. In one, context is region masking plus observed values; in the other, context is a prompt-and-mask specification over spatio-temporal tokens. Both replace task-specific fine-tuning with conditional inference from a shared model.
4. Multimodal language-centered urban reasoning
Language-centered Urban In-Context Learning entered the field first through representation learning and then through instruction-tuned multimodal reasoning. UrbanCLIP asked whether textual modality could enhance urban region profiling and answered by generating a detailed textual description for each satellite image using an open-source image-to-text LLM, then training on image-text pairs with contrastive loss and language modeling loss (Yan et al., 2023). Its image encoder is a ViT with a learnable 5 token and attention pooler; its text side is a decoder-only Transformer that also supports generative supervision (Yan et al., 2023). The joint objective is
6
Across three urban indicators in four Chinese metropolises, the method reported an average improvement of 7 on 8 compared to the state of the art, with the largest average 9 gain over the best baseline on carbon emissions at 0 (Yan et al., 2023). Cross-city transfer results showed UrbanCLIP outperforming PG-SimCLR on 1 out of 2 source-target pairs, with average 3 about 4 versus 5 (Yan et al., 2023). The work is significant because text here acts as contextual supervision rather than as a downstream label.
UrbanLLaVA extends the idea from language-enhanced representation learning to a domain-specialized MLLM for urban intelligence. Built primarily on VILA 1.5 at 6B parameters, it supports text, street-view images, satellite images, structured geospatial data converted into natural language, and trajectory data (Feng et al., 29 Jun 2025). Its instruction corpus, UData, is organized into location view, trajectory view, and global view, spanning tasks such as GeoQA, STV-Address, STV-Landmark, SAT-Address, SAT-Landuse, SceneComp, ImgRetrieval, CameraLoc, TrajPredict, and Navigation (Feng et al., 29 Jun 2025). Training uses a three-stage schedule—task alignment learning, knowledge learning, and mixture learning—designed to decouple spatial reasoning enhancement from domain knowledge learning (Feng et al., 29 Jun 2025).
The article’s relevance to Urban In-Context Learning lies in its prompt interface. All tasks are cast into instruction-style chat format with user messages containing images as tokens and assistant messages containing answers, often with chain-of-thought reasoning (Feng et al., 29 Jun 2025). Cross-city experiments showed that a model trained only on Beijing improved performance on London and New York across all task groups relative to the untuned base VILA1.5-8B, with especially large gains for challenging aspects such as trajectory and regional tasks (Feng et al., 29 Jun 2025). On the three-city benchmark, UrbanLLaVA was best on most urban tasks, including large margins in Beijing such as 7 on GeoQA, 8 on Geo+Traj, and 9 on Geo+SS relative to the base VILA1.5-8B (Feng et al., 29 Jun 2025). A plausible implication is that multi-modal instruction tuning creates a general urban prompt interface on top of which more explicit few-shot urban adaptation could be built.
A broader survey of LLMs in urban computing makes the same point from a systems perspective. It treats prompt engineering, chain of thought, instruction tuning, PEFT, and retrieval-augmented generation as core mechanisms for processing urban context at inference time (Li et al., 2 Apr 2025). Across traffic prediction, public safety, mobility, environmental monitoring, travel planning, urban planning, smart energy, geoscience, and autonomous driving, it repeatedly describes a pattern in which numerical, spatial, temporal, or visual urban data are textualized or retrieved into prompts so that a mostly fixed LLM adapts to the immediate task and city context (Li et al., 2 Apr 2025).
5. Alignment, bias mitigation, and control-oriented variants
Urban-R1 situates Urban In-Context Learning within the broader agenda of Urban General Intelligence. Its base system is Qwen2.5-VL-7B-Instruct, post-trained with Group Relative Policy Optimization on the proxy task of Urban Region Profiling, where inputs include satellite image 0, location information 1, and auxiliary text 2, and the model predicts a scalar indicator such as GDP, carbon emissions, population, poverty, or house price (Wang et al., 18 Oct 2025). The reward combines normalized absolute error with a formatting reward: 3 GRPO constructs multiple rollouts per prompt, computes group-normalized advantages,
4
and updates the policy with a clipped PPO-style objective plus KL regularization to a reference model (Wang et al., 18 Oct 2025).
This is not few-shot prompting in the classical sense. The paper states that URP training uses instruction prompts with single instances, and downstream tasks are evaluated in zero-shot settings without example–answer pairs in the prompt (Wang et al., 18 Oct 2025). Its relevance to Urban In-Context Learning is instead that reinforcement learning is used to make the model actually use context rather than fall back on geospatial priors. The paper argues that reducing geospatial bias and increasing cross-region generalization are prerequisites for in-context urban reasoning, because a model that heavily depends on EU/US priors will not use contextual clues about underrepresented regions effectively (Wang et al., 18 Oct 2025). On unseen regions, Urban-R1 achieved Spearman 5 for GDP, 6 for poverty, and 7 for house price, compared with GPT-4o at 8, 9, and 0, respectively (Wang et al., 18 Oct 2025). A plausible implication is that context sensitivity is not only an architectural property but also an alignment property.
A control-oriented variant appears in CCIL for autonomous urban driving. The policy is explicitly context-conditioned: instead of predicting ego action from both ego and context states, it maps context state into the ego vehicle’s future trajectory (Guo et al., 2023). Context includes HD map features, other agents, and ego goal, but excludes ego past trajectory (Guo et al., 2023). The method also uses an ego-perturbed goal-oriented coordinate system in which the origin is the ego vehicle’s position plus a zero-mean Gaussian perturbation and the 1-axis points toward the goal (Guo et al., 2023). The training objective predicts a horizon of future poses from a history of context states and adds auxiliary multi-horizon prediction plus 2 regularization to encourage stability (Guo et al., 2023).
Its empirical results were unusually strong in closed-loop evaluation. On Lyft, CCIL reduced collision rate to 3, off-road rate to 4, and L2 error to 5 m, compared with Vector-Chauffeur at 6, 7, and 8 m (Guo et al., 2023). On nuPlan, it achieved 9 collision, 0 off-road, 1 discomfort, and 2 m L2, all substantially better than the reported baselines (Guo et al., 2023). Although the paper does not use the term Urban In-Context Learning, it is a concrete example of behavior inferred from structured urban context rather than from ego-state continuation.
6. Misconceptions, evaluation criteria, and open research directions
A recurrent misconception is that Urban In-Context Learning must resemble the classical few-shot prompt format of LLMs. The literature does not support that restriction. UrbanDiT does not consume example–label pairs in context; it uses learned prompt vectors and mask patterns (Yuan et al., 2024). The masked-diffusion profiling framework uses observed regions and their values as in-context examples rather than textual demonstrations (Zhang et al., 5 Aug 2025). Urban-R1 uses single-instance instruction prompts during training and zero-shot downstream evaluation, yet is still relevant because it strengthens context-grounded reasoning (Wang et al., 18 Oct 2025). CCIL is context-conditioned control rather than prompt-based language modeling (Guo et al., 2023). Urban In-Context Learning is therefore better understood as a family resemblance among models that adapt from inference-time urban context without parameter updates.
A second misconception is that context must be natural language. The surveyed systems use masks, observed region values, prompt tokens, retrieved memory vectors, satellite captions, address strings, POIs, trajectories, traffic states, tool outputs, and structured geospatial descriptions (Li et al., 2 Apr 2025, Yuan et al., 2024). In some cases, natural language is central; in others, it is only one carrier among several. This suggests that the defining property is not linguistic form but conditional adaptation.
Evaluation has likewise diversified. For one-stage urban profiling, gains are reported in MAE, RMSE, and PCC on city-specific downstream indicators, together with ablations on diffusion, alignment, and mask losses (Zhang et al., 5 Aug 2025). For urban spatio-temporal foundation models, zero-shot and few-shot protocols across cities and tasks are central, together with prompt ablations that isolate frequency, time, spatial, and mask prompts (Yuan et al., 2024). For multimodal MLLMs, benchmarks now span GeoQA, street-view understanding, satellite understanding, cross-view retrieval, navigation, and cross-city transfer (Feng et al., 29 Jun 2025). For bias-sensitive urban reasoning, cross-region Spearman rank correlation on seen and unseen regions has become a salient diagnostic because it measures ranking fidelity across space (Wang et al., 18 Oct 2025). A plausible implication is that future urban in-context benchmarks will need to combine task performance, transfer with longer context, fairness across geographic groups, and robustness under distribution shift.
Open problems are explicit across the literature. Urban profiling by masked diffusion notes data scarcity, computational cost of iterative sampling, limited experiments on only two cities, and the absence of explicit geometry in region embeddings (Zhang et al., 5 Aug 2025). UrbanDiT notes that prompts are embedding-level rather than human-interpretable, that the system does not yet consume explicit example–query pairs, and that memory scalability may become an issue as the number of cities and tasks grows (Yuan et al., 2024). UrbanLLaVA identifies limited model size, incomplete modality coverage, and geographic coverage restricted to Beijing, London, and New York (Feng et al., 29 Jun 2025). Urban-R1 leaves open direct in-context benchmarking with city-specific few-shot prompts and fairness-aware RL objectives (Wang et al., 18 Oct 2025). The broader survey adds problems of context length, noisy heterogeneous urban data, spatio-temporal reasoning limits, hallucinations, latency, and interpretability of in-context reasoning (Li et al., 2 Apr 2025).
Theoretical work on world models sharpens these concerns. It argues that long context and diverse environments are necessary for the emergence of strong in-context environment learning, while over-training and strong in-weight learning can bias models toward environment recognition rather than genuine in-context learning (Wang et al., 26 Sep 2025). Applied to cities, this suggests a strategic distinction. If deployment is confined to a known set of cities and regimes, environment recognition may suffice. If the objective is adaptation to new cities, neighborhoods, or policy regimes, then training must emphasize environment diversity, long trajectories, and architectures capable of exploiting very long contexts (Wang et al., 26 Sep 2025). That distinction is increasingly central to the field’s future development.
Urban In-Context Learning has thus become a general organizing idea for urban AI: use observed urban context—whether regions, masks, prompts, multimodal metadata, retrieved documents, or scene state—to drive behavior at inference time. Its current realizations remain heterogeneous, but together they define a coherent research direction: replacing proliferating city-specific and task-specific models with urban systems whose competence is conditioned, compositional, and transferable across contexts (Zhang et al., 5 Aug 2025, Yuan et al., 2024, Feng et al., 29 Jun 2025, Li et al., 2 Apr 2025).