AddressVLM: Fine-Grained Address Localization

• AddressVLM is a large vision-language model that specializes in fine-grained address localization from street-view imagery by integrating microscopic visual cues with macroscopic map context.
• It utilizes a two-stage training protocol with cross-view alignment tuning and street-view VQA to answer detailed district and street-level localization queries.
• Empirical results on Pitts-VQA and SF-Base-VQA datasets demonstrate notable accuracy gains, showcasing the benefits of injecting city-scale spatial priors into a generative multimodal framework.

Searching arXiv for AddressVLM and closely related work on image address localization and geo-semantic grounding. AddressVLM is a large vision-LLM (LVLM) specialized for fine-grained image address localization from street-view imagery. Rather than limiting localization to country- or city-level recognition, it is trained to answer district- and street-level address questions in a flexible visual question answering (VQA) format, including free generation, judgment, and multiple-choice queries. Its central claim is that street-view images alone provide only microscopic visual cues, which are often insufficient for reliable street-level localization within a city. AddressVLM therefore introduces a cross-view alignment tuning stage that injects macroscopic spatial knowledge from satellite or map imagery before a second-stage address localization tuning on street-view VQA data (Xu et al., 14 Aug 2025).

1. Task definition and problem setting

AddressVLM studies image address localization with visual question answering. The input is a street-view image together with an address-related question, and the output is a natural-language answer identifying the district, street, or the correctness of a candidate address. The paper explicitly supports three interaction modes: generation, judgment, and multiple-choice (Xu et al., 14 Aug 2025).

The underlying problem is framed as a fine-grained alternative to conventional geo-localization. Existing LVLMs can often infer broad location categories such as country or city, but they struggle to distinguish among visually similar streets within the same urban area. The paper attributes this to two constraints: street-view images expose only local appearance, and direct street-view fine-tuning does not teach the model the global distribution of streets within a city (Xu et al., 14 Aug 2025).

This formulation differs from earlier work on Image Address Localization (IAL), especially AddressCLIP, which directly aligned images with human-readable address text using a CLIP-style dual-encoder and geography-aware regularization (Xu et al., 2024). AddressCLIP defined the task as predicting a readable textual address from an image and emphasized an end-to-end image-to-address mapping rather than a coordinate-to-address pipeline (Xu et al., 2024). AddressVLM extends this line by moving from a discriminative retrieval-oriented framework to a generative LVLM capable of flexible address question answering (Xu et al., 14 Aug 2025).

A plausible implication is that AddressVLM targets a broader class of downstream applications than closed-set image-to-address retrieval, because the model can answer location questions in multiple linguistic forms rather than only ranking candidate address strings.

2. Architectural formulation and cross-view alignment tuning

AddressVLM follows a LLaVA-style architecture with three modules: a vision encoder $g$, a vision-language adapter $h$, and a pre-trained LLM $f$. For an image $I$, visual features are computed as

$Z_v = g(I),$

then adapted into language-compatible visual tokens

$H_v = h(Z_v).$

For a text query $Q$, text embeddings are denoted

$T_v = \Theta(Q),$

and answer generation is performed by

$A = f(H_v, T_v).$

The main reported backbone uses CLIP as vision encoder and Phi-3.1-mini as the LLM, implemented in the xtuner framework. The paper also evaluates a stronger backbone, MiniCPM-v2.6 = SigLIP + Qwen2-7B (Xu et al., 14 Aug 2025).

The paper’s main innovation is cross-view alignment tuning, a first-stage training process that aligns street-view images with satellite or map views. AddressVLM constructs a single composite input via a satellite-view and street-view image grafting mechanism: $$I_s = \mathbf{M} \odot I_{sa} + (\mathbf{1}-\mathbf{M}) \odot I_{st},$$ where $h$0 is the satellite image, $h$1 is the street-view image, and $h$2 is a binary mask (Xu et al., 14 Aug 2025). The street-view image is placed in the upper-right corner, and the best overlap ratio is reported as $h$3 (Xu et al., 14 Aug 2025).

This compositing strategy is preferred over stitching or dual-image input because it preserves map context while keeping the local street appearance visible. On Pitts-VQA, the grafted combination yields $h$4 and $h$5, compared with stitched at $h$6, $h$7, and separate at $h$8, $h$9 (Xu et al., 14 Aug 2025).

The model’s goal during this stage is not merely to identify local appearance but to learn how a local street-view corresponds to a position on a labeled map. This suggests that the cross-view stage acts as a form of city-scale spatial prior injection, compensating for the narrow field of view of street imagery.

3. Training protocol, objectives, and automatic supervision

AddressVLM is trained in two stages: cross-view alignment tuning followed by address localization tuning (Xu et al., 14 Aug 2025).

The general training set is defined as

$f$0

where $f$1 is the image, $f$2 the question at turn $f$3, and $f$4 the answer (Xu et al., 14 Aug 2025). The multi-turn instruction sequence is

$f$5

and the model is trained with standard autoregressive token prediction: $f$6 The paper gives this generic formulation for both stages (Xu et al., 14 Aug 2025).

Stage 1: cross-view alignment tuning

In the first stage, the model sees a grafted map+street image and is trained to produce an address localization explanation. Crucially, the labels are generated automatically. The pipeline is procedural rather than formally parameterized: an off-the-shelf LVLM receives the grafted image together with the true address as a hint and produces an explanation of why the street-view matches the marked location; this generated explanation is then used as the training target, while the answer hint is removed during actual tuning (Xu et al., 14 Aug 2025).

The paper does not provide a dedicated contrastive alignment loss such as InfoNCE. Instead, stage 1 is implemented as supervised generative instruction tuning over automatically generated cross-view explanation labels (Xu et al., 14 Aug 2025). This distinguishes AddressVLM from AddressCLIP, whose core training objective was a weighted sum of image-address contrastive loss, image-caption contrastive loss, and geography-aware manifold loss (Xu et al., 2024).

Stage 2: address localization tuning

The second stage uses street-view images only and trains the model on address VQA. The same autoregressive likelihood is used for generation, yes/no judgments, and multiple-choice answers (Xu et al., 14 Aug 2025).

AddressVLM defines ten prompt templates each for district and street localization. Examples include “Tell me the district where this image was captured.” and “Identify the street in this image, please.” These are combined with task-type suffixes such as “Answer the question using a single word or phrase.” for generation or “Please select the correct option (A/B/C/D).” for multiple choice (Xu et al., 14 Aug 2025).

The implementation details reported in the appendix are:

• Framework: xtuner
• Hardware: 8 RTX 3090 GPUs
• PyTorch: 2.4.0
• CUDA: 12.1
• transformers: 4.37.2
• Batch size: $f$7
• Gradient accumulation: 16
• Learning rate: $f$8
• Weight decay: 0
• Betas: $f$9
• Warmup ratio: 0.03
• LoRA rank: 128
• LoRA dropout: 0.05
• Image resolution: $I$0 (Xu et al., 14 Aug 2025)

The paper does not explicitly state the number of epochs (Xu et al., 14 Aug 2025).

4. Datasets and benchmark construction

AddressVLM introduces two VQA datasets derived from existing image address localization corpora:

• Pitts-VQA
• SF-Base-VQA (Xu et al., 14 Aug 2025)

These are constructed from Pitts-IAL and SF-Base-IAL, respectively (Xu et al., 14 Aug 2025). The earlier AddressCLIP work had introduced the underlying IAL datasets and defined the broader city-scale problem of aligning street-view imagery with readable address text (Xu et al., 2024).

Each AddressVLM image receives 7 rounds of QA, covering district generation, district judgment, district multiple-choice, street generation, street judgment, street multiple-choice, and a combined street-and-district round (Xu et al., 14 Aug 2025). Labels are automatically generated from templates and then manually checked (Xu et al., 14 Aug 2025).

The datasets use a 7:2:1 train/val/test split (Xu et al., 14 Aug 2025).

Dataset statistics

Dataset	Locations	Images per location	Train images	Test images
Pitts-VQA	10,586	24	177,840	19,152
SF-Base-VQA	17,067	12	143,352	20,484

Additional reported properties are substantial. Pitts-VQA covers 20 km$I$1, contains 19 districts, and has 194 train streets and 165 test streets. SF-Base-VQA covers 6 km$I$2, contains 15 districts, and has 121 train streets and 110 test streets (Xu et al., 14 Aug 2025).

These VQA datasets are part of a broader progression from AddressCLIP’s IAL task. AddressCLIP had focused on neighborhood, street, and sub-street semantics and used a semantic address partition strategy to represent locations by main street, intersecting streets, and neighborhood (Xu et al., 2024). AddressVLM instead operationalizes the task as dialogue-style address QA, but still inherits the city-wide street-level supervision structure (Xu et al., 14 Aug 2025).

A plausible implication is that the VQA reformulation provides richer supervision for generative LVLMs than the earlier retrieval-style text alignment used by AddressCLIP.

5. Empirical performance and ablation findings

AddressVLM is evaluated against:

• LLaVA-Phi3-mini zero-shot
• Baseline: same backbone with only stage-2 address localization tuning
• GeoReasoner adapted to district/street localization (Xu et al., 14 Aug 2025)

The principal evaluation metrics are:

• $I$3: district generation accuracy
• $I$4: district judgment accuracy
• $I$5: district multiple-choice accuracy
• $I$6: average district accuracy
• $I$7: street generation accuracy
• $I$8: street judgment accuracy
• $I$9: street multiple-choice accuracy
• $Z_v = g(I),$0: average street accuracy
• $Z_v = g(I),$1: overall average
• $Z_v = g(I),$2: joint street-and-district generation accuracy (Xu et al., 14 Aug 2025)

Main results

On Pitts-VQA, AddressVLM achieves:

• $Z_v = g(I),$3
• $Z_v = g(I),$4
• $Z_v = g(I),$5
• $Z_v = g(I),$6 (Xu et al., 14 Aug 2025)

The stage-2-only Baseline reaches:

• $Z_v = g(I),$7
• $Z_v = g(I),$8 (Xu et al., 14 Aug 2025)

Thus AddressVLM improves over the baseline by:

• +2.75 on $Z_v = g(I),$9
• +9.08 on $H_v = h(Z_v).$0 (Xu et al., 14 Aug 2025)

On SF-Base-VQA, AddressVLM achieves:

• $H_v = h(Z_v).$1
• $H_v = h(Z_v).$2
• $H_v = h(Z_v).$3
• $H_v = h(Z_v).$4 (Xu et al., 14 Aug 2025)

The baseline reaches:

• $H_v = h(Z_v).$5
• $H_v = h(Z_v).$6 (Xu et al., 14 Aug 2025)

So the gains are:

• +2.82 on $H_v = h(Z_v).$7
• +11.83 on $H_v = h(Z_v).$8 (Xu et al., 14 Aug 2025)

The paper summarizes these as improvements of over 9% and 12% in average address localization accuracy on the Pittsburgh and San Francisco datasets, respectively (Xu et al., 14 Aug 2025).

Comparison to prior address-localization methods

The earlier AddressCLIP achieved strong closed-set performance on IAL benchmarks, including SSA-1 80.39 on Pitts-IAL, 86.32 on SF-IAL-Base, and 85.92 on SF-IAL-Large, outperforming prompt-learning CLIP baselines and a two-stage geolocalization-plus-reverse-geocoding pipeline (Xu et al., 2024). However, AddressCLIP remained a discriminative model limited by the set of candidate addresses at inference time (Xu et al., 2024). AddressVLM trades some of that closed-set simplicity for a more general-purpose generative QA interface (Xu et al., 14 Aug 2025).

Ablations

The ablation studies establish several consistent trends (Xu et al., 14 Aug 2025):

• Road-name overlays matter: using satellite images without road labels lowers Pitts-VQA performance from $H_v = h(Z_v).$9, $Q$0 to $Q$1, $Q$2.
• Overlap ratio matters: the best grafting ratio is $Q$3; both 0.3 and 0.7 are worse.
• Grafting is best: both stitched and separate-image variants underperform grafting.
• More trainable components help: the best setup unfreezes both vision encoder and LLM in stage 1 and the vision encoder in stage 2.
• Data sparsity is tolerated: with viewpoint density reduced to 50%, performance remains over 88% $Q$4; with location density reduced to 50%, $Q$5 remains over 71%.
• Multi-city mixing helps slightly: training on Pitts + SF gives modest improvements over city-specific training.
• Mixing general VQA hurts: adding generic LLaVA VQA data during stage 2 substantially degrades address localization.
• Backbone scaling helps: SigLIP + Qwen2-7B slightly improves over CLIP + Phi3-mini (Xu et al., 14 Aug 2025).

The degradation under generic VQA mixing is particularly noteworthy. On Pitts, mixing with mix665k (5:1) reduces $Q$6 to 79.95 and $Q$7 to 42.08, compared with 90.02 and 69.60 without mixing (Xu et al., 14 Aug 2025). This suggests that address localization requires strong task specialization and that generic multimodal instruction tuning can dilute fine-grained geo-semantic discrimination.

6. Relation to broader research and limitations

AddressVLM sits within a broader cluster of work on address-grounded, map-grounded, and place-grounded vision-language systems.

AddressCLIP introduced the core city-wide image address localization problem and showed that direct image-address alignment with geographic regularization can outperform image-retrieval-plus-reverse-geocoding pipelines (Xu et al., 2024). VLG-Loc demonstrated that a VLM can localize a robot from a weakly supervised labeled footprint map containing only landmark names and areas, using map-conditioned visual landmark search and pose likelihood scoring (Aoki et al., 14 Dec 2025). While VLG-Loc is not an address-localization system, it shows that human-readable place labels can function as localization anchors without dense metric appearance models (Aoki et al., 14 Dec 2025). This suggests a conceptual continuity: AddressVLM extends language-mediated place grounding from named landmarks to urban street-level addresses.

AddressVLM also differs sharply from place-recognition models that predict coordinates or retrieve nearest database images. Its outputs are district/street answers, not GPS coordinates, and its stage-1 cross-view training uses map imagery only during training; at inference time, the model consumes only the street-view image and question (Xu et al., 14 Aug 2025).

The paper notes several limitations (Xu et al., 14 Aug 2025):

• Image grafting is preliminary rather than a learned fusion module.
• Street-view resolution may limit fine-grained disambiguation.
• Open-set generative models still lag closed-set discriminative models such as AddressCLIP on some strict localization settings.
• The method depends on city-specific structured data, including aligned street-view images, address annotations, and map or satellite images with road labels.

A plausible implication is that AddressVLM’s strongest regime is a city for which paired street-view and labeled map data can be assembled at scale. Generalization to rural regions, weakly mapped cities, or address systems with different textual conventions is not established in the reported experiments.

7. Significance

AddressVLM marks a transition from coarse geo-localization toward city-wide street-level address question answering with LVLMs. Its main contribution is not merely better accuracy, but a concrete training recipe for injecting global street-distribution knowledge into a generative multimodal model via cross-view alignment tuning (Xu et al., 14 Aug 2025).

Relative to AddressCLIP’s discriminative formulation, AddressVLM demonstrates that address localization can be embedded in a more flexible conversational interface while still benefiting from strong city-specific priors (Xu et al., 2024, Xu et al., 14 Aug 2025). Relative to weakly supervised semantic localization systems such as VLG-Loc, it shows that map text and spatial layout can be exploited not only for robotic pose hypotheses but also for open-ended district/street reasoning (Aoki et al., 14 Dec 2025).

The central technical lesson is that fine-grained address localization requires both microscopic and macroscopic cues. Street-view images provide local visual evidence, but street-level inference becomes substantially stronger when the model is first taught to align those local views with a city-wide labeled spatial scaffold (Xu et al., 14 Aug 2025).