---
title: 'AddressVLM: Fine-Grained Address Localization'
url: https://www.emergentmind.com/topics/addressvlm
type: topic
---

# AddressVLM: Fine-Grained Address Localization

Searching arXiv for AddressVLM and closely related work on image address localization and geo-semantic grounding.
AddressVLM is a large vision-language model (LVLM) specialized for **fine-grained image address localization** from street-view imagery. Rather than limiting localization to country- or city-level recognition, it is trained to answer **district- and street-level address questions** in a flexible visual question answering (VQA) format, including free generation, judgment, and multiple-choice queries. Its central claim is that street-view images alone provide only **microscopic visual cues**, which are often insufficient for reliable street-level localization within a city. AddressVLM therefore introduces a **cross-view alignment tuning** stage that injects **macroscopic spatial knowledge** from satellite or map imagery before a second-stage **address localization tuning** on street-view VQA data [2508.10667].

## 1. Task definition and problem setting

AddressVLM studies **image address localization with visual question answering**. The input is a **street-view image** together with an **address-related question**, and the output is a natural-language answer identifying the district, street, or the correctness of a candidate address. The paper explicitly supports three interaction modes: **generation**, **judgment**, and **multiple-choice** [2508.10667].

The underlying problem is framed as a fine-grained alternative to conventional geo-localization. Existing LVLMs can often infer broad location categories such as country or city, but they struggle to distinguish among visually similar streets within the same urban area. The paper attributes this to two constraints: street-view images expose only local appearance, and direct street-view fine-tuning does not teach the model the **global distribution of streets within a city** [2508.10667].

This formulation differs from earlier work on **Image Address Localization (IAL)**, especially AddressCLIP, which directly aligned images with human-readable address text using a CLIP-style dual-encoder and geography-aware regularization [2407.08156]. AddressCLIP defined the task as predicting a readable textual address from an image and emphasized an end-to-end **image-to-address** mapping rather than a coordinate-to-address pipeline [2407.08156]. AddressVLM extends this line by moving from a discriminative retrieval-oriented framework to a **generative LVLM** capable of flexible address question answering [2508.10667].

A plausible implication is that AddressVLM targets a broader class of downstream applications than closed-set image-to-address retrieval, because the model can answer location questions in multiple linguistic forms rather than only ranking candidate address strings.

## 2. Architectural formulation and cross-view alignment tuning

AddressVLM follows a **LLaVA-style** architecture with three modules: a **vision encoder** \(g\), a **vision-language adapter** \(h\), and a **pre-trained LLM** \(f\). For an image \(I\), visual features are computed as
\[
Z_v = g(I),
\]
then adapted into language-compatible visual tokens
\[
H_v = h(Z_v).
\]
For a text query \(Q\), text embeddings are denoted
\[
T_v = \Theta(Q),
\]
and answer generation is performed by
\[
A = f(H_v, T_v).
\]
The main reported backbone uses **CLIP** as vision encoder and **Phi-3.1-mini** as the language model, implemented in the **xtuner** framework. The paper also evaluates a stronger backbone, **MiniCPM-v2.6 = SigLIP + Qwen2-7B** [2508.10667].

The paper’s main innovation is **cross-view alignment tuning**, a first-stage training process that aligns street-view images with satellite or map views. AddressVLM constructs a single composite input via a **satellite-view and street-view image grafting mechanism**:
\[
I_s = \mathbf{M} \odot I_{sa} + (\mathbf{1}-\mathbf{M}) \odot I_{st},
\]
where \(I_{sa}\) is the satellite image, \(I_{st}\) is the street-view image, and \(\mathbf{M}\) is a binary mask [2508.10667]. The street-view image is placed in the **upper-right corner**, and the best overlap ratio is reported as **\(\delta = 0.5\)** [2508.10667].

This compositing strategy is preferred over stitching or dual-image input because it preserves map context while keeping the local street appearance visible. On Pitts-VQA, the **grafted** combination yields \(\bar{A}=90.02\) and \(A_{sd}=69.60\), compared with **stitched** at \(\bar{A}=87.91\), \(A_{sd}=66.45\), and **separate** at \(\bar{A}=88.14\), \(A_{sd}=67.39\) [2508.10667].

The model’s goal during this stage is not merely to identify local appearance but to learn how a local street-view corresponds to a position on a labeled map. This suggests that the cross-view stage acts as a form of **city-scale spatial prior injection**, compensating for the narrow field of view of street imagery.

## 3. Training protocol, objectives, and automatic supervision

AddressVLM is trained in **two stages**: **cross-view alignment tuning** followed by **address localization tuning** [2508.10667].

The general training set is defined as
\[
D_{train} = \{(I_i, Q_i^j, A_i^j)\}_{i=1}^{M}, \; j \in [1...N_i],
\]
where \(I_i\) is the image, \(Q_i^j\) the question at turn \(j\), and \(A_i^j\) the answer [2508.10667]. The multi-turn instruction sequence is
\[
S_i^t=
\begin{cases}
[I_i, Q_i^1], & t=1 \\
Q_i^t, & t>1
\end{cases}
\]
and the model is trained with standard autoregressive token prediction:
\[
p(A_i|I_i,S_i) = \prod_{j=1}^{N} p_{\theta}(x_j|I_i, S_{<j}, A_{<j}).
\]
The paper gives this generic formulation for both stages [2508.10667].

### Stage 1: cross-view alignment tuning

In the first stage, the model sees a **grafted map+street image** and is trained to produce an address localization explanation. Crucially, the labels are generated automatically. The pipeline is procedural rather than formally parameterized: an **off-the-shelf LVLM** receives the grafted image together with the true address as a hint and produces an explanation of why the street-view matches the marked location; this generated explanation is then used as the training target, while the answer hint is removed during actual tuning [2508.10667].

The paper does **not** provide a dedicated contrastive alignment loss such as InfoNCE. Instead, stage 1 is implemented as **supervised generative instruction tuning** over automatically generated cross-view explanation labels [2508.10667]. This distinguishes AddressVLM from AddressCLIP, whose core training objective was a weighted sum of **image-address contrastive loss**, **image-caption contrastive loss**, and **geography-aware manifold loss** [2407.08156].

### Stage 2: address localization tuning

The second stage uses **street-view images only** and trains the model on address VQA. The same autoregressive likelihood is used for generation, yes/no judgments, and multiple-choice answers [2508.10667].

AddressVLM defines ten prompt templates each for **district** and **street** localization. Examples include “Tell me the district where this image was captured.” and “Identify the street in this image, please.” These are combined with task-type suffixes such as “Answer the question using a single word or phrase.” for generation or “Please select the correct option (A/B/C/D).” for multiple choice [2508.10667].

The implementation details reported in the appendix are:

- **Framework**: xtuner
- **Hardware**: 8 RTX 3090 GPUs
- **PyTorch**: 2.4.0
- **CUDA**: 12.1
- **transformers**: 4.37.2
- **Batch size**: \(4 \times 8\)
- **Gradient accumulation**: 16
- **Learning rate**: \(1\times10^{-5}\)
- **Weight decay**: 0
- **Betas**: \((0.9, 0.999)\)
- **Warmup ratio**: 0.03
- **LoRA rank**: 128
- **LoRA dropout**: 0.05
- **Image resolution**: \(336\times336\) [2508.10667]

The paper does not explicitly state the number of epochs [2508.10667].

## 4. Datasets and benchmark construction

AddressVLM introduces two VQA datasets derived from existing image address localization corpora:

- **Pitts-VQA**
- **SF-Base-VQA** [2508.10667]

These are constructed from **Pitts-IAL** and **SF-Base-IAL**, respectively [2508.10667]. The earlier AddressCLIP work had introduced the underlying IAL datasets and defined the broader city-scale problem of aligning street-view imagery with readable address text [2407.08156].

Each AddressVLM image receives **7 rounds of QA**, covering district generation, district judgment, district multiple-choice, street generation, street judgment, street multiple-choice, and a combined street-and-district round [2508.10667]. Labels are automatically generated from templates and then **manually checked** [2508.10667].

The datasets use a **7:2:1** train/val/test split [2508.10667].

### Dataset statistics

| Dataset | Locations | Images per location | Train images | Test images |
|---|---:|---:|---:|---:|
| Pitts-VQA | 10,586 | 24 | 177,840 | 19,152 |
| SF-Base-VQA | 17,067 | 12 | 143,352 | 20,484 |

Additional reported properties are substantial. Pitts-VQA covers **20 km\(^2\)**, contains **19 districts**, and has **194 train streets** and **165 test streets**. SF-Base-VQA covers **6 km\(^2\)**, contains **15 districts**, and has **121 train streets** and **110 test streets** [2508.10667].

These VQA datasets are part of a broader progression from AddressCLIP’s IAL task. AddressCLIP had focused on **neighborhood**, **street**, and **sub-street** semantics and used a **semantic address partition strategy** to represent locations by main street, intersecting streets, and neighborhood [2407.08156]. AddressVLM instead operationalizes the task as dialogue-style address QA, but still inherits the city-wide street-level supervision structure [2508.10667].

A plausible implication is that the VQA reformulation provides richer supervision for generative LVLMs than the earlier retrieval-style text alignment used by AddressCLIP.

## 5. Empirical performance and ablation findings

AddressVLM is evaluated against:

- **LLaVA-Phi3-mini** zero-shot
- **Baseline**: same backbone with only stage-2 address localization tuning
- **GeoReasoner** adapted to district/street localization [2508.10667]

The principal evaluation metrics are:

- \(A_d^G\): district generation accuracy
- \(A_d^J\): district judgment accuracy
- \(A_d^M\): district multiple-choice accuracy
- \(\bar{A}_d\): average district accuracy
- \(A_s^G\): street generation accuracy
- \(A_s^J\): street judgment accuracy
- \(A_s^M\): street multiple-choice accuracy
- \(\bar{A}_s\): average street accuracy
- \(\bar{A}\): overall average
- \(A_{sd}\): joint street-and-district generation accuracy [2508.10667]

### Main results

On **Pitts-VQA**, AddressVLM achieves:

- \(\bar{A}_d = 92.70\)
- \(\bar{A}_s = 87.46\)
- \(\bar{A} = 90.02\)
- \(A_{sd} = 69.60\) [2508.10667]

The stage-2-only **Baseline** reaches:

- \(\bar{A} = 87.27\)
- \(A_{sd} = 60.52\) [2508.10667]

Thus AddressVLM improves over the baseline by:

- **+2.75** on \(\bar{A}\)
- **+9.08** on \(A_{sd}\) [2508.10667]

On **SF-Base-VQA**, AddressVLM achieves:

- \(\bar{A}_d = 92.06\)
- \(\bar{A}_s = 86.66\)
- \(\bar{A} = 89.33\)
- \(A_{sd} = 70.45\) [2508.10667]

The baseline reaches:

- \(\bar{A} = 86.51\)
- \(A_{sd} = 58.62\) [2508.10667]

So the gains are:

- **+2.82** on \(\bar{A}\)
- **+11.83** on \(A_{sd}\) [2508.10667]

The paper summarizes these as improvements of over **9%** and **12%** in average address localization accuracy on the Pittsburgh and San Francisco datasets, respectively [2508.10667].

### Comparison to prior address-localization methods

The earlier AddressCLIP achieved strong closed-set performance on IAL benchmarks, including **SSA-1 80.39** on Pitts-IAL, **86.32** on SF-IAL-Base, and **85.92** on SF-IAL-Large, outperforming prompt-learning CLIP baselines and a two-stage geolocalization-plus-reverse-geocoding pipeline [2407.08156]. However, AddressCLIP remained a **discriminative** model limited by the set of candidate addresses at inference time [2407.08156]. AddressVLM trades some of that closed-set simplicity for a more general-purpose generative QA interface [2508.10667].

### Ablations

The ablation studies establish several consistent trends [2508.10667]:

- **Road-name overlays matter**: using satellite images **without road labels** lowers Pitts-VQA performance from \(\bar{A}=90.02\), \(A_{sd}=69.60\) to \(\bar{A}=88.06\), \(A_{sd}=64.63\).
- **Overlap ratio matters**: the best grafting ratio is **\(\delta=0.5\)**; both 0.3 and 0.7 are worse.
- **Grafting is best**: both stitched and separate-image variants underperform grafting.
- **More trainable components help**: the best setup unfreezes both **vision encoder** and **LLM** in stage 1 and the **vision encoder** in stage 2.
- **Data sparsity is tolerated**: with viewpoint density reduced to 50%, performance remains over **88% \(\bar{A}\)**; with location density reduced to 50%, \(A_{sd}\) remains over **71%**.
- **Multi-city mixing helps slightly**: training on **Pitts + SF** gives modest improvements over city-specific training.
- **Mixing general VQA hurts**: adding generic LLaVA VQA data during stage 2 substantially degrades address localization.
- **Backbone scaling helps**: **SigLIP + Qwen2-7B** slightly improves over **CLIP + Phi3-mini** [2508.10667].

The degradation under generic VQA mixing is particularly noteworthy. On Pitts, mixing with **mix665k (5:1)** reduces \(\bar{A}\) to **79.95** and \(A_{sd}\) to **42.08**, compared with **90.02** and **69.60** without mixing [2508.10667]. This suggests that address localization requires strong task specialization and that generic multimodal instruction tuning can dilute fine-grained geo-semantic discrimination.

## 6. Relation to broader research and limitations

AddressVLM sits within a broader cluster of work on address-grounded, map-grounded, and place-grounded vision-language systems.

**AddressCLIP** introduced the core city-wide **image address localization** problem and showed that direct image-address alignment with geographic regularization can outperform image-retrieval-plus-reverse-geocoding pipelines [2407.08156]. **VLG-Loc** demonstrated that a VLM can localize a robot from a weakly supervised **labeled footprint map** containing only landmark names and areas, using map-conditioned visual landmark search and pose likelihood scoring [2512.12793]. While VLG-Loc is not an address-localization system, it shows that human-readable place labels can function as localization anchors without dense metric appearance models [2512.12793]. This suggests a conceptual continuity: AddressVLM extends language-mediated place grounding from named landmarks to urban street-level addresses.

AddressVLM also differs sharply from place-recognition models that predict coordinates or retrieve nearest database images. Its outputs are **district/street answers**, not GPS coordinates, and its stage-1 cross-view training uses map imagery only during training; at inference time, the model consumes only the **street-view image and question** [2508.10667].

The paper notes several limitations [2508.10667]:

- **Image grafting is preliminary** rather than a learned fusion module.
- **Street-view resolution** may limit fine-grained disambiguation.
- **Open-set generative models still lag closed-set discriminative models** such as AddressCLIP on some strict localization settings.
- The method depends on **city-specific structured data**, including aligned street-view images, address annotations, and map or satellite images with road labels.

A plausible implication is that AddressVLM’s strongest regime is a city for which paired street-view and labeled map data can be assembled at scale. Generalization to rural regions, weakly mapped cities, or address systems with different textual conventions is not established in the reported experiments.

## 7. Significance

AddressVLM marks a transition from coarse geo-localization toward **city-wide street-level address question answering** with LVLMs. Its main contribution is not merely better accuracy, but a concrete training recipe for injecting **global street-distribution knowledge** into a generative multimodal model via **cross-view alignment tuning** [2508.10667].

Relative to AddressCLIP’s discriminative formulation, AddressVLM demonstrates that address localization can be embedded in a more flexible conversational interface while still benefiting from strong city-specific priors [2407.08156; 2508.10667]. Relative to weakly supervised semantic localization systems such as VLG-Loc, it shows that map text and spatial layout can be exploited not only for robotic pose hypotheses but also for open-ended district/street reasoning [2512.12793].

The central technical lesson is that **fine-grained address localization requires both microscopic and macroscopic cues**. Street-view images provide local visual evidence, but street-level inference becomes substantially stronger when the model is first taught to align those local views with a city-wide labeled spatial scaffold [2508.10667].

Source: https://www.emergentmind.com/topics/addressvlm