VectorLLM: Multimodal Vector Extraction

• VectorLLM is a multimodal model that reformulates remote-sensing contour extraction as an autoregressive sequence generation task, moving away from traditional mask-based methods.
• It combines a vision encoder, learnable positional embeddings, and a Qwen3-based LLM to predict ordered polygons as a structured sequence of coordinate tokens.
• Evaluations on WHU, WHU-Mix, and CrowdAI benchmarks show significant improvements in AP and demonstrate the model’s capacity for zero-shot generalization to unseen object categories.

Searching arXiv for the cited VectorLLM paper and closely related remote-sensing contour extraction work. arXiv search: "VectorLLM Human-like Extraction of Structured Building Contours multimodal LLMs" VectorLLM is a multimodal LLM for remote-sensing vector extraction that was introduced as the first MLLM designed specifically for regular building contour extraction from remote sensing images. Rather than following the conventional multi-stage route of mask prediction, polygon conversion, and refinement, it formulates contour extraction as autoregressive sequence generation: given an image and an instruction, the model emits a polygon as a sequence of coordinate tokens, corner point by corner point, in a manner intended to mimic human annotators. The architecture combines a vision foundation backbone, an MLP connector, learnable positional embeddings, and a pretrained Qwen3 LLM, and the reported evaluation shows state-of-the-art performance on WHU, WHU-Mix, and CrowdAI, with improvements of 5.6 AP, 7.1 AP, and 13.6 AP over prior methods on the respective benchmarks. The same study also reports zero-shot transfer to unseen object categories such as aircraft, water bodies, and oil tanks, positioning the model as a candidate for unified contour extraction beyond buildings (Zhang et al., 7 Jul 2025).

1. Problem formulation and conceptual departure

VectorLLM is motivated by the observation that building-contour extraction in remote sensing has traditionally been dominated by two broad families of methods. The first is segmentation-first: predict a building mask, convert the mask to polygons, and then simplify or refine the polygon through post-processing. The second is direct polygon or vertex regression, but typically with a large number of auxiliary tasks such as vertex heatmaps, edge or line detection, redundant-point removal, and topological relation reconstruction. The paper characterizes both families as complex, difficult to scale, and poorly aligned with the way human annotators actually trace object boundaries (Zhang et al., 7 Jul 2025).

The central reformulation is to cast contour extraction as a language-generation or sequence-prediction problem. In this view, the polygon is not recovered indirectly from a raster mask and is not assembled from multiple handcrafted geometric submodules. Instead, the model receives an image and a textual prompt and generates the contour directly as a structured sequence of coordinate tokens. This introduces three explicit departures from prior pipelines: no mask-to-polygon post-processing, no hand-designed geometric proxy tasks, and reliance on LLM-style sequential reasoning to model contour structure and polygon closure.

The contour ordering is fixed to a human-like convention. The sequence begins from the top-left corner, proceeds clockwise, and returns to the starting point to close the polygon. The output is represented as alternating x/y tokens, for example:

$< 0.8$1

This ordering is not incidental. The model is trained to predict the next corner conditioned on the image, the instruction, and all previously generated points. The autoregressive setup is intended to enforce local geometric consistency, encourage closed contours, and provide greater topological stability than unordered point prediction. A plausible implication is that the LLM contributes less through generic semantic captioning than through sequential constraint modeling over contour topology.

2. Architecture and coordinate representation

VectorLLM uses an intentionally simple and modular architecture composed of a vision encoder, learnable positional embeddings, an MLP projector, and an LLM. The vision backbone is a pretrained RADIO ViT-L model, with RADIO described as distilled from CLIP, SAM, and DINOv2. Input images are resized to $128 \times 128$, and the encoder outputs image features downsampled by a factor of 16. In ablations, the paper compares CLIP, DINOv2, and RADIO, and reports that CLIP performs much worse, while RADIO and DINOv2 are close, with RADIO selected for broader alignment and future extensibility (Zhang et al., 7 Jul 2025).

A key architectural addition is the use of learnable positional embeddings on top of the visual features before projection into the language space. The rationale is explicitly spatial: contour extraction requires the LLM to know where each visual token originates, so positional information is required for geometric reasoning. The reported oracle-box WHU ablation shows performance dropping from 88.1 to 86.2 mAP when the learnable positional embeddings are removed, approximately a 1.9 AP decrease. This makes the positional component one of the paper’s clearest pieces of evidence that multimodal alignment alone is insufficient for precise vector geometry.

The cross-modal bridge is a simple MLP projector in the style of LLaVA-like systems. It maps visual features into the LLM embedding space and produces vision tokens without introducing a heavy, task-specific fusion module. This design choice is presented as a way to keep the model lightweight and scalable.

The LLM itself is a pretrained Qwen3 variant. Two model sizes are reported: VectorLLM-0.6B uses Qwen3-0.6B, and VectorLLM-1.7B uses Qwen3-1.7B. The LLM handles instruction following, sequential reasoning, and coordinate generation. To make coordinate emission compatible with autoregressive decoding, the vocabulary is extended with discrete coordinate tokens of the form [x0] ... [x(W-1)] and [y0] ... [y(H-1)]. Coordinates are therefore not emitted as free-form numbers; they are emitted as special tokens, converting contour extraction into a structured sequence-generation problem. The paper further reports that a single special token per coordinate value outperforms multi-token digit decomposition because it yields higher accuracy, shorter sequences, and lower compute cost (Zhang et al., 7 Jul 2025).

3. Training pipeline and optimization objectives

VectorLLM is trained with a three-stage procedure consisting of pretraining, supervised fine-tuning, and direct preference optimization. Each stage targets a different aspect of the contour-generation problem.

In pretraining, the goal is to align visual features with the language space and teach basic geometric sequence prediction. The vision encoder is frozen, while the MLP projector, learnable positional embeddings, and LLM are trained. The training format concatenates the image with the target contour sequence, and optimization uses standard next-token prediction loss over text tokens only:

$$\max \log p(\text{next coordinate token} \mid \text{image}, \text{previous coordinate tokens})$$

The paper emphasizes that pretraining is critical for alignment, contour-generation ability, and downstream stability, and reports that removing pretraining causes large performance degradation and instability. During this stage, the starting point of the contour is randomly shuffled, an intervention intended to prevent overfitting to a fixed start token and to encourage more robust topological reasoning (Zhang et al., 7 Jul 2025).

Supervised fine-tuning converts the building datasets into instruction-answer pairs. The prompt specifies that the model should extract the regular vector contour of the central building, start from the left-top corner, and go clockwise. Unlike pretraining, SFT fine-tunes the entire stack: vision encoder, MLP projector, positional embeddings, and LLM. The loss remains standard next-token prediction. The claimed role of SFT is to make the model instruction-following, more stable in contour ordering, and better at standardized polygon output.

The final stage is direct preference optimization, introduced to correct residual failure modes after SFT, including repeated outputs, missing small lines, roof-plane interference, and minor topological errors. Preference data are constructed using chosen answers equal to correct labels and rejected answers derived either from SFT predictions with IoU $< 0.8$ or from corrupted complex labels created by deleting points or inserting random points. The DPO objective is given as

$$\mathcal{L}_{dpo} = -\log \sigma\left(\beta \log\frac{\pi_{\theta}(y_c \mid x)}{\pi_0(y_c \mid x)} - \beta \log\frac{\pi_{\theta}(y_r \mid x)}{\pi_0(y_r \mid x)}\right)$$

where $x$ is the instruction/input, $y_c$ the chosen answer, $y_r$ the rejected answer, $\pi_\theta$ the current policy, $\pi_0$ the frozen SFT reference model, and $\beta$ the KL penalty coefficient. On the WHU ablation, DPO contributes about $$\max \log p(\text{next coordinate token} \mid \text{image}, \text{previous coordinate tokens})$$0 AP and $$\max \log p(\text{next coordinate token} \mid \text{image}, \text{previous coordinate tokens})$$1 AP75. The staged training ablation reports 83.6 AP for SFT only, 86.3 AP for pretrain + SFT, and 88.1 AP for pretrain + SFT + DPO, supporting the paper’s claim that each stage has a distinct contribution (Zhang et al., 7 Jul 2025).

4. Instance-centric inference protocol, datasets, and metrics

VectorLLM is designed to extract the contour of one prominent building instance at a time rather than perform dense multi-instance prediction directly. The paper attributes this choice to LLM token-budget limitations. The inference protocol therefore relies on an external detector to identify building boxes, after which the building crop is passed to VectorLLM. The detector used is FCOS with a Swin-L backbone. At evaluation time, either oracle boxes or predicted detector boxes are used. Boxes are enlarged by about $$\max \log p(\text{next coordinate token} \mid \text{image}, \text{previous coordinate tokens})$$2 during testing and are randomly scaled between $$\max \log p(\text{next coordinate token} \mid \text{image}, \text{previous coordinate tokens})$$3 and $$\max \log p(\text{next coordinate token} \mid \text{image}, \text{previous coordinate tokens})$$4 during training. This crop-based design isolates a single object and reduces complexity (Zhang et al., 7 Jul 2025).

The study jointly trains and evaluates on three main building datasets: WHU, WHU-Mix, and CrowdAI. WHU is treated as the main building extraction benchmark. WHU-Mix includes both in-domain and out-of-domain settings. CrowdAI is characterized as harder, lower resolution, and noisier in annotation quality. In a scaling ablation, the paper also incorporates COCO as part of joint co-training, though not as a main building benchmark.

Evaluation uses mAP, AP50, AP75, and AR. These are standard detection or instance-style metrics, repurposed here for vector contour extraction. The use of instance-style metrics is consistent with the crop-based protocol: the system predicts a structured polygon for a single detected instance, and quality is then measured at different overlap thresholds.

An important practical implication follows from this design. VectorLLM is not, in its reported form, a full end-to-end detector-plus-contour-generator. It is instead an instance-conditioned contour generator whose performance depends in part on the quality of the provided or detected bounding boxes. This is reflected in the consistent performance gap between oracle-box and detector-box evaluations.

5. Quantitative performance, ablations, and scaling behavior

The reported quantitative results place VectorLLM-1.7B ahead of previous state of the art on all three building benchmarks. On WHU with detector-predicted boxes, the model achieves 79.4 AP, 91.6 AP50, 85.9 AP75, and 83.8 AR. The paper compares this to P2PFormer at 72.7 AP and Line2Poly at 73.8 AP, corresponding to improvements of $$\max \log p(\text{next coordinate token} \mid \text{image}, \text{previous coordinate tokens})$$5 AP over P2PFormer and $$\max \log p(\text{next coordinate token} \mid \text{image}, \text{previous coordinate tokens})$$6 AP over Line2Poly. Under oracle boxes, the same benchmark reaches 90.2 AP, 98.9 AP50, 98.3 AP75, and 94.3 AR (Zhang et al., 7 Jul 2025).

On WHU-Mix, VectorLLM-1.7B reports 68.4 AP on Test-1 and 56.4 AP on Test-2 with predicted boxes, surpassing P2PFormer by $$\max \log p(\text{next coordinate token} \mid \text{image}, \text{previous coordinate tokens})$$7 AP on the in-domain test and $$\max \log p(\text{next coordinate token} \mid \text{image}, \text{previous coordinate tokens})$$8 AP on the out-of-domain test. With oracle boxes, the results are 78.5 AP, 96.7 AP50, and 86.0 AP75 on Test-1, and 64.6 AP, 96.6 AP50, and 74.2 AP75 on Test-2. On CrowdAI, the detector-box evaluation reaches 79.6 AP, 91.5 AP50, and 81.3 AP75, while oracle-box evaluation reaches 87.7 AP, 94.2 AP50, and 90.4 AP75. The paper states that this surpasses P2PFormer by $$\max \log p(\text{next coordinate token} \mid \text{image}, \text{previous coordinate tokens})$$9 AP, the largest improvement reported across the benchmarks.

Dataset	Predicted-box result	Oracle-box result
WHU	79.4 AP	90.2 AP
WHU-Mix Test-1	68.4 AP	78.5 AP
WHU-Mix Test-2	56.4 AP	64.6 AP
CrowdAI	79.6 AP	87.7 AP

The ablations are used to support several architectural and methodological claims. Learnable positional embeddings are empirically important, with oracle WHU mAP falling from 88.1 to 86.2 when they are removed. The vision-encoder comparison gives 77.1 AP for CLIP, 88.1 AP for RADIO, and 88.2 AP for DINOv2. The training-stage ablation shows monotonic gains from SFT only to pretrain + SFT to pretrain + SFT + DPO. Coordinate representation matters: the single-token coordinate scheme is favored over multi-token digit decomposition. Data scaling also produces gains, with WHU only at 85.9, WHU + COCO at 86.2, and WHU + COCO + Mix + CrowdAI at 88.1. The paper interprets this pattern as evidence that data scaling matters more than architectural complexity. It also reports that the larger Qwen3-1.7B backbone outperforms Qwen3-0.6B across datasets, indicating that LLM scale contributes directly to contour extraction accuracy (Zhang et al., 7 Jul 2025).

6. Generalization, limitations, and terminological context

A major claim of the paper is that VectorLLM generalizes beyond buildings. Zero-shot experiments are reported on unseen objects and scene types including aircraft, water bodies, oil tanks, ships, roads, vegetation, playgrounds, backpacks, and motorcycles. The study interprets this as evidence that the model is not merely memorizing building geometry but has learned a more general notion of closed shapes, boundary tracing, geometric topology, and contour granularity adaptation. Specifically, simple objects are said to receive sparse vertices while complex objects receive denser vertices. This suggests that the sequence model is capturing some relation between structural complexity and vertex allocation (Zhang et al., 7 Jul 2025).

The paper is also explicit about limitations. Large buildings can exhibit edge localization bias because the $< 0.8$0 input resolution induces compression and detail loss. Very complex buildings may lose fine structural details. Hollow buildings cannot be handled because the formulation does not jointly represent multiple polygons. Multi-part objects remain difficult because disconnected components are not naturally expressed by the current single-polygon output space. These limitations indicate that the model’s strongest regime is regular, single-instance, single-contour extraction rather than fine-grained multi-ring or multi-component vector geometry.

Several misconceptions are therefore addressed by the reported design itself. VectorLLM is not a dense detector, since it depends on an external detector and crop-based processing. It is not a multi-polygon topology engine, since the current formulation targets one contour at a time. Nor should the name be conflated with unrelated “vector” usages in the LLM literature: systems such as "VQ-LLM" (Liu et al., 4 Mar 2025), "VecTrans" (Zheng et al., 25 Mar 2025), "Vec-LUT" (Li et al., 6 Dec 2025), and "Leech Lattice Vector Quantization for Efficient LLM Compression" (Ouderaa et al., 11 Mar 2026) address vector quantization, compiler auto-vectorization, lookup-table vectorization, or lattice-based model compression, whereas VectorLLM denotes an autoregressive multimodal model for remote-sensing contour extraction.

In the paper’s own framing, the broader significance of VectorLLM lies in re-specifying vector extraction as language-style autoregressive geometry generation rather than pixel classification followed by handcrafted polygon processing. A plausible implication is that the model’s value is not only its benchmark gains but its problem formulation: contour extraction becomes a structured multimodal generation task in which topological reasoning, instruction following, and geometric sequencing are handled within a single MLLM.