VLM-FO1: Enhancing Vision-Language Perception

• VLM-FO1 is a plug-and-play perception enhancement framework that redefines object localization by converting continuous coordinates into discrete token-based region selection.
• It integrates an external proposal generator with dual-encoder region representations and a symbolic region-index interface to boost fine-grained perception tasks.
• A two-stage training strategy preserves the base model’s global reasoning while significantly improving object grounding, counting, and region reasoning.

VLM-FO1 is a plug-and-play perception enhancement framework for vision-LLMs that reframes object-centric localization from coordinate generation into feature retrieval and token-based referencing (Liu et al., 30 Sep 2025). It is designed to attach to a pretrained VLM without retraining that base model from scratch, and it targets the longstanding gap between high-level multimodal reasoning and fine-grained perception in tasks such as object grounding, referring expression comprehension, counting, OCR, region captioning, and region question answering. Its central claim is that language-centric autoregressive architectures are poorly matched to exact numerical coordinate emission, whereas selecting among precomputed region tokens is structurally aligned with LLM decoding. The framework therefore combines an external proposal generator, a dual-encoder region representation stack, and a symbolic region-index interface inside the LLM.

1. Problem setting and conceptual shift

Modern VLMs such as LLaVA, Qwen-VL, and InternVL are strong at global visual understanding but remain weak on fine-grained perception tasks requiring precise spatial localization, robust handling of many instances, and exact numerical coordinates (Liu et al., 30 Sep 2025). The reported contrast is stark: on COCO, strong detection models achieve 50–60 mAP, while Qwen2.5-VL-72B reaches recall $< 40\%$. VLM-FO1 attributes this not primarily to missing data, but to a core architectural mismatch: LLMs are optimized for discrete token generation, whereas bounding boxes are continuous tuples such as $(x_1,y_1,x_2,y_2)$, and autoregressive coordinate emission is brittle because a single token error invalidates the full box.

This diagnosis extends to quantized-coordinate formulations. Even when coordinates are discretized, the model must still emit long structured sequences, incurs quantization error at high resolution, and struggles with negative categories and multi-instance disambiguation. VLM-FO1 therefore reformulates localization as retrieval over a finite set of proposal regions. Instead of asking the model to generate coordinates, it asks the model to select or mention symbolic identifiers such as $\texttt{<region0>}$ or $\texttt{<region10>}$, which correspond to precomputed region embeddings.

The resulting inference view is discrete rather than regressive. Conceptually, the LLM solves

$$\arg\max_i p(\texttt{<region}i\texttt{>} \mid \text{prompt}, \{r_j\}),$$

where $\{r_j\}$ is the set of region tokens derived from proposal boxes. This suggests that VLM-FO1 is less a detector in the conventional sense than a mechanism for turning a pretrained VLM into a region-addressable reasoner.

2. Architectural composition

The framework preserves the original VLM and augments it with four additions: an Omni Proposal Network (OPN), a Dual-Vision Encoder, a Hybrid Fine-grained Region Encoder (HFRE), and a token-based referencing system (Liu et al., 30 Sep 2025). Given an image $I$, proposal boxes $B=\{b_1,\dots,b_N\}$, and a text prompt, the original vision tower still produces the global image tokens expected by the base VLM. Fine-grained perception is introduced in parallel through region proposals and region tokens rather than by altering the base image-token pathway.

Proposal generation is decoupled from the VLM. VLM-FO1 uses OPN, a variant of OmDet-Turbo, to identify potential foreground objects and takes the top $N$ proposals per image, up to 100, sorted by confidence. This decoupling is central to the plug-and-play design: the detector can be swapped, and users can in principle provide task-specific proposals or manually specified regions of interest.

The Dual-Vision Encoder consists of a primary and an auxiliary branch. The primary branch is the original VLM vision tower, specifically QwenViT from Qwen2.5-VL-3B, which is semantically aligned with language but not optimized for dense, high-resolution localization. The auxiliary branch is DaViT-Large, a high-resolution specialist that outputs four multi-scale feature maps $\{A_1,A_2,A_3,A_4\}$. HFRE fuses both branches. In the auxiliary branch, the feature maps are upsampled to a common resolution, concatenated, and processed with RoIAlign plus mean pooling to obtain $(x_1,y_1,x_2,y_2)$0 with $(x_1,y_1,x_2,y_2)$1. In the primary branch, a Simple Feature Pyramid inspired by ViTDet converts the last QwenViT feature map into four scales via convolution and deconvolution with strides $(x_1,y_1,x_2,y_2)$2, after which RoIAlign yields $(x_1,y_1,x_2,y_2)$3 with $(x_1,y_1,x_2,y_2)$4.

The hybrid representation is then formed as

$(x_1,y_1,x_2,y_2)$5

followed by addition of sine-cosine positional embeddings derived from the box coordinates,

$(x_1,y_1,x_2,y_2)$6

and projection into the LLM embedding space through a Region-Language Connector,

$(x_1,y_1,x_2,y_2)$7

Each row of $(x_1,y_1,x_2,y_2)$8 is a region token embedding. The design aim is explicit: combine the semantic alignment of the base VLM vision tower with the perceptual precision of DaViT-Large.

3. Token-based referencing and task interface

VLM-FO1 makes visual regions addressable by introducing special region index tokens $(x_1,y_1,x_2,y_2)$9, each paired with a visual region token embedding from HFRE (Liu et al., 30 Sep 2025). The LLM input sequence is structured as

$\texttt{<region0>}$0

where $\texttt{<region0>}$1 is the HFRE-derived embedding for region $\texttt{<region0>}$2. The LLM can then attend jointly over global image tokens, region index tokens, region embeddings, and ordinary text tokens using the base transformer mechanism.

Grounding outputs are expressed through markup tokens $\texttt{<region0>}$3, $\texttt{<region0>}$4, $\texttt{<region0>}$5, and $\texttt{<region0>}$6. A phrase to be grounded is wrapped by $\texttt{<region0>}$7, while the corresponding region indices are wrapped by $\texttt{<region0>}$8. The canonical example is:

“The $\texttt{<region0>}$9people$\texttt{<region10>}$0 $\texttt{<region10>}$1 are dancing.”

This formulation supports many-to-one and one-to-many mappings: multiple regions can ground a single phrase, and multiple phrases can refer to the same region. For simpler tasks such as region QA or region captioning, the model can directly mention region indices in free-form text.

The same interface generalizes across task families. In VLM-FO1’s evaluation, the object-grounding slice covers COCO val2017, ODinW13, and OVDEval; region generative understanding covers LVIS, PACO, COCO-Text, and Ferret Bench; visual region reasoning covers RefCOCO, RefCOCO+, RefCOCOg, HumanRef, CountBench, and PixMo-Count (Liu et al., 30 Sep 2025). The framework’s counting behavior is explicitly characterized as a “Detect-then-Count” pattern: localize instances through region tokens and then count them, rather than directly predicting a numeral from the image.

A common misconception is that token-based referencing eliminates detection. It does not. Proposal generation remains an external stage, and VLM-FO1 can only reason over the regions it is given. Missing or poor proposals therefore remain a fundamental failure source.

4. Two-stage training strategy

VLM-FO1 uses a two-stage training regime intended to add perception without degrading the base model’s general capabilities (Liu et al., 30 Sep 2025). Stage 1 is Region-Language Alignment. In this phase, the entire pretrained VLM—both primary vision encoder and LLM—is frozen, as are the embeddings of the original vocabulary. The trainable components are HFRE, the Region-Language Connector, and the embeddings of new tokens such as $\texttt{<region10>}$2, $\texttt{<region10>}$3, and $\texttt{<region10>}$4. The stage uses COCO, O365, V3Det, GOLDG, and Rexverse-2M to teach the model how region tokens should align with the LLM space.

Stage 2 is Perception Instruction Fine-tuning. Here the auxiliary vision encoder, HFRE, Region-Language Connector, and LLM are unfrozen, while the primary vision encoder remains frozen as a stable semantic anchor. The data mixture is broad: detection data from COCO, V3Det, O365, VAW, VisDrone2019, and LVIS; REC data from RefCOCO/+/g, FineCOPS-Ref, GRES, and CREC; grounding data from GRIT; region caption data from PACO, Visual Genome, Osprey, ShareGPT4V, and Rexverse-2M; region reasoning data from VisualCoT and HumanRef-CoT; counting data from COCO, LVIS, HumanRef-CoT, CrowdHuman, and TallyQA; region QA data from Osprey, VCR, MDVP, and DoclayNet; OCR data from MLT2019, ICDAR15, and CurvedSynText150k; and general VLM instruction data from OmChat-SFT.

The optimization objective remains standard autoregressive language modeling rather than explicit contrastive or regression loss:

$\texttt{<region10>}$5

Alignment is therefore enforced implicitly by requiring correct text, correct region indices, and correct grounding markup. The training recipe also includes negative supervision: for 20% of detection-related data, the model is asked for objects that do not exist in the image, and the correct output is effectively refusal or “none.” This is intended to reduce hallucinations and improve handling of negative categories.

The implementation is concrete. The base VLM is Qwen2.5-VL-3B; the auxiliary encoder is DaViT-Large; OPN contributes the top 100 proposals per image; Stage 1 uses learning rate $\texttt{<region10>}$6; Stage 2 uses $\texttt{<region10>}$7; and only the new token embeddings are trainable among vocabulary parameters. A second misconception is that fine-grained perception necessarily requires unfreezing the whole base VLM. VLM-FO1’s ablations show the opposite: keeping QwenViT frozen produces the best average score.

5. Empirical performance and ablation findings

VLM-FO1 is evaluated from three perspectives—object grounding, region generative understanding, and visual region reasoning—and is also checked on general VLM benchmarks through OpenCompass (Liu et al., 30 Sep 2025). A representative subset of the reported numbers is summarized below.

Evaluation slice	VLM-FO1-3B	Comparison highlight
COCO val2017 mAP	44.4	Qwen2.5-VL-7B: 17.7; Griffon-G-7B: 40.2
ODinW13 mAP	44.0	Matches or exceeds models reported under simplified protocols
OVDEval mAP	43.7	OmDet-Turbo: 25.9; VLM-R1-7B: 31.0
LVIS SS / S-IoU	92.4 / 86.4	New SOTA on LVIS
COCO-Text	59.0	Previous best listed: 45.4
Ferret Bench referring reasoning	80.1	Ferret-v2-13B: 79.4
HumanRef	Precision 87.1, Recall 83.3, DF1 82.6	Surpasses RexSeek-7B and Molmo-7B-D
PixMo-Count	86.0	Slightly better than Molmo-72B at 85.2
OpenCompass average	64.6	Base Qwen2.5-VL-3B: 64.5

The object-grounding results are particularly important because they directly target the paper’s motivating gap. On COCO val2017, GPT-4o obtains 3.1 mAP, Qwen2.5-VL-7B 17.7, Griffon-G-7B 40.2, and VLM-FO1-3B 44.4. On OVDEval, which emphasizes open-vocabulary detection with hard negatives, VLM-FO1-3B reaches 43.7 mAP, compared with 25.9 for OmDet-Turbo and 31.0 for VLM-R1-7B. This supports the paper’s claim that token referencing plus the underlying VLM’s semantic knowledge is especially effective when linguistic disambiguation and negative categories matter.

Region-level understanding is similarly strong. VLM-FO1-3B reaches Selection Score 92.4 and S-IoU 86.4 on LVIS, 88.1 and 77.6 on PACO, 59.0 on COCO-Text, and 80.1 on Ferret Bench referring reasoning. In visual region reasoning, it is reported as consistently comparable or better than InternVL2.5-8B, Groma-7B, Qwen2.5-VL-7B, and ChatRex-7B across RefCOCO, RefCOCO+, and RefCOCOg, and it achieves 87.8 on CountBench and 86.0 on PixMo-Count. The HumanRef result—Precision 87.1, Recall 83.3, DF1 82.6—indicates that the method also handles human-centric REC with hard negatives.

The ablation studies identify the main design drivers. Full hybrid fusion with frozen QwenViT reaches an average score of 67.65; unfreezing QwenViT reduces this to 66.35; using only auxiliary region features gives 65.89; using only primary region features gives 65.76 with QwenViT frozen and 66.15 with QwenViT unfrozen. SimpleFP improves average score from 64.94 to 66.15 when using only primary region features. These results indicate that both branches contribute and that the primary encoder is most useful as a fixed semantic anchor rather than a freely updated localization backbone.

6. Limitations, misconceptions, and broader significance

The main limitation is proposal dependence (Liu et al., 30 Sep 2025). Because localization is reframed as selection over proposed regions, VLM-FO1 cannot recover objects that OPN fails to propose. This is not incidental but structural: the framework is a two-stage system, and the ceiling on recall is bounded by proposal quality. The paper also notes additional compute and latency costs from the extra proposal stage, the DaViT-Large auxiliary encoder, and HFRE.

The qualitative analysis implies further practical limits. Remaining difficulty is expected for extremely small objects beyond proposal recall, scenes with ambiguous or noisy textual descriptions, and cases where the proposal set is wrong. The paper does not present extensive failure taxonomies, but these constraints follow directly from the architecture. Another misconception is that VLM-FO1 merely replaces coordinate outputs with arbitrary symbolic tags. In fact, the region tokens encode both semantics and spatial information through hybrid features and positional embeddings; the symbolic layer is coupled to a region representation that remains explicitly grounded in proposal geometry.

The broader significance lies in the demonstration that fine-grained perception can be added to a pretrained VLM without sacrificing its general multimodal competence. On OpenCompass, the average score is 64.6 for VLM-FO1-3B versus 64.5 for the base Qwen2.5-VL-3B, with individual benchmark differences within roughly 1–2 points. This suggests that the two-stage recipe, mixed instruction tuning, and frozen primary encoder are sufficient to avoid catastrophic forgetting while adding specialized perception behavior.

VLM-FO1 therefore establishes a general paradigm for perception-aware VLMs: move precise localization into proposal generation and region encoding, expose those regions as addressable tokens, and let the LLM operate on symbolic indices rather than coordinates. A plausible implication is that the same recipe can be extended beyond boxes to richer perceptual units such as segmentation masks or 3D regions, an extension the paper explicitly identifies as a future direction.