FineCap-450M: Fine-Grained Region Annotation Dataset

• The paper introduces FineCap-450M as a high-quality, region-level recaption dataset integrated in FineViT’s Stage III to improve local perception, OCR, grounding, and counting tasks.
• FineCap-450M comprises over 454M annotated regions across 80M images, combining global captions, local captions, and OCR data for dense spatial supervision.
• The dataset is curated using advanced methods including Qwen3-VL, Grounding DINO, and PaddleOCR to generate context-aware local annotations and balanced category sampling.

Searching arXiv for papers mentioning FineCap-450M and directly related work. FineCap-450M is a large-scale fine-grained, multi-granularity recaption dataset introduced as the Stage III supervision corpus in FineViT’s progressive training pipeline. In the FineViT framework, it is not the billion-scale contrastive pretraining corpus and it is not a 450M-parameter model; rather, it is the dataset used for the final LLM-alignment stage to teach local perception, region-language grounding, OCR-sensitive understanding, and coordinate-language interconversion. The paper positions FineCap-450M as a response to the limitations of coarse web image-text pairs, arguing that dense recaptions and region-level supervision are required to improve OCR, grounding, counting, and other dense spatial tasks (Zhao et al., 18 Mar 2026).

1. Position within the FineViT training curriculum

FineCap-450M belongs specifically to the third stage of FineViT’s three-stage progressive training procedure. The pipeline is explicitly separated into: Stage I masked image modeling on unlabeled images, Stage II contrastive alignment on global recaptioned image-text pairs, and Stage III LLM autoregression with multi-granularity alignment using FineCap-450M (Zhao et al., 18 Mar 2026).

This placement is central to its interpretation. Stage I uses only unlabeled images for foundational visual perception and geometry. Stage II uses 1.56B global recaptioned image-text pairs for global semantic alignment. FineCap-450M enters only in Stage III, where the objective shifts from global alignment to local perception. The Stage III corpus is described as containing over 450M region-level QA samples, and the training modules at this stage are the ViT, projector, and LLM, with resolution scaled to 1K and a reported sample count of 0.5B. The paper’s framing is that FineCap-450M provides the supervision required to convert a strong global encoder into one with fine-grained spatial and text-sensitive capability.

The intended distinction between “dense recaptions” and conventional web captions is also stage-specific. In Stage II, density primarily means more detailed global recaptioning. In FineCap-450M, density becomes explicitly local: captions are attached to localized regions, enriched with OCR-specific supervision, and reformulated into sequence-to-sequence tasks such as bbox-to-string, string-to-bbox, bbox-to-ocr, and ocr-to-bbox. A plausible implication is that the dataset’s role is less to improve coarse semantic retrieval than to serve as the alignment substrate for localized reasoning and grounded generation.

2. Dataset composition and scale

The paper describes FineCap-450M as the largest fine-grained annotated dataset to date. Its reported structure combines global captions, local captions, rich-text OCR, and document OCR, with 63M images carrying global captions and 454M total regions distributed across three region categories (Zhao et al., 18 Mar 2026).

Component	Count	Average token length
Global captions	63M images	211.25
Local captions	226M regions	56.58
Rich-text OCR	142M regions	3.88
Doc OCR	86M regions	4.33

The paper’s main table also reports 631,252 categories for local captions. However, the appendix prose gives 63,125 classes for general local regions. This discrepancy appears in the paper itself. The most defensible reading is that FineCap-450M is intended to have extremely broad category coverage, but the exact category count is inconsistent across sections of the source.

The appendix broadens the description of the Stage III corpus further, stating that it contains 80M images and 454M regions, consisting of 63M general images and 17M OCR-specific images. This clarifies that the full Stage III alignment corpus is somewhat broader than the 63M globally captioned images emphasized in the main table. The dataset therefore spans natural scenes, text-rich environments, and document-only scenarios, with supervision ranging from long global captions to very short OCR strings.

This composition matters because the paper repeatedly contrasts FineCap-450M with raw web image-text data. Raw web pairs are characterized as noisy, brief, misaligned, and globally descriptive. FineCap-450M instead adds region-centric supervision, OCR augmentation, context-aware local recaptioning, and category-balanced sampling. In the paper’s terms, it is not merely a cleaner caption corpus but a spatially grounded, OCR-augmented supervision resource.

3. Curation pipeline and annotation mechanics

FineCap-450M is built on top of the 1.56B-sample global recaption pool created for Stage II, after which a dedicated curation pipeline selects higher-quality images and constructs region-level annotations (Zhao et al., 18 Mar 2026).

The image filtering stage removes images with short side smaller than 448 px, blurry or out-of-focus images, excessively bright or dark images, and unnaturally high color saturation images; the main text states that roughly 3% of images are excluded. The appendix adds an aspect-ratio constraint of $[1/3, 3]$. At this stage, images are annotated at both global and region levels using Qwen3-VL 32B, which the paper says was chosen to balance performance and computation.

Region proposal generation begins by extracting nouns and noun phrases from global recaptions, then using Grounding DINO to generate candidate bounding boxes for those nouns. Candidate regions are filtered by confidence below 0.3, followed by NMS; the appendix adds a minimum area threshold of 1% of image area and class-agnostic NMS with IoU threshold 0.7. After proposal filtering, the dataset is balanced by category frequency. The appendix specifies that balancing is performed over 10M-image batches, that categories with fewer than 1,000 instances are fully retained, and that overrepresented classes are downsampled proportionally with a cap of 100,000 instances per batch.

Local caption generation is explicitly context-aware. For each region, the pipeline feeds both the local crop and the full image into an MLLM, again using Qwen3-VL 32B, to produce captions that preserve local detail while incorporating global context. The paper emphasizes that these local captions go beyond object naming and include textures, materials, posture, brand or logo identity, and relative or positional cues.

OCR augmentation is introduced because generic local captioning performed poorly on text-rich regions, often producing inaccurate descriptions or generic outputs such as “text.” To address this, PaddleOCR is used to detect text in natural images and documents, and MLLM-generated captions are replaced with OCR text for rich-text regions. This establishes a structurally different supervision channel for textual content, rather than treating OCR as a special case of free-form captioning.

4. Supervision format in Stage III alignment

FineCap-450M supervises Stage III through autoregressive MLLM training rather than through a separate local region-text contrastive objective. The paper gives the Stage III loss as

$\mathcal{L}_{\text{MLLM} = - \sum_{i=1}^{L} \log P\left(y_i \mid y_{<i}, g\left(\Phi(x)\right)\right)$

where $\Phi(x)$ is the visual encoder output, $g(\cdot)$ is a trainable projector into language space, and $y_i$ are target tokens (Zhao et al., 18 Mar 2026).

This is a consequential design choice. FineCap-450M does not appear as a specialized matching loss over region embeddings. Instead, local supervision is reformulated as text generation. The paper states that Stage III uses tasks including bbox-to-string, string-to-bbox, bbox-to-ocr, and ocr-to-bbox. In effect, the model is trained to translate between localized image regions, textual spans, OCR strings, and coordinates within a unified autoregressive interface.

The supervision signal provided by FineCap-450M is therefore multi-granular in two senses. First, it covers both global and local levels through global captions, local captions, rich-text OCR, and document OCR. Second, it supports bidirectional mappings between spatial and linguistic representations. This suggests that the dataset is intended not just to sharpen recognition of local content, but to align the encoder with an LLM in a form usable for downstream MLLM tasks requiring grounding, OCR, and coordinate-sensitive reasoning.

The architecture around this stage is also relevant. FineViT itself is a 28-layer ViT with patch size $14 \times 14$, hidden size 1536, intermediate size 4608, 16 heads, SiLU, and 2D RoPE, with about 0.86B parameters. In Stage III, the image is encoded by $\Phi(x)$, mapped into language space by a projector, and consumed by the LLM; the training modules are the ViT, projector, and LLM. The paper later notes that unfreezing the visual backbone is important for local tasks, which reinforces that FineCap-450M is meant to reshape visual features rather than merely supervise a frozen encoder.

5. Empirical effect on FineViT

The paper attributes the most pronounced Stage II-to-Stage III improvements to the introduction of FineCap-450M. General-task average improves from 53.08 to 54.83, whereas local-task average increases from 64.80 to 73.29, indicating that the dataset’s primary effect is on local perception rather than on broad semantic competence (Zhao et al., 18 Mar 2026).

The reported gains are substantial on dense spatial and text-sensitive benchmarks. From Stage II to Stage III, OCRBench rises from 681 to 745, DocVQA from 79.85 to 85.33, ChartQA from 73.00 to 77.56, InfoVQA from 48.70 to 56.25, TextVQA from 73.82 to 77.35, CountBenchQA from 72.07 to 90.76, PointBench from 59.04 to 67.55, RefCOCO average from 72.93 to 83.70, RefCOCO+ average from 63.18 to 73.87, RefCOCOg average from 69.51 to 80.92, VisualWebBench from 53.03 to 56.21, and 100-FSC-147 val from 48.57 to 70.33. The paper explicitly highlights the OCRBench gain of +64 points and the CountBenchQA gain of +18.69 points.

These results are consistent with the stated design of FineCap-450M. OCR-heavy benchmarks benefit from the rich-text OCR and document OCR channels. Grounding benchmarks benefit from region-caption pairing and coordinate-language supervision. Counting and point-based tasks benefit from improved preservation of local visual detail and from the region-centric training format.

The same pattern appears in comparisons against alternative visual encoders within MLLM settings. With Qwen3-1.7B, FineViT achieves local average 70.07 against 58.76 for SigLIP2-naflex; with Qwen3-8B, FineViT reaches local average 73.49 against 67.15. The paper interprets these gains as evidence that Stage III multi-granularity alignment, enabled by FineCap-450M, alleviates localization bottlenecks and improves document parsing and visual grounding. By contrast, the paper associates FineViT’s strongest long-context retrieval results primarily with Stage II global dense recaptions rather than with FineCap-450M specifically, which is an important distinction.

6. Ambiguities, limitations, and common misconceptions

A common misconception is to treat FineCap-450M as a model because of the “450M” suffix. In the FineViT paper, FineCap-450M is a dataset, not a 450M-parameter system (Zhao et al., 18 Mar 2026).

The paper also leaves several ambiguities. The clearest is the inconsistency in category count: the main table reports 631,252 categories, while appendix prose reports 63,125 classes. Another ambiguity concerns corpus scope: the main table foregrounds 63M images with global captions, whereas the appendix states that the full Stage III corpus contains 80M images including 17M OCR-specific images. These are not necessarily contradictions, but they do indicate that the precise boundary between “FineCap-450M” and the broader Stage III corpus is not fully harmonized across the paper.

The curation pipeline is computationally intensive and heavily automated. It depends on Qwen3-VL 32B for global and local recaptioning, Grounding DINO for region proposals, and PaddleOCR for OCR-specific supervision. This introduces plausible error propagation from the recaptioning model, the detector, and the OCR stack. The paper partially mitigates this through image and region filtering, OCR replacement for text-rich regions, and category balancing, but it does not report a human-annotation validation rate or detailed error analysis for the dataset itself.

The paper’s formal limitation is that FineViT is restricted to static images, with video and temporal reasoning left for future work. By extension, FineCap-450M is also an image-only resource. The dataset is therefore best understood as a large-scale alignment corpus for fine-grained static-image perception in MLLMs, especially where local grounding, OCR, counting, and coordinate-sensitive reasoning are central.