FineViT: Fine-Grained Visual Encoder

• FineViT is a vision encoder engineered for fine-grained perception using high-resolution training, dense recaptions, and a progressive three-stage curriculum.
• It utilizes a ViT-style backbone with 28 transformer blocks, 2D-RoPE, and native-resolution tiling to maintain local spatial details and support tasks like OCR and grounding.
• Empirical results demonstrate state-of-the-art performance in long-context retrieval and local tasks, consistently outperforming conventional CLIP-based models.

Searching arXiv for FineViT and closely related works mentioned in the provided data. FineViT is a vision encoder introduced for fine-grained perception in multimodal LLMs (MLLMs). It is designed to address limitations attributed to conventional CLIP-based encoders, specifically the loss of visual detail associated with low-resolution pretraining and supervision from noisy, coarse web-crawled image-text pairs. The method replaces coarse web data with dense recaptions and adopts a progressive training paradigm consisting of masked image modeling, global contrastive learning, and local LLM alignment. In the reported experiments, FineViT achieves state-of-the-art zero-shot recognition and retrieval performance, with especially strong results in long-context retrieval, and it consistently outperforms multimodal visual encoders such as SigLIP2 and Qwen-ViT when integrated into MLLMs (Zhao et al., 18 Mar 2026).

1. Architectural specification

FineViT uses a ViT-style encoder backbone with a patch size of $14\times 14$ pixels, native-resolution tiling with inputs adjusted to multiples of $28\times 28$, and 2D Rotary Positional Embeddings (2D-RoPE). The backbone has 28 Transformer blocks, a hidden dimension of 1,536, a feed-forward inner dimension of 4,608, 16 attention heads, SiLU activations, and approximately $0.86$ billion parameters (Zhao et al., 18 Mar 2026).

The design is contrasted with standard CLIP encoders in four explicit respects. First, FineViT is trained at native high resolution throughout, reaching up to $448^2$ during contrastive training and up to $1\text{K}^2$ during LLM alignment, whereas CLIP is often trained at $224^2$. Second, it replaces 1D positional embeddings with 2D-RoPE, which enables continuous high-resolution attention. Third, it uses a progressive three-stage curriculum from self-supervised MIM to global contrastive learning to multi-granularity LLM alignment, rather than a single-stage contrastive procedure. Fourth, it incorporates dense, region-level supervision through FineCap-450M in order to preserve local features.

These choices position FineViT as a high-resolution visual encoder whose training regime is explicitly organized around preserving spatial detail and supporting localized correspondence between visual content and text. A plausible implication is that the architecture is intended not merely to improve global semantic alignment, but to maintain token-level utility for OCR, grounding, and other dense spatial tasks.

2. Progressive training curriculum

FineViT employs a three-stage curriculum.

In Stage I, the encoder is initialized with masked image modeling. The data consist of 1.8 billion unlabeled high-quality images, filtered to minimum side length at least 224 pixels and deduplicated with p-Hash. Training is conducted at $256\times 256$ resolution with 75% masking on patches. DINOv3 provides teacher feature targets, and the loss is

$$L_{\mathrm{MIM}} = \sum_{i\in M} \|\Phi(x)_i - T(x)_i\|_2^2 .$$

The stated goal is to build foundational spatial awareness and reconstruction ability (Zhao et al., 18 Mar 2026).

In Stage II, FineViT performs global contrastive learning on 1.56 billion image-text pairs that were recaptioned by an ensemble of three MLLMs: Qwen2.5-7B, Intern3-VL-8B, and MiniCPM-V-8B. Empty and redundant captions are removed. The training resolution is progressively scaled from $336^2$ to $448^2$, and the text context length is increased from 64 to 256 tokens. The visual branch is initialized from Stage I, while the text branch is initialized from the SigLIP2 giant model, first frozen and then jointly fine-tuned. The pairwise sigmoid contrastive formulation is

$28\times 28$0

and

$28\times 28$1

Its stated purpose is to align global image-text semantics at high resolution while maintaining local details.

In Stage III, the encoder is further trained through local LLM alignment using FineCap-450M. This dataset contains 80 million images with 454 million region-level annotations. The model is trained at resolutions up to $28\times 28$2 on tasks including bbox-to-caption, caption-to-bbox, OCR localization and recognition, and a sequence-to-sequence format trained autoregressively. Visual features $28\times 28$3 are projected into language space by $28\times 28$4 and fed into the LLM, with standard autoregressive cross-entropy

$28\times 28$5

Taken together, the curriculum makes information preservation the organizing principle of training: Stage I supplies spatial priors, Stage II establishes global semantic alignment using recaptioned image-text pairs, and Stage III adds dense local supervision through grounding and OCR-oriented sequence prediction.

3. Data regime and dense recaptions

The Stage II recaptioned corpus draws from COCO, CC3M, CC12M, YFCC15M, LAION, Wukong, DataComp, COYO-700M, SBU, Flickr30k, and additional web crawl data. Images are filtered to short side at least 224 pixels, with near-duplicate removal via p-Hash. A total of 1.8 billion images are passed through Qwen2.5-7B, Intern3-VL-8B, and MiniCPM-V-8B; after removing empty and redundant captions, 1.56 billion image-text pairs remain. The reported resolution distribution indicates that 97.5% of images are at most $28\times 28$6, with broad aspect-ratio coverage up to 3.0 (Zhao et al., 18 Mar 2026).

FineCap-450M supplies the region-level supervision used in Stage III. Images are filtered by short side at least 448 pixels, aspect ratio in $28\times 28$7, and blur, dark, and saturation checks. Region proposals are generated with Grounding-DINO using confidence at least 0.3, bounding-box area at least 1% of the image, class-agnostic NMS with IoU 0.7, and frequency-balanced sampling across 631K categories. Captioning is performed with Qwen3-VL-32B conditioned on crop plus global context, while PaddleOCR is used for natural and document text regions.

The dataset statistics are as follows.

Component	Count	Average tokens
Global captions	63 M	211
Region captions	226 M	56.6
OCR natural	142 M	3.9
OCR document	86 M	4.3

Within the 454 million region-level annotations, the paper further distinguishes 226 million general region captions, 142 million “natural-image” OCR captions, and 86 million document OCR captions. The general region captions span 631K categories. This suggests that FineCap-450M is not restricted to object naming, but is explicitly structured to cover object parts, text-bearing regions, and localized visual-textual correspondences at multiple granularities.

4. Optimization and training setup

The three training stages use distinct data scales, resolutions, learning rates, and trainable modules.

Stage	Data size	Resolution	LR
I: MIM	1.8 B images	$28\times 28$8	$28\times 28$9
II: Contrastive	1.56 B pairs	$0.86$0	$0.86$1
III: LLM Align	0.5 B regions	$0.86$2	$0.86$3

The corresponding batch sizes are 4,096 for Stage I, 49,152 for Stage II, and 4,096 for Stage III. The trainable modules are, respectively, the ViT alone; the ViT plus text encoder; and the ViT, projector, and LLM. Across all stages, the optimizer is AdamW, training uses BF16 precision, and the scheduler is cosine decay. Data augmentation follows the standard CLIP recipe: random crop, horizontal flip, and color jitter. The total number of tokens seen in Stage II is approximately 9.3 billion (Zhao et al., 18 Mar 2026).

These settings indicate that the progressive curriculum is accompanied by a corresponding optimization schedule: the earliest stage emphasizes high-volume self-supervision, the middle stage emphasizes very large-batch cross-modal alignment, and the final stage lowers the learning rate while coupling the visual encoder to an LLM-facing autoregressive objective. A plausible implication is that the schedule is intended to reduce catastrophic loss of low-level spatial information during multimodal alignment.

5. Empirical performance

In zero-shot classification and retrieval, FineViT/14 is reported with 84.2 on IN-1k, 75.5 on IN-v2, and 88.7 on IN-R. On retrieval, it achieves 60.7 on COCO T2I, 80.7 on COCO I2T, 84.8 on Flickr T2I, and 96.7 on Flickr I2T. The comparison table includes SigLIP-so400m/14, SigLIP2-so400m/14, SigLIP2-g/16, and Seed-ViT/14. Within that table, FineViT attains the strongest reported values on IN-R, COCO T2I, COCO I2T, and Flickr I2T, while not leading on every classification or retrieval column (Zhao et al., 18 Mar 2026).

The strongest contrast appears in long-text zero-shot retrieval. FineViT/14 records Recall@1 scores of 84.8 and 83.4 on DCI T2I and I2T, 99.7 and 99.8 on IIW T2I and I2T, and 99.1 and 98.9 on Urban-1k T2I and I2T. The comparison set includes SigLIP-so400m/14, SigLIP2-so400m/14, LongCLIP-L/14, and FixCLIP-L/14, and FineViT leads all listed models on all six long-context retrieval columns.

When integrated into MLLMs, FineViT-VL with Qwen3-1.7B is reported to outperform Qwen3-VL and Intern3.5-VL across four broad categories: general VQA, multimodal reasoning, OCR/Chart, and grounding/counting. The largest gains are stated to occur on localized tasks such as OCR and grounding, including gains of +3 to +18 points on DocVQA, InfoVQA, and CountBenchQA.

The results support the paper’s central claim that fine-grained perception remains a bottleneck for multimodal systems if the visual encoder is trained primarily under coarse global objectives. The especially large margins in long-context retrieval and localized downstream tasks suggest that the dense recaption and local alignment stages are not merely auxiliary refinements, but core contributors to the final behavior.

6. Ablations, interpretation, and limitations

The ablation studies isolate the contribution of the progressive stages. Moving from MIM to contrastive training yields +4.3 overall in general reasoning. Moving from contrastive training to LLM alignment yields +9.7 in local tasks, with OCRBench increasing from 681 to 745 and CountBenchQA increasing by +18.7 points (Zhao et al., 18 Mar 2026).

A second ablation compares frozen and unfrozen visual backbones during LLM alignment. Updating the visual encoder during this stage adds +5 to +15 points on fine-grained tasks. This result is consistent with the paper’s broader emphasis on preserving and refining local features rather than treating the vision tower as a fixed frontend.

The paper’s discussion attributes the observed gains to three factors. High-resolution global pretraining preserves spatial detail that is lost in $0.86$4 regimes. Dense region captions in FineCap-450M provide explicit supervision for object parts, textures, text regions, and bounding-box grounding. Joint LLM alignment enforces correspondence between localized visual tokens and textual sequences, improving spatial precision.

The stated limitations are also specific. The current focus is on static images, and temporal reasoning on video remains unexplored. Construction of FineCap-450M is compute-intensive, and lighter annotation strategies are identified as a potential direction for investigation. The discussion also identifies integration with larger backbone sizes beyond 1 billion parameters and with extended-context LLMs as future work. These limitations indicate that the method’s current scope is concentrated on dense image understanding rather than multimodal temporal reasoning or compute-efficient data curation.

7. Position within multimodal vision research

FineViT is presented as a new baseline for fine-grained visual perception built around a high-resolution, from-scratch encoder and dense recaption supervision. Relative to conventional CLIP-style practice, the method replaces single-stage contrastive pretraining with a three-stage curriculum and supplements global image-text alignment with region-level captioning, grounding, and OCR-oriented supervision (Zhao et al., 18 Mar 2026).

Its empirical profile is correspondingly uneven in a technically informative way. FineViT does not dominate every zero-shot classification column in the reported comparisons, but it is markedly stronger on long-text retrieval and localized multimodal understanding. This pattern suggests that the contribution is not simply a generic increase in scale or parameter count. Rather, the reported evidence is more consistent with a redistribution of representational capacity toward dense spatial semantics and local text-visual correspondence.

In that sense, FineViT occupies a specific niche within multimodal vision research: a vision encoder optimized for scenarios in which image understanding depends on preserving local detail across pretraining, cross-modal alignment, and downstream LLM integration. The released FineViT and FineCap-450M are framed accordingly as baselines for fine-grained visual perception.