Jina-VLM: Small Multilingual Vision Language Model (2512.04032v2)

Abstract: We present Jina-VLM, a 2.4B parameter vision-LLM that achieves state-of-the-art multilingual visual question answering among open 2B-scale VLMs. The model couples a SigLIP2 vision encoder with a Qwen3 language backbone through an attention-pooling connector that enables token-efficient processing of arbitrary-resolution images. The model achieves leading results on standard VQA benchmarks and multilingual evaluations while preserving competitive text-only performance. Model weights and code are publicly released at https://huggingface.co/jinaai/jina-vlm .

• The paper presents a 2.4B parameter vision-language model that achieves state-of-the-art multilingual VQA and maintains robust text-only performance.
• It introduces innovative token-efficient tiling and attention pooling that reduce visual tokens by 4x while preserving fine-grained spatial detail.
• The model integrates diverse multilingual datasets and a two-stage training regimen to prevent catastrophic forgetting and enhance cross-modal feature adaptation.

Jina-VLM: Architecture and Multilingual Vision-Language Alignment

Introduction

The paper "Jina-VLM: Small Multilingual Vision LLM" (2512.04032) introduces a 2.4B parameter vision-LLM (VLM) optimized for multilingual visual question answering (VQA) among open-source models of comparable scale. The model combines a SigLIP2 vision encoder with a Qwen3 language backbone using an attention-pooling connector. Its architectural choices enable token-efficient arbitrary-resolution processing and strong multilingual generalization, with empirical results establishing state-of-the-art performance for multilingual multimodal understanding without significant tradeoffs in English-centric or text-only tasks.

Model Architecture

Jina-VLM couples the SigLIP2-So400M/14-384 vision encoder—a 27-layer ViT architecture—with Qwen3-1.7B-Base as its language decoding backbone. Arbitrary-resolution image processing is addressed by decomposing input images into a grid of up to 12 overlapping $378 \times 378$ pixel tiles, combined with a global thumbnail. Each tile is processed independently; spatial overlap preserves high-frequency information and local context. This design is derived from previous VLMs employing tiling and dynamic resolution strategies, but emphasizes efficient connector design.

The connector concatenates intermediate vision representations from two ViT layers (24 and 18), capturing both detailed and abstract features. Attention pooling over local $2 \times 2$ patch neighborhoods compresses 729 tokens per tile to 182, yielding a $4 \times$ reduction in visual token count while maintaining spatial structure. Projection to the language embedding relies on a SwiGLU transformation. Special visual input tokens delimit images and grid rows, accommodating efficient multimodal fusion.

Notably, the architecture is fully trainable end-to-end. No encoder or backbone freezing is applied during training, enabling optimal cross-modal feature adaptation.

Training Regimen

The model training is organized into two distinct stages:

• Stage 1: Semantic grounding and alignment, utilizing multi-caption datasets with extensive language coverage (PixmoCap, PangeaIns) and 15% text-only data (PleiAS/common_corpus) to preempt text-only degradation.
• Stage 2: Instruction fine-tuning, combining multimodal instruction and VQA datasets (LLaVA OneVision, Cauldron, Cambrian, FineVision) with diverse multilingual tasks and further text-only instructions.

Learning rate schedules favor aggressive connector updates during alignment, and single-source batching is initially preferred for instruction fine-tuning. No component freezing is used. Across both stages, 5M multimodal samples and 12B text tokens in 30+ languages are included, ensuring multilingual coverage.

Empirical Results

Jina-VLM demonstrates consistently strong performance across visual question answering, document/scene understanding, mathematical reasoning, and multilingual benchmarks. Key claims substantiated by evaluation:

• General VQA: Jina-VLM achieves the highest accuracy (72.3%) across eight VQA benchmarks (AI2D, ChartQA, TextVQA, DocVQA, InfoVQA, OCR, SEED-Bench-2-Plus, CharXiv) compared to Qwen2-VL-2B, Qwen3-VL-2B, InternVL3-2B, and InternVL3.5-2B.
• Multilingual Evaluation: The model sets a new state-of-the-art among 2B-scale open VLMs on MMMB (78.8% avg) and Multilingual MMBench (74.3% avg), outperforming comparable models in Arabic, Chinese, Portuguese, Russian, and Turkish.
• Text-Only Preservation: Despite multimodal training, text-only performance remains robust; Jina-VLM matches or exceeds the Qwen3 backbone on ARC-C and HellaSwag and demonstrates minimal degradation on MMLU and GSM-8K.
• Hallucination Robustness: The model attains best-in-class scores on POPE (90.3%), indicating effective hallucination mitigation.
• Mathematical and Multi-image Reasoning: Results on MathVista, MMMU, and LogicVista benchmarks are competitive with InternVL family models; multi-image performance trails slightly, commensurate with training data composition.

These results support the assertion that small-scale models can be architected for broad and multilingual efficacy in vision-language tasks without severe tradeoffs in text reasoning or standard VQA domains.

Technical Contributions and Innovations

• Attention-based Token Pooling: By compressing patch-level features via attention pooling and concatenation across distinct encoder layers, the model achieves $4 \times$ reduction in visual tokens per tile, mitigating compute/memory costs for high-resolution input and allowing arbitrarily large images to be processed with manageable token budgets.
• Tiling and Thumbnail Fusion: Overlapping tiling ensures spatial context and fine detail retention, while including a global thumbnail augments coarse scene context.
• Multilingual Data Integration: Explicit incorporation of diverse multilingual datasets at both alignment and instruction-tuning stages prevents the characteristic drop in non-English performance. The balanced, data-driven approach extends beyond English-centric benchmarks.
• Text-Only Performance Safeguards: Inclusion of text-only data during both training stages neutralizes catastrophic forgetting, a common VLM concern, with minimal tradeoff even when employing aggressive multimodal objectives.

Practical and Theoretical Implications

Jina-VLM demonstrates that small VLM architectures (≤2.5B parameters) can rival or exceed larger and more resource-intensive models in multilingual and multimodal domains for open-source deployments. This has direct consequences for democratized model training, inference cost reduction, and broad accessibility. From a theoretical stance, the efficacy of mid-layer feature fusion and attention pooling suggests new directions for vision-language connectors, potentially generalizing to higher model scales or more complex modalities.

The observed preservation of text-only and multilingual capability under aggressive multimodal training implies that catastrophic forgetting can be controlled with data diversity and curriculum design. The results encourage further investigation into resolution-agnostic vision encoding and token-efficient multimodal fusion, particularly for scenarios where high-resolution visual detail is critical (e.g., medical imaging, OCR-heavy domains).

Future Directions

The computational overhead associated with multi-tile processing scales with resolution; more algorithmic work on efficient memory and compute scheduling is warranted. Additionally, while the current model sidesteps explicit safety and alignment training, integrating safety-critical objectives (e.g., toxic content filtering, adversarial robustness) will be essential for wider deployment—especially in multilingual settings. Scaling this architecture and training recipe to larger backbone models remains an open question for future work.

Conclusion

Jina-VLM establishes a state-of-the-art standard for small-scale multilingual vision-LLMs in both multimodal and text-only domains. Its architectural innovations—attention-pooling connectors and tile-based visual encoding—sustain high performance and efficiency. The model validates that multilingual vision-language understanding and reasoning is realizable at scale and cost favorable to open research and deployment. Further advances should concentrate on scalable resolution handling, robust safety alignment, and extension to larger backbone frameworks.