---
title: 'VARCO-VISION-2.0: Bilingual Open-Weight VLM'
url: https://www.emergentmind.com/topics/varco-vision-2-0
type: topic
---

# VARCO-VISION-2.0: Bilingual Open-Weight VLM

VARCO-VISION-2.0 is an open-weight bilingual vision-language model (VLM) for Korean and English developed by NCSOFT and presented as a successor to VARCO-VISION-14B. The model family is defined by three closely linked aims: stronger Korean-localized multimodal performance, explicit support for multi-image understanding, and layout-aware OCR that predicts both textual content and spatial location. The released family contains a full-scale 14B model and a lightweight 1.7B model, both implemented in the Hugging Face Transformers format and compatible with vLLM. In the accompanying technical report, the 14B model is described as achieving 8th place on the OpenCompass VLM leaderboard among models of comparable scale, while an external medical benchmark study identifies VARCO-VISION-2.0-14B as the strongest Korean-specialized model in its evaluation but still substantially behind the strongest proprietary and general-purpose open models on Korean medical multimodal reasoning [2509.10105, 2602.13650].

## 1. Lineage, scope, and released variants

VARCO-VISION-2.0 belongs to the VARCO-VISION family introduced as an open-source Korean-English bilingual VLM line oriented toward image-text understanding, image-grounded generation, OCR, grounding, and referring. The earlier VARCO-VISION paper centered on a single released model, VARCO-VISION-14B, built from Qwen-2.5-14B-Instruct, SigLIP, and an MLP projector in a LLaVA-OneVision-style configuration, and paired that release with five Korean multimodal benchmarks. VARCO-VISION-2.0 preserves the family’s bilingual and open-weight orientation while shifting to Qwen3 backbones, SigLIP2, multi-image support, and a revised preference-optimization stage [2411.19103, 2509.10105].

Two variants are released. **VARCO-VISION-2.0-14B** is the full-scale model intended for server/cloud deployment, research, and high-quality production inference. **VARCO-VISION-2.0-1.7B** is optimized for on-device deployment such as smartphones and PCs, with lower deployment cost and smaller memory footprint. The technical report states that both models are available on Hugging Face as `NCSOFT/VARCO-VISION-2.0-14B` and `NCSOFT/VARCO-VISION-2.0-1.7B` [2509.10105].

The designation **bilingual** is central rather than incidental. The model is explicitly trained and evaluated for both Korean and English, with Korean query regeneration from English tasks, bilingual OCR data, Korean-localized contextual preference data, and benchmark coverage in both languages. The designation **open-weight** is equally consequential: the model parameters are publicly released rather than restricted to API access, enabling local execution, adaptation, benchmarking, and deployment in data-sensitive environments [2509.10105].

## 2. Architecture and multimodal interface

VARCO-VISION-2.0 is built on the **LLaVA-OneVision** architecture and consists of three main components: a language model backbone, a vision encoder, and a multimodal connector/projector. The language model is **Qwen3-14B** for the 14B variant and **Qwen3-1.7B** for the lightweight variant. The vision encoder is **SigLIP2** with **patch-16** configuration, replacing the earlier family’s SigLIP patch-14 setting. A **two-layer MLP connector** projects visual features into the language-model embedding space [2509.10105].

For a **384 × 384** image, the report gives a concrete token-count tradeoff: **patch-16** produces \(24^2 = 576\) visual tokens, whereas **patch-14** produces \(27^2 = 729\). The change is framed as an efficiency choice, based on the observation that prior work suggested token count has limited effect on overall VLM quality. The model also uses the **AnyRes** strategy from LLaVA-OneVision, with tiled or cropped views and resolution-aware aggregation for arbitrary image resolutions. Training-time maximum grid settings are **\(1 \times 1\)** in Stage 1, **\(4 \times 4\)** in Stage 2, and **\(6 \times 6\)** in Stages 3 and 4; with encoder resolution 384, the \(6 \times 6\) grid corresponds to an effective maximum resolution of \(384 \times 6 = 2304\) [2509.10105].

One of the major architectural changes relative to VARCO-VISION-14B is explicit **multi-image input** support. The technical report states that the model supports multiple images simultaneously and that an internal document QA dataset contains up to **12 Korean-English images per sample**. For multi-image scenarios in Stage 3, a **fixed-size image representation strategy** is used to control sequence length and remain compatible with the context window. The vocabulary size is reported as **approximately 150K tokens**, which is operationally significant because large-vocabulary logit computation becomes a memory bottleneck during training [2509.10105].

A distinctive feature of the family is its structured text interface for OCR and grounding. The model uses **`<gro>`** for grounding and **`<ocr>`** for OCR. Grounding outputs take the serialized form
\[
\texttt{<obj>\{object</obj><bbox>\{x1\}, \{y1\}, \{x2\}, \{y2\}</bbox>\}
\]
with normalized coordinates in \([0,1]\). OCR outputs take the form
\[
\texttt{<char>\{word</char><bbox>\{x1\}, \{y1\}, \{x2\}, \{y2\}</bbox>\}
\]
again with normalized coordinates. The report states that OCR words are segmented at whitespace boundaries and ordered by **y-coordinate clustering**, with reading order **top to bottom, left to right**. The grounding and OCR mechanisms are therefore implemented as **sequence-generation problems** with text-serialized spatial outputs rather than through a separate detector or coordinate-regression head [2509.10105].

## 3. Four-stage curriculum and training system

VARCO-VISION-2.0 is trained with a **four-stage curriculum**, inspired by Qwen2.5-VL and InternVL3, and the report states total training consumption of approximately **6.5B text tokens**, **30.4B image tokens**, and **36.9B total tokens**. The stage-wise allocation is highly skewed toward Stage 3, which uses **5.7B** text tokens and **25B** image tokens, for **30.7B** total [2509.10105].

| Stage | Text tokens | Image tokens |
|---|---:|---:|
| 1 | 4M | 560M |
| 2 | 760M | 4.7B |
| 3 | 5.7B | 25B |
| 4 | 61M | 93M |

**Stage 1, Feature Alignment Pre-training**, trains only the MLP connector while freezing both the vision encoder and the language model. The supervision consists of filtered image-caption data with real-world images and concise English descriptions of key objects. This stage is used to align the independently pretrained visual and linguistic modules and stabilize image-to-text mapping [2509.10105].

**Stage 2, Basic Supervised Fine-tuning**, jointly trains the full model in single-image settings with AnyRes at relatively low resolutions. The described data categories include re-captioned image, chart, and table datasets, together with a bilingual OCR dataset containing Korean and English text, diverse fonts and styles, and both collected and synthetically generated images. The paper provides a standardized OCR-style prompt instructing section-by-section reading from top to bottom and left to right. This stage is intended to build broad multimodal instruction-following, chart and table understanding, and bilingual OCR reading [2509.10105].

**Stage 3, Advanced Supervised Fine-tuning**, is the principal capability-building phase. It introduces higher-resolution perception, multi-image reasoning, document QA, grounding, OCR with localization, and strengthened bilingual instruction following. Korean queries are regenerated from English prompts and answers, and human annotators enrich prompts with output-format instructions such as “Answer the question using a single word or phrase” and “Explain the solution process step by step and provide the final answer.” The report describes an in-house document QA dataset with up to **12 Korean-English images per sample**, plus fine-grained grounding and word-level OCR-localization data [2509.10105].

**Stage 4, Preference Optimization**, applies **Direct Preference Optimization (DPO)** to improve safer responses, reduced hallucination, better cultural and contextual localization, and improved user-intent alignment. A major change from VARCO-VISION-14B is that the previous model updated only **LLM layers** in Stage 4, whereas VARCO-VISION-2.0 **unfreezes the entire model**. The paper describes three preference-data categories: general anti-hallucination image preferences, safety preferences and refusals, and Korean-localized culturally aware responses. It does not print the DPO formula in the report itself, but it explicitly identifies DPO as the method [2509.10105].

Language preservation is treated as a first-order objective. The report attributes strong text-only retention to the **Qwen3** base model, the use of **compact, high-quality multimodal datasets**, and an efficient training strategy that does not excessively erode language competence. This emphasis extends a design principle already visible in the original VARCO-VISION paper, which incorporated text-only data during later stages to preserve backbone language proficiency [2411.19103, 2509.10105].

The training system is unusually explicit about memory management. The technical report states that training used a single node with **8×H100 GPUs** for about **700 wall-clock hours**, or roughly **5,600 GPU-hours**. Memory-efficiency techniques include **FSDP**, **activation checkpointing**, **8-bit Adam**, **Liger kernel**, **CPU offloading in DPO**, and **custom logit chunking**. For the 14B model, the authors also use **merge-train-merge**, merging several Stage 3 checkpoints to initialize Stage 4 and then merging Stage 4 checkpoints to obtain the final model. The 1.7B model does not use checkpoint merging; instead, its vision encoder is initialized from the 14B model after Stage 3, and ablations show that shared initialization helps while freezing hurts [2509.10105].

## 4. Evaluation profile beyond medicine

The 14B model is reported to rank **8th on the OpenCompass VLM leaderboard** among models with fewer than 20B parameters, and its **English benchmark average** is given as **77.2**, compared with **68.1** for VARCO-VISION-1.0-14B, **78.4** for InternVL3-14B, **77.9** for Ovis2-16B, and **76.0** for Qwen2.5-VL-7B. Selected 14B English benchmark results include **RealWorldQA 74.6**, **SEEDBench_IMG 78.0**, **Q-Bench1_VAL 79.9**, **A-Bench_VAL 79.5**, **ChartQA_TEST 84.2**, **MMMU_VAL 61.9**, **MathVista 73.2**, **DocVQA_TEST 90.9**, **TextVQA_VAL 80.4**, and **OCRBench 86.9**. The report explicitly characterizes the model as strongest in real-world perception and spatial tasks, but not best in knowledge-heavy or document-centric tasks [2509.10105].

On **Korean multimodal benchmarks**, the 14B model has an average of **80.7**, ahead of **InternVL3-14B 80.2**, **Qwen2.5-VL-7B 77.2**, **Ovis2-16B 71.4**, and **VV-1.0-14B 77.3**. The key 14B scores are **K-MMBench_DEV 87.7**, **K-MMStar 63.6**, **K-SEED 77.2**, **K-LLaVA-W 96.5**, and **K-DTCBench 78.3**. The **K-LLaVA-W** result is described as best by a wide margin and is used to support the claim of improved Korean generation and dialogue ability. The relatively low **K-DTCBench** score is presented as evidence that structured document, table, and chart understanding remains a comparative weakness [2509.10105].

On **text-only benchmarks**, the 14B model has an average of **75.5**, compared with **74.4** for Ovis2-16B, **71.9** for InternVL3-14B, and **60.2** for VV-1.0-14B. Selected scores are **MMLU 77.9**, **MT-Bench 89.8**, **KMMLU 57.5**, **KoMT-Bench 78.3**, and **LogicKor 74.0**. This continues a pattern already emphasized by the earlier VARCO-VISION work, where the authors argued that multimodal training should not destroy text competence and reported unusually strong text-only retention for a VLM [2411.19103, 2509.10105].

OCR is one of the most favorable parts of the 2.0 evaluation. On recognition accuracy benchmarks, the report gives **VV-2.0-14B** as **CORD 97.1**, **ICDAR2013 95.7**, **ICDAR2015 79.4**, and **Average 90.7**. The 1.7B model is also strong, with **Average 88.6**, and even slightly exceeds the 14B model on **ICDAR2013** with **95.9**. The report states that the 14B model nearly matches **CLOVA OCR** on average, beats it on **CORD**, and clearly surpasses **PaddleOCR** and **EasyOCR** on average. The OCR evaluation further reports that upscaling to a minimum longer side of **2304 px** significantly improves OCR because it matches the finest grid tokenization supported by the model [2509.10105].

The 1.7B variant occupies a distinct position. Its **English average** is **68.3**, below **InternVL3-2B 70.5** and **Ovis2-2B 71.7**, but it is described as especially strong in **physical-world visual understanding**, with **RealWorldQA 66.8**, **LLaVABench 77.3**, **Q-Bench1_VAL 72.3**, **MathVista 61.1**, and **OCRBench 83.0**. On Korean multimodal benchmarks, the 1.7B model has an average of **65.4**, which the report identifies as best among lightweight models, though it suffers notably on **K-MMStar 40.8** due to response-format violations under Korean instructions [2509.10105].

## 5. KorMedMCQA-V and Korean medical multimodal reasoning

An important external evaluation of VARCO-VISION-2.0-14B appears in **KorMedMCQA-V**, a Korean medical licensing-exam-style multimodal multiple-choice question answering benchmark for VLMs. The benchmark contains **1,534 multimodal questions** drawn from the **2012–2023** Korean Medical Licensing Examination, paired with **2,043 images**, and each item is a **5-way MCQA** problem in which the model receives the full question stem, five answer options, and one or more associated images, then outputs a single option label. The evaluation is **zero-shot**, **closed-book**, uses a **single prompt template**, requires output in **JSON format**, and scores predictions by exact match on the option label; for open-source models, the paper runs **three random seeds (42, 43, 44)** and reports mean ± std [2602.13650].

Within this benchmark, the medical paper treats VARCO-VISION-2.0 as a **Korean-specialized** and **open-source** model. It does not provide a bespoke architectural description of VARCO-VISION-2.0; instead, it identifies the evaluated checkpoints as **NCSOFT/VARCO-VISION-2.0-1.7B** and **NCSOFT/VARCO-VISION-2.0-14B**, both run with **vLLM**, under the same unified protocol as the other open-source baselines. This makes the benchmark comparison operationally consistent, while the authors also note broader comparability caveats arising from each model’s own default processor and default or recommended generation settings [2602.13650].

The headline result is that **VARCO-VISION-2.0-14B** achieves **43.2%** accuracy, reported in Appendix Table A3 as **\(43.2 \pm 0.1\)**. This makes it the **best Korean-specialized model** in the study, ahead of **A.X-4.0-VL-Light 41.8**, **Kanana-1.5-V-3B-Instruct 30.7**, **HyperCLOVAX-SEED-Vision-Instruct-3B 26.3**, and **VARCO-VISION-2.0-1.7B 24.4**. At the same time, it is **36th of 51 overall**, with **35 models scoring higher** and only **15 scoring lower**, placing it in the lower third of the leaderboard. The strongest proprietary model, **Gemini-3.0-Pro**, reaches **96.9%**; the strongest open-source model, **Qwen3-VL-32B-Thinking**, reaches **83.7%**; and the strongest medical-specialized model, **Lingshu-32B**, reaches **66.1%** [2602.13650].

The modality breakdown shows generally low and uneven performance: **XRAY 42.4**, **CT 35.4**, **ECG 46.3**, **US 39.6**, **Endoscopy 41.3**, **NST 32.6**, **PBS 48.1**, **MRI 44.4**, and **Other 48.5**. Its weakest modalities are **NST** and **CT**, and the paper notes that the model is below the overall 51-model average in every listed modality. By image count, the scores are **1 image 43.7**, **2 images 43.0**, and **3+ images 33.3**, indicating a **10.4-point** drop from single-image to 3+ image questions. This is consistent with the paper’s broader conclusion that all models degrade on multi-image questions and that integrating evidence across multiple medical images remains an open challenge [2602.13650].

A particularly informative analysis comes from the combined **KorMedMCQA-Mixed** benchmark. For **Mixed-2022**, VARCO-VISION-2.0-14B scores **Text 59.3**, **Vision 46.6**, **Total 52.7**. For **Mixed-2023**, it scores **Text 57.3**, **Vision 45.9**, **Total 51.5**. The medical benchmark paper explicitly interprets this as evidence that Korean-specialized models show large and consistent **text-over-vision gaps**, and states that **VARCO-VISION-2.0-14B exhibits +12.7 and +11.4 percentage point text advantages**, identifying **medical visual reasoning as the primary weakness of Korean-specialized VLMs**. Under KMLE-derived pass criteria, which require **\(\geq 40\%\)** per exam session and **\(\geq 60\%\)** overall, the model fails both years [2602.13650].

The medical benchmark therefore provides a narrow but important correction to the more favorable general-domain evaluation profile. VARCO-VISION-2.0-14B is competitive in several bilingual and OCR-heavy settings, but the evidence in KorMedMCQA-V suggests that Korean language specialization and general multimodal competence do not by themselves yield strong **KMLE-style multimodal medical reasoning** [2602.13650].

## 6. Limitations, reproducibility, and broader significance

The technical report is explicit about several limitations. **Instruction robustness** is weak, with outputs sensitive to superficial formatting changes such as whitespace and newlines. The model remains weaker on **knowledge-intensive tasks** and **document-centric tasks** than on perception-heavy tasks. **Referring capability** is said to be weaker than in the previous VARCO-VISION model. In OCR, the report states that while content understanding can extrapolate with larger grids, **precise text localization fails sharply beyond training grid settings**; with \(8 \times 8\) grids and longer token sequences, OCRBench improves but localization accuracy may drop to zero. The report also notes continued **Korean cultural data scarcity** and benchmark artifacts caused by **response-format violations** rather than pure understanding failure [2509.10105].

Reproducibility is only partial. The report provides the architecture family, key backbone choices, token counts by stage, context lengths, hardware, training time, and major optimization methods, but it does **not** publish a complete list of training datasets or data proportions. It also does **not** state the license in the report text provided here. The earlier VARCO-VISION paper had a similar profile: checkpoint and benchmark release were clear, but the full training-data recipe, hyperparameters, and some supervision formats were not exhaustively documented [2411.19103, 2509.10105].

In the broader landscape of open VLMs, VARCO-VISION-2.0 occupies a specific niche. Its strongest documented contributions are an **open-weight Korean-English multimodal model**, explicit **multi-image capability**, strong **spatial grounding and layout-aware OCR**, and a paired **1.7B** deployment-oriented variant. The empirical record supports the claim that it is substantially stronger than VARCO-VISION-14B across many general and Korean-localized benchmarks, especially on Korean generation and OCR. At the same time, the KorMedMCQA-V results indicate that **language specialization alone is insufficient** for high-end medical multimodal reasoning and that performance in demanding professional domains remains much lower than that of stronger proprietary and reasoning-oriented open models. This suggests that VARCO-VISION-2.0 is best understood not as a universal frontier model, but as a technically distinctive bilingual open VLM family whose main value lies in Korean-English multimodal coverage, deployability, and structured visual-text output interfaces rather than across-the-board dominance [2509.10105, 2602.13650].

Source: https://www.emergentmind.com/topics/varco-vision-2-0