Korean Open-Weight Models in NLP & Multimodal AI

• Korean open-weight models are large-scale neural architectures specifically developed for Korean NLP and multimodal tasks using diverse, augmented data.
• They employ innovative training pipelines, including morpheme-aware tokenization, synthetic data augmentation, and instruction tuning to optimize performance.
• Benchmark results show competitive accuracy across Korean language and vision-language tasks, promoting reproducible and transparent research.

Korean open-weight models are large-scale neural LLMs and multimodal architectures specifically trained, released, and maintained under open licenses to support research and downstream applications involving Korean text and, increasingly, vision, speech, and cross-lingual data. These models address persisting language coverage gaps in global LLMs and are developed through dedicated pretraining on large-scale Korean corpora (including synthetic augmentation), curriculum-aligned instruction tuning, and open-weight distribution. Major open-weight efforts span language-only models, bilingual systems, and multimodal models, collectively establishing high-performance, reproducible baselines for both Korean NLP and vision-language research.

1. Core Model Architectures and Scaling

Korean open-weight models employ a spectrum of neural network architectures, from classic transformer encoders to cutting-edge sparse Mixture-of-Experts designs, adopted at both moderate (1–14B) and extreme (≥100B) parameter scales.

Model Family	Architecture/Scale	Distinctive Features
Polyglot-Ko	Decoder-only transformer, 1.3B–12.8B params	Rotary PEs, morpheme-aware BPE, GPT-NeoX codebase (Ko et al., 2023)
KLUE-BERT/RoBERTa	Encoder-only transformer, 110M–337M params	Morpheme-based BPE, MLM+NSP (BERT), dynamic masking (RoBERTa) (Park et al., 2021)
Mi:dm 2.0	Decoder-only transformer, Base (11.5B), Mini (2.3B)	Grouped Query Attention, depth-up scaling, Korean-opt. tokenizer (Shin et al., 14 Jan 2026)
KIT-19	Polyglot-Ko backbone, 1.3B/5.8B + instruction tuning	Balanced 100K instruction data w/200 prompt templates (Jang et al., 2024)
Thunder-LLM	LLaMA 3.1-8B, extended Unigram tokenizer	72K Korean tokens, continual pretrain, FP8 training (Kim et al., 18 Jun 2025)
Llama-3-Motif	Llama 3, 102B dense transformer	LlamaPro (depth), MSG (width), 96L × 9,216d, 128K vocab (Lim et al., 4 Sep 2025)
Solar Open	MoE Transformer, 102.6B params (12B active/token)	48L, 129 experts/layer, custom BPE (196K), RL via SnapPO (Park et al., 11 Jan 2026)
KORMo	Transformer-decoder, 10.8B params, GQA+MQA	68.7% synthetic Korean data, bfloat16, 40L, 4,096d (Kim et al., 10 Oct 2025)

Innovations include morpheme-aware tokenization, depth-up scaling (layer duplication), width pruning/distillation for resource-constrained variants, RoPE for long-context handling, and MoE routing for efficient conditional computation in large models.

2. Training Data Composition and Preprocessing

A central property is the use of dedicated Korean corpora and augmented Korean-English datasets, often coupled with elaborate filtering and synthetic data pipelines to overcome data scarcity and boost diversity.

• Data volume and sources: Top-tier models pretrain on up to 3.4T tokens (KORMo (Kim et al., 10 Oct 2025)), with Solar Open reaching 19.7T tokens via heavy synthetic augmentation (4.5T synthetic Korean tokens) (Park et al., 11 Jan 2026). Sources include web crawls (CommonCrawl, Naver, Daum, Tistory, news), AIHub, NIKL, patents, technical and academic corpora, and domain-specialist synthetic generations.
• Synthetic augmentation: Synthetic Korean is produced by prompting frontier LLMs (Qwen-3-235B, GPT-OSS-120B, DeepSeek) on diverse seeds (web, encyclopedic, reasoning, code, math), with style and domain balancing. Models like KORMo rely on up to ~69% synthetic Korean, demonstrating that high fractions of synthetic data do not induce training collapse when rigorously filtered (via quality classifiers, deduplication, and entropy/coverage metrics) (Kim et al., 10 Oct 2025).
• Preprocessing and balancing: Datasets undergo multi-stage deduplication, language/length/token-ratio heuristics, statistical and model-based filtering (KenLM, FastText), PII redaction, and explicit per-task balancing (as in KIT-19: max 5K per source to avoid skew) (Jang et al., 2024).

This corpus engineering yields diverse, high-quality, and scalable Korean data streams essential for robust language and multimodal modeling.

3. Model Training, Tuning, and Optimization

The predominant training paradigm is large-scale autoregressive pretraining, followed by focused instruction tuning and, in advanced models, reinforcement learning from human or synthetic feedback.

• Autoregressive pretraining: All major models minimize negative log-likelihood over next-token predictions:

$$\mathcal{L}_\text{CE} = -\sum_{t=1}^N \log p(x_t \mid x_{<t}; \theta)$$

Exemplar model configurations: Polyglot-Ko (up to 320K steps, 1,024–5,885M params, 1.2TB raw text) (Ko et al., 2023), Mi:dm 2.0 (up to 32,768 context, curriculum-mixed Korean share ramped over steps) (Shin et al., 14 Jan 2026).

• Instruction tuning: Models such as KIT-19 and Mi:dm 2.0’s variants perform SFT using balanced, template-rich instruction corpora (e.g., KIT-19: 100K examples, 200 prompt templates covering 10 task types, all human-generated in Korean) (Jang et al., 2024); Thunder-LLM leverages supervised and preference-based tuning (SFT+DPO) for bilingual alignment (Kim et al., 18 Jun 2025).
• Reinforcement learning for preference/safety: Solar Open employs SnapPO, a multi-phase policy optimization framework, with off-policy gradients using sampled response caches and RL signals for both STEM correctness and preference/safety alignment (Park et al., 11 Jan 2026). KORMo and Mi:dm 2.0 add cultural alignment losses to reinforce outputs reflecting Korean values and reasoning (Shin et al., 14 Jan 2026, Kim et al., 10 Oct 2025).
• Multimodal/fusion curriculum: VARCO-VISION and HyperCLOVA X-OMNI structure training in four/five stages: vision-feature alignment, basic/advanced cross-modal SFT, and DPO for alignment and safety. Multimodal connectors are first frozen, later unfrozen for joint vision-language optimization (VARCO-VISION: vision encoder + 2-layer MLP + LLM backbone; HyperCLOVA X-OMNI: unified next-token prediction for text, audio, vision, both discrete and continuous embeddings) (Ju et al., 2024, Team, 5 Jan 2026).

4. Performance Evaluation and Benchmarks

Korean open-weight models are benchmarked against custom Korean NLU, instruction-following, generation, reasoning, and multimodal tasks, as well as cross-lingual and English tasks for bilingual models. Prominent suites include KoBEST, KMMLU, KLUE, K-MMBench, KoMT-Bench, HAE-RAE, and custom internal tests.

Selected Benchmark Summaries

Model / Task	KMMLU (%)	KorMedMCQA (%)	K-MMBench (%)	HAERAE (%)	Comments
Llama-3-Motif	64.74	77.19 / 83.34	–	–	Parity with GPT-4 on medical; superior general Korean QA (Lim et al., 4 Sep 2025)
Mi:dm 2.0	47.7	–	–	78.2	Outperforms Qwen3-14B, Exaone-7.8B on humanities/social tasks (Shin et al., 14 Jan 2026)
KORMo	58.2	–	–	–	Comparable to Qwen3-8B, Llama3.1-8B (Korean accuracy); SFT KO-MT-Bench 8.54/10 (Kim et al., 10 Oct 2025)
Solar Open	73.0	84.4	–	73.3	Leads open models across finance, law, medical, and reasoning (Park et al., 11 Jan 2026)
VARCO-VISION-2.0	–	–	80.7 (avg)	–	Highest among open VLMs (multi-image, OCR, spatial grounding) (Cha et al., 12 Sep 2025)
HyperCLOVA X-OMNI	64.9	–	80.2	75.3	First omni-modal Korean model: text, vision, and speech (Team, 5 Jan 2026)
KIT-19-5.8B	91.6 (KoBEST-COPA)	–	–	–	+~15 pts over Polyglot-Ko, KoAlpaca in several zero-shot settings (Jang et al., 2024)

A key trend is the consistent outperformance of open-weight, Korean-focused models (notably those with targeted data curation, such as Llama-3-Motif, Mi:dm 2.0, Solar Open) over multilingual baselines and prior domestic models. Multimodal competence is established through strong MCQA, OCR, and spatial grounding—e.g., VARCO-VISION-2.0 leads K-MMBench and K-DTCBench for Korean vision-language understanding (Cha et al., 12 Sep 2025).

5. Model Release, Licensing, and Deployment

Open-weight Korean models are distributed under highly permissive licenses (Apache 2.0, MIT, or project-specific frameworks) and are accessible with associated code, recipes, and, where possible, preprocessed data.

• Access and licensing: Checkpoints for Polyglot-Ko, KIT-19, Mi:dm 2.0, Thunder-LLM, VARCO-VISION, VARCO-VISION-2.0, KORMo, Solar Open, and HyperCLOVA X-OMNI are hosted on Hugging Face, with extensive training scripts and logs. Notably, full datasets (excluding copyright-restricted raw web data) are released by KORMo, KIT-19, and Solar Open for maximal transparency (Jang et al., 2024, Kim et al., 10 Oct 2025, Park et al., 11 Jan 2026).
• Inference and fine-tuning recipes: Loading procedures employ Hugging Face's Transformers; SFT and DPO recipes are provided, including context on hyperparameter sweeps, adapter-based tuning (LoRA), and requirements for resource-constrained deployment (VARCO-VISION-2.0-1.7B for on-device use) (Cha et al., 12 Sep 2025).
• Practical guidelines: Researchers are encouraged to maintain per-task or per-domain balance in fine-tuning; exploit template diversity to minimize overfitting (KIT-19: 200 prompt templates); combine open-weight Korean LLMs with retrieval-augmented generation; and apply SFT/DPO for task-specific adaptation (Jang et al., 2024, Kim et al., 18 Jun 2025).

6. Challenges, Open Issues, and Future Directions

Despite major advances, open-weight Korean models face ongoing challenges:

• Dialects and colloquialisms: Current data and architectures underrepresent regional varieties and spoken dialects, leading to lower performance in colloquial Korean (Llama-3-Motif) (Lim et al., 4 Sep 2025, Team, 5 Jan 2026).
• Specialist domains: Performance degrades on high-difficulty domains (legal, finance, medicine), especially when synthetic augmentation is not aligned with real-world distributions (KORMo (Kim et al., 10 Oct 2025), Mi:dm 2.0 (Shin et al., 14 Jan 2026)).
• Long-context retention: Degradation beyond ~13K tokens context in Korean, with 71% → 60% accuracy drop at 32K for some tasks (KORMo (Kim et al., 10 Oct 2025)).
• Synthetic data risks: While large synthetic components do not induce pretraining collapse, monitoring for style or knowledge artifacts remains crucial (as in Solar Open, KORMo).
• Generalization and alignment: RL-based and DPO-based safety/cultural alignment require further scale and benchmarking, especially for maintaining reasoning alongside stylistic/ethical controls (Solar Open (Park et al., 11 Jan 2026), Mi:dm 2.0 (Shin et al., 14 Jan 2026)).

Future research includes adaptive tokenizers, multimodal scaling strategies (LlamaPro/MSG to vision/speech), more transparent cultural and ethical audit tools, retrieval integration, and continual model scaling to 100B+ parameters. Extending these approaches to additional under-served languages is an explicit goal (Solar Open, KORMo).

7. Significance and Research Implications

Korean open-weight models constitute foundational infrastructure for Korean NLP and bilingual/multimodal AI, catalyzing academic research, applied development, and education. Their reproducible open releases, curriculum-driven training pipelines, and strong empirical results set new standards for non-English language modeling and serve as exemplars for comparable efforts in other data-sparse languages (Jang et al., 2024, Park et al., 11 Jan 2026, Shin et al., 14 Jan 2026, Lim et al., 4 Sep 2025).