---
title: Korean Open-Weight Models in NLP & Multimodal AI
url: https://www.emergentmind.com/topics/korean-open-weight-models
type: topic
---

# Korean Open-Weight Models in NLP & Multimodal AI

Korean open-weight models are large-scale neural language models and multimodal architectures specifically trained, released, and maintained under open licenses to support research and downstream applications involving Korean text and, increasingly, vision, speech, and cross-lingual data. These models address persisting language coverage gaps in global LLMs and are developed through dedicated pretraining on large-scale Korean corpora (including synthetic augmentation), curriculum-aligned instruction tuning, and open-weight distribution. Major open-weight efforts span language-only models, bilingual systems, and multimodal models, collectively establishing high-performance, reproducible baselines for both Korean NLP and vision-language research.

## 1. Core Model Architectures and Scaling

Korean open-weight models employ a spectrum of neural network architectures, from classic transformer encoders to cutting-edge sparse Mixture-of-Experts designs, adopted at both moderate (1–14B) and extreme (≥100B) parameter scales.

| Model Family        | Architecture/Scale                                       | Distinctive Features                                               |
|---------------------|----------------------------------------------------------|--------------------------------------------------------------------|
| Polyglot-Ko         | Decoder-only transformer, 1.3B–12.8B params              | Rotary PEs, morpheme-aware BPE, GPT-NeoX codebase [2306.02254]     |
| KLUE-BERT/RoBERTa   | Encoder-only transformer, 110M–337M params               | Morpheme-based BPE, MLM+NSP (BERT), dynamic masking (RoBERTa) [2105.09680] |
| Mi:dm 2.0           | Decoder-only transformer, Base (11.5B), Mini (2.3B)      | Grouped Query Attention, depth-up scaling, Korean-opt. tokenizer [2601.09066]    |
| KIT-19              | Polyglot-Ko backbone, 1.3B/5.8B + instruction tuning     | Balanced 100K instruction data w/200 prompt templates [2403.16444]     |
| Thunder-LLM         | LLaMA 3.1-8B, extended Unigram tokenizer                 | 72K Korean tokens, continual pretrain, FP8 training [2506.21595]      |
| Llama-3-Motif       | Llama 3, 102B dense transformer                          | LlamaPro (depth), MSG (width), 96L × 9,216d, 128K vocab [2509.03972]  |
| Solar Open          | MoE Transformer, 102.6B params (12B active/token)        | 48L, 129 experts/layer, custom BPE (196K), RL via SnapPO [2601.07022] |
| KORMo              | Transformer-decoder, 10.8B params, GQA+MQA               | 68.7% synthetic Korean data, bfloat16, 40L, 4,096d [2510.09426]         |

Innovations include morpheme-aware tokenization, depth-up scaling (layer duplication), width pruning/distillation for resource-constrained variants, RoPE for long-context handling, and MoE routing for efficient conditional computation in large models.

## 2. Training Data Composition and Preprocessing

A central property is the use of dedicated Korean corpora and augmented Korean-English datasets, often coupled with elaborate filtering and synthetic data pipelines to overcome data scarcity and boost diversity.

- **Data volume and sources:** Top-tier models pretrain on up to 3.4T tokens (KORMo [2510.09426]), with Solar Open reaching 19.7T tokens via heavy synthetic augmentation (4.5T synthetic Korean tokens) [2601.07022]. Sources include web crawls (CommonCrawl, Naver, Daum, Tistory, news), AIHub, NIKL, patents, technical and academic corpora, and domain-specialist synthetic generations.
- **Synthetic augmentation:** Synthetic Korean is produced by prompting frontier LLMs (Qwen-3-235B, GPT-OSS-120B, DeepSeek) on diverse seeds (web, encyclopedic, reasoning, code, math), with style and domain balancing. Models like KORMo rely on up to ~69% synthetic Korean, demonstrating that high fractions of synthetic data do not induce training collapse when rigorously filtered (via quality classifiers, deduplication, and entropy/coverage metrics) [2510.09426].
- **Preprocessing and balancing:** Datasets undergo multi-stage deduplication, language/length/token-ratio heuristics, statistical and model-based filtering (KenLM, FastText), PII redaction, and explicit per-task balancing (as in KIT-19: max 5K per source to avoid skew) [2403.16444].

This corpus engineering yields diverse, high-quality, and scalable Korean data streams essential for robust language and multimodal modeling.

## 3. Model Training, Tuning, and Optimization

The predominant training paradigm is large-scale autoregressive pretraining, followed by focused instruction tuning and, in advanced models, reinforcement learning from human or synthetic feedback.

- **Autoregressive pretraining:** All major models minimize negative log-likelihood over next-token predictions:
  $$
  \mathcal{L}_\text{CE} = -\sum_{t=1}^N \log p(x_t \mid x_{<t}; \theta)
  $$
  Exemplar model configurations: Polyglot-Ko (up to 320K steps, 1,024–5,885M params, 1.2TB raw text) [2306.02254], Mi:dm 2.0 (up to 32,768 context, curriculum-mixed Korean share ramped over steps) [2601.09066].

- **Instruction tuning:** Models such as KIT-19 and Mi:dm 2.0’s variants perform SFT using balanced, template-rich instruction corpora (e.g., KIT-19: 100K examples, 200 prompt templates covering 10 task types, all human-generated in Korean) [2403.16444]; Thunder-LLM leverages supervised and preference-based tuning (SFT+DPO) for bilingual alignment [2506.21595].

- **Reinforcement learning for preference/safety:** Solar Open employs SnapPO, a multi-phase policy optimization framework, with off-policy gradients using sampled response caches and RL signals for both STEM correctness and preference/safety alignment [2601.07022]. KORMo and Mi:dm 2.0 add cultural alignment losses to reinforce outputs reflecting Korean values and reasoning [2601.09066], [2510.09426].

- **Multimodal/fusion curriculum:** VARCO-VISION and HyperCLOVA X-OMNI structure training in four/five stages: vision-feature alignment, basic/advanced cross-modal SFT, and DPO for alignment and safety. Multimodal connectors are first frozen, later unfrozen for joint vision-language optimization (VARCO-VISION: vision encoder + 2-layer MLP + LLM backbone; HyperCLOVA X-OMNI: unified next-token prediction for text, audio, vision, both discrete and continuous embeddings) [2411.19103], [2601.01792].

## 4. Performance Evaluation and Benchmarks

Korean open-weight models are benchmarked against custom Korean NLU, instruction-following, generation, reasoning, and multimodal tasks, as well as cross-lingual and English tasks for bilingual models. Prominent suites include KoBEST, KMMLU, KLUE, K-MMBench, KoMT-Bench, HAE-RAE, and custom internal tests.

### Selected Benchmark Summaries

| Model / Task   | KMMLU (%) | KorMedMCQA (%) | K-MMBench (%) | HAERAE (%) | Comments                                                           |
|----------------|-----------|----------------|---------------|------------|---------------------------------------------------------------------|
| Llama-3-Motif | 64.74     | 77.19 / 83.34  | –             | –          | Parity with GPT-4 on medical; superior general Korean QA [2509.03972]|
| Mi:dm 2.0     | 47.7      | –              | –             | 78.2       | Outperforms Qwen3-14B, Exaone-7.8B on humanities/social tasks [2601.09066]    |
| KORMo         | 58.2      | –              | –             | –          | Comparable to Qwen3-8B, Llama3.1-8B (Korean accuracy); SFT KO-MT-Bench 8.54/10 [2510.09426] |
| Solar Open    | 73.0      | 84.4           | –             | 73.3       | Leads open models across finance, law, medical, and reasoning [2601.07022]      |
| VARCO-VISION-2.0 | –      | –              | 80.7 (avg)    | –          | Highest among open VLMs (multi-image, OCR, spatial grounding) [2509.10105]    |
| HyperCLOVA X-OMNI | 64.9  | –              | 80.2          | 75.3       | First omni-modal Korean model: text, vision, and speech [2601.01792]      |
| KIT-19-5.8B   | 91.6 (KoBEST-COPA) | –   | –             | –          | +~15 pts over Polyglot-Ko, KoAlpaca in several zero-shot settings [2403.16444]|

A key trend is the consistent outperformance of open-weight, Korean-focused models (notably those with targeted data curation, such as Llama-3-Motif, Mi:dm 2.0, Solar Open) over multilingual baselines and prior domestic models. Multimodal competence is established through strong MCQA, OCR, and spatial grounding—e.g., VARCO-VISION-2.0 leads K-MMBench and K-DTCBench for Korean vision-language understanding [2509.10105].

## 5. Model Release, Licensing, and Deployment

Open-weight Korean models are distributed under highly permissive licenses (Apache 2.0, MIT, or project-specific frameworks) and are accessible with associated code, recipes, and, where possible, preprocessed data.

- **Access and licensing:** Checkpoints for Polyglot-Ko, KIT-19, Mi:dm 2.0, Thunder-LLM, VARCO-VISION, VARCO-VISION-2.0, KORMo, Solar Open, and HyperCLOVA X-OMNI are hosted on Hugging Face, with extensive training scripts and logs. Notably, full datasets (excluding copyright-restricted raw web data) are released by KORMo, KIT-19, and Solar Open for maximal transparency [2403.16444], [2510.09426], [2601.07022].
- **Inference and fine-tuning recipes:** Loading procedures employ Hugging Face's Transformers; SFT and DPO recipes are provided, including context on hyperparameter sweeps, adapter-based tuning (LoRA), and requirements for resource-constrained deployment (VARCO-VISION-2.0-1.7B for on-device use) [2509.10105].
- **Practical guidelines:** Researchers are encouraged to maintain per-task or per-domain balance in fine-tuning; exploit template diversity to minimize overfitting (KIT-19: 200 prompt templates); combine open-weight Korean LLMs with retrieval-augmented generation; and apply SFT/DPO for task-specific adaptation [2403.16444], [2506.21595].

## 6. Challenges, Open Issues, and Future Directions

Despite major advances, open-weight Korean models face ongoing challenges:

- **Dialects and colloquialisms:** Current data and architectures underrepresent regional varieties and spoken dialects, leading to lower performance in colloquial Korean (Llama-3-Motif) [2509.03972], [2601.01792].
- **Specialist domains:** Performance degrades on high-difficulty domains (legal, finance, medicine), especially when synthetic augmentation is not aligned with real-world distributions (KORMo [2510.09426], Mi:dm 2.0 [2601.09066]).
- **Long-context retention:** Degradation beyond ~13K tokens context in Korean, with 71% → 60% accuracy drop at 32K for some tasks (KORMo [2510.09426]).
- **Synthetic data risks:** While large synthetic components do not induce pretraining collapse, monitoring for style or knowledge artifacts remains crucial (as in Solar Open, KORMo).
- **Generalization and alignment:** RL-based and DPO-based safety/cultural alignment require further scale and benchmarking, especially for maintaining reasoning alongside stylistic/ethical controls (Solar Open [2601.07022], Mi:dm 2.0 [2601.09066]).

Future research includes adaptive tokenizers, multimodal scaling strategies (LlamaPro/MSG to vision/speech), more transparent cultural and ethical audit tools, retrieval integration, and continual model scaling to 100B+ parameters. Extending these approaches to additional under-served languages is an explicit goal (Solar Open, KORMo).

## 7. Significance and Research Implications

Korean open-weight models constitute foundational infrastructure for Korean NLP and bilingual/multimodal AI, catalyzing academic research, applied development, and education. Their reproducible open releases, curriculum-driven training pipelines, and strong empirical results set new standards for non-English language modeling and serve as exemplars for comparable efforts in other data-sparse languages [2403.16444], [2601.07022], [2601.09066], [2509.03972].

Source: https://www.emergentmind.com/topics/korean-open-weight-models