---
title: 'HyperCLOVA X 32B THINK: Vision-Language Model'
url: https://www.emergentmind.com/topics/hyperclova-x-32b-think
type: topic
---

# HyperCLOVA X 32B THINK: Vision-Language Model

HyperCLOVA X 32B Think (“THINK”) is a 32-billion-parameter vision-language model developed as part of the HyperCLOVA X family, uniquely optimized for advanced reasoning within Korean linguistic and cultural contexts, with strong bilingual, multimodal, and agentic capabilities. It introduces architectural, training, and alignment techniques targeting sovereign AI innovation, competitive benchmarking, and efficient large-scale deployment across Korean and English tasks, while paving the way for compact, open-source successors [2601.03286] [2506.22403] [2404.01954].

## 1. Architectural Foundations and Scaling Principles

THINK employs a decoder-only Transformer backbone with 72 layers, each featuring a hidden size of 5,120, culminating in approximately 32 billion parameters [2601.03286]. Pre-normalization is handled via RMSNorm (sometimes Peri-LN or pre-norm in earlier variants), and SwiGLU is deployed as the activation in feed-forward modules. Architectural design is informed by Chinchilla-style “shallower but wider” scaling: compared to conventional deep stacks, width and FFN dimension are increased, layer count trimmed (e.g., 30% reduction in depth, 57% increase in FFN width), yielding 13.7% theoretical TFLOPs reduction for 8k-token sequences compared to baseline designs [2506.22403]. This trade-off minimizes quadratic self-attention compute while preserving representational power.

Grouped-query attention (8 groups) is used for key-value caching efficiency. Rotary positional embeddings (RoPE) provide long-context support, with the base frequency increasing progressively (500,000 in early stages, up to 5,000,000 during long-context adaptation), and untied input/output embeddings boost independence. Embedding bias terms are systematically removed.

The tokenizer is a custom morpheme-aware byte-pair encoder (BPE), adapted from tiktoken, featuring non-Korean merge pruning and explicit morpheme-aligned Korean merges for optimal compression across language boundaries. Token vocabulary is sized at 100,000 to maximize compression on Korean text [2601.03286] [2404.01954].

Vision inputs are processed via a Qwen2.5-VL ViT backbone, incorporating 3D convolutional patch embedding (2×14×14 kernel) and local-window attention, then projected via linear adapters to match the decoder’s 5,120-dim embedding space. Visual and textual tokens are seamlessly interleaved for joint multimodal processing; there is no distinct cross-attention or gating mechanism [2601.03286].

## 2. Pre-training Data, Objectives, and Curriculum

THINK is pre-trained on approximately 6 trillion tokens from high-quality Korean and English sources, including targeted synthetic Korean corpora covering education, law, and history, curated using a dual-filter (heuristic plus model-based) pipeline [2506.22403]. Earlier base models use roughly 1.5 trillion tokens equally split among Korean, English, and code [2404.01954]. Quality filtering strategies aggressively remove duplicates, PII, hate speech, low-quality advertising, and short documents, while upsampling knowledge-rich sources.

Pre-training follows a three-stage curriculum:

- **Foundational Knowledge**: Balanced Korean–English, <8k token sequences; linear warm-up/cosine decay learning rate.
- **Domain Specialization**: Addition of 1T high-fidelity Korean material (e.g., medical, economics, synthetic QA); learning rate steps and domain-specific upsampling.
- **Long-Context Adaptation**: Progressive expansion of context windows 8k→32k→64k→128k; RoPE base scaling to θ=100M; length-proportionate resampling ensures dataset stratification; fine-tuning on chain-of-thought samples up to 128k tokens.

The primary objective is standard autoregressive language modeling:
$$
L_{LM} = -\sum_{t} \log p(x_t|x_{<t})
$$
with an additional “fill-in-the-middle” infilling task applied to 10% of tokens to improve code and long-context robustness. No vision–language contrastive loss is deployed; multimodal fusion is accomplished by embedding interleaving with shared positional encoding [2601.03286] [2506.22403].

## 3. Post-training: Supervised and Reinforcement Alignment

Post-training is conducted in two main phases: supervised fine-tuning (SFT) and reinforcement learning (RL) [2601.03286]:

- **Text SFT (TSFT):** Data comprises a mix of instruction-following, chain-of-thought (CoT), and agentic tool-use trajectories (including execution, debugging, verification).
- **Multimodal SFT (MSFT):** A four-stage regimen encompassing cross-modal alignment, visual knowledge acquisition, task-based multimodal tuning, and advanced reasoning/video data; core text SFT data are interleaved to mitigate catastrophic forgetting.

Reinforcement learning phases include:

- **Multimodal RL with Verifiable Rewards (MRLVR):** Tasks supply pass/fail signals as rewards, optimized via Group Relative Policy Optimization (GRPO), employing dynamic sampling and “clip-higher” importance ratio relaxation.
- **Agent-RL:** Two sub-phases (general/Software Engineering agents; up to 128k token windows). Rewards reflect task success, format adherence, and language consistency; KL regularization is omitted from GRPO.
- **Multimodal RLHF (MRLHF):** A reward model is trained on either human or LLM-judge rankings; the final policy is optimized with PPO, including auxiliary format/language consistency penalties.

Generic RL objective:
$$
\max_\theta \mathbb{E}_{\tau\sim\pi_\theta}[R(\tau)] - \beta \text{KL}(\pi_\theta \| \pi_\text{ref})
$$
with β omitted in GRPO [2601.03286].

Rewards are decomposed into format, language, verifiability, and length-penalty components. Controllable reasoning-length is introduced using L1-Exact and L1-Max penalties, enforcing response length constraints [2506.22403].

## 4. Multimodal Processing and Agentic Abilities

THINK’s joint vision-language processing is realized by integrating ViT-extracted dense patch embeddings via linear projection, interleaving these with text tokens in the Transformer stream. There are no special cross-modal adapters or gating; both modalities rely on the core decoder [2601.03286].

The pre-training data pipeline includes Korean-script OCR, web-crawled multimodal corpora, and manually-assembled multimodal instruction datasets, explicitly targeting visual question answering (QA), image captioning, and video understanding.

Agentic behaviors are instilled and benchmarked via supervised tool-using trajectories and RL, covering shell command execution, error-driven self-correction, and composite reasoning chains. Caching and “thinking-mode fusion” strategies are employed in RL to enable context-adaptive, multi-step planning [2601.03286].

## 5. Empirical Performance and Benchmark Results

THINK establishes state-of-the-art or near-SOTA performance across Korean text, vision-language, and agent tasks for its size. Key evaluations are summarized below (selected from [2601.03286], [2506.22403]):

| Task Type            | Benchmark         | THINK 32B | EXAONE 32B | Qwen3-VL 32B | InternVL3.5 |
|----------------------|------------------|-----------|------------|--------------|-------------|
| Korean Text-to-Text  | KMMLU            | 71.3      | 75.2       | 47.4         | 61.2        |
|                      | KoBALT           | 50.6      | 48.3       | 21.1         | 31.9        |
|                      | CLIcK            | 75.2      | 73.1       | 62.4         | 62.7        |
|                      | HAERAE-1.0       | 87.4      | 64.3       | 51.5         | 43.0        |
| English Text-to-Text | MMLU             | 87.7      | 89.9       | 71.5         | 76.7        |
|                      | HellaSwag        | 57.2      | 65.7       | 62.1         | 59.0        |
| Korean Vision-to-Text| KoNET            | 75.1      | -          | -            | -           |
|                      | K-MMBench        | 88.1      | -          | -            | -           |
| Agent Tasks          | Tau²-Airline     | 58.0      | 45.6       | 54.4         | -           |
|                      | TerminalBench    | 21.8      | 10         | 11.3         | -           |
|                      | TerminalBench Hard| 9.9       | 3.6        | 6.9          | -           |

A vision-augmented THINK variant achieves 46.4% on the KCSAT STEM (Korean SAT) vision-language benchmark, exceeding GPT-4.1 (40.3%) and approaching GPT-o1 (50.9%). Disabling explicit “think-mode” reasoning halves this performance, attesting to the model’s transfer of chain-of-thought skills to visual domains [2506.22403]. In global cross-lingual consistency (Global-MMLU-Lite), the model yields symmetric agreement rates of 74.5%, with error rates nearly matching Qwen3 32B. On machine translation, Flores xCOMET-XL scores reach 90.3 (Ko→En) and 85.8 (En→Ko) [2506.22403].

Compute efficiency is a central outcome: THINK achieves 30–40% lower training cost than similarly sized baselines, by leveraging width-centric design and μP zero-shot hyperparameter porting, minimizing resource-intensive grid searches [2506.22403].

## 6. Qualitative Reasoning, Agentic Workflows, and Limitations

In vision-language QA, THINK correctly identifies geological features, differentiates samples via stepwise CoT reasoning embedded in <think> ... </think> format, and explains both high-level and low-level visual distinctions [2601.03286]. For agentic workflows, THINK is capable of issuing multi-step shell command sequences, observing errors, self-correcting, and reaching specified goals with explicit error feedback and service caching.

Principal strengths include outstanding Korean language/cultural reasoning, best-in-class Korean-centric multimodal performance, and advanced agentic ability for the 32B scale. Documented limitations involve degraded English-centric benchmark performance due to training-focus trade-offs, and catastrophic forgetting of text-only reasoning during sequential multimodal SFT [2601.03286].

## 7. Open-Source Roadmap and Future Directions

A phased open-source and business-friendly deployment is planned via progressive magnitude pruning and multi-stage knowledge distillation. The initial “SEED 0.5B” distilled variant (pruned to 50% sparsity) demonstrates 39× reduced training cost relative to open model comparators and high Korean accuracy; analogous pipelines are intended for THINK 32B, aiming for sub-10B deployment with ≥95% retention of reasoning performance [2506.22403].

Future research is directed toward joint omnimodal (text, vision, audio) pre-training to avoid catastrophic forgetting, explicit contrastive losses for vision–language alignment, and expanded curation of high-quality Korean multimodal data. These directions address core bottlenecks in model scalability, robustness, and cross-modal generalization as identified in the technical literature [2601.03286].

---

*References: HyperCLOVA X 32B Think [2601.03286]; HyperCLOVA X THINK Technical Report [2506.22403]; HyperCLOVA X Technical Report [2404.01954].*

Source: https://www.emergentmind.com/topics/hyperclova-x-32b-think