HyperCLOVA X 32B THINK: Vision-Language Model

• HyperCLOVA X 32B THINK is a 32-billion parameter vision-language model that integrates advanced transformer technologies with Korean-centric reasoning and multimodal processing.
• It employs a decoder-only Transformer with 72 layers, a custom morpheme-aware BPE tokenizer, and width-centric scaling for efficient computation and robust performance.
• The training combines autoregressive language modeling with fill-in-the-middle tasks, bolstered by supervised and reinforcement alignment to support complex agentic workflows.

HyperCLOVA X 32B Think (“THINK”) is a 32-billion-parameter vision-LLM developed as part of the HyperCLOVA X family, uniquely optimized for advanced reasoning within Korean linguistic and cultural contexts, with strong bilingual, multimodal, and agentic capabilities. It introduces architectural, training, and alignment techniques targeting sovereign AI innovation, competitive benchmarking, and efficient large-scale deployment across Korean and English tasks, while paving the way for compact, open-source successors (Team, 3 Jan 2026, Team, 27 Jun 2025, Yoo et al., 2024).

1. Architectural Foundations and Scaling Principles

THINK employs a decoder-only Transformer backbone with 72 layers, each featuring a hidden size of 5,120, culminating in approximately 32 billion parameters (Team, 3 Jan 2026). Pre-normalization is handled via RMSNorm (sometimes Peri-LN or pre-norm in earlier variants), and SwiGLU is deployed as the activation in feed-forward modules. Architectural design is informed by Chinchilla-style “shallower but wider” scaling: compared to conventional deep stacks, width and FFN dimension are increased, layer count trimmed (e.g., 30% reduction in depth, 57% increase in FFN width), yielding 13.7% theoretical TFLOPs reduction for 8k-token sequences compared to baseline designs (Team, 27 Jun 2025). This trade-off minimizes quadratic self-attention compute while preserving representational power.

Grouped-query attention (8 groups) is used for key-value caching efficiency. Rotary positional embeddings (RoPE) provide long-context support, with the base frequency increasing progressively (500,000 in early stages, up to 5,000,000 during long-context adaptation), and untied input/output embeddings boost independence. Embedding bias terms are systematically removed.

The tokenizer is a custom morpheme-aware byte-pair encoder (BPE), adapted from tiktoken, featuring non-Korean merge pruning and explicit morpheme-aligned Korean merges for optimal compression across language boundaries. Token vocabulary is sized at 100,000 to maximize compression on Korean text (Team, 3 Jan 2026, Yoo et al., 2024).

Vision inputs are processed via a Qwen2.5-VL ViT backbone, incorporating 3D convolutional patch embedding (2×14×14 kernel) and local-window attention, then projected via linear adapters to match the decoder’s 5,120-dim embedding space. Visual and textual tokens are seamlessly interleaved for joint multimodal processing; there is no distinct cross-attention or gating mechanism (Team, 3 Jan 2026).

2. Pre-training Data, Objectives, and Curriculum

THINK is pre-trained on approximately 6 trillion tokens from high-quality Korean and English sources, including targeted synthetic Korean corpora covering education, law, and history, curated using a dual-filter (heuristic plus model-based) pipeline (Team, 27 Jun 2025). Earlier base models use roughly 1.5 trillion tokens equally split among Korean, English, and code (Yoo et al., 2024). Quality filtering strategies aggressively remove duplicates, PII, hate speech, low-quality advertising, and short documents, while upsampling knowledge-rich sources.

Pre-training follows a three-stage curriculum:

• Foundational Knowledge: Balanced Korean–English, <8k token sequences; linear warm-up/cosine decay learning rate.
• Domain Specialization: Addition of 1T high-fidelity Korean material (e.g., medical, economics, synthetic QA); learning rate steps and domain-specific upsampling.
• Long-Context Adaptation: Progressive expansion of context windows 8k→32k→64k→128k; RoPE base scaling to θ=100M; length-proportionate resampling ensures dataset stratification; fine-tuning on chain-of-thought samples up to 128k tokens.

The primary objective is standard autoregressive language modeling:

$L_{LM} = -\sum_{t} \log p(x_t|x_{<t})$

with an additional “fill-in-the-middle” infilling task applied to 10% of tokens to improve code and long-context robustness. No vision–language contrastive loss is deployed; multimodal fusion is accomplished by embedding interleaving with shared positional encoding (Team, 3 Jan 2026, Team, 27 Jun 2025).

3. Post-training: Supervised and Reinforcement Alignment

Post-training is conducted in two main phases: supervised fine-tuning (SFT) and reinforcement learning (RL) (Team, 3 Jan 2026):

• Text SFT (TSFT): Data comprises a mix of instruction-following, chain-of-thought (CoT), and agentic tool-use trajectories (including execution, debugging, verification).
• Multimodal SFT (MSFT): A four-stage regimen encompassing cross-modal alignment, visual knowledge acquisition, task-based multimodal tuning, and advanced reasoning/video data; core text SFT data are interleaved to mitigate catastrophic forgetting.

Reinforcement learning phases include:

• Multimodal RL with Verifiable Rewards (MRLVR): Tasks supply pass/fail signals as rewards, optimized via Group Relative Policy Optimization (GRPO), employing dynamic sampling and “clip-higher” importance ratio relaxation.
• Agent-RL: Two sub-phases (general/Software Engineering agents; up to 128k token windows). Rewards reflect task success, format adherence, and language consistency; KL regularization is omitted from GRPO.
• Multimodal RLHF (MRLHF): A reward model is trained on either human or LLM-judge rankings; the final policy is optimized with PPO, including auxiliary format/language consistency penalties.

Generic RL objective:

$$\max_\theta \mathbb{E}_{\tau\sim\pi_\theta}[R(\tau)] - \beta \text{KL}(\pi_\theta \| \pi_\text{ref})$$

with β omitted in GRPO (Team, 3 Jan 2026).

Rewards are decomposed into format, language, verifiability, and length-penalty components. Controllable reasoning-length is introduced using L1-Exact and L1-Max penalties, enforcing response length constraints (Team, 27 Jun 2025).

4. Multimodal Processing and Agentic Abilities

THINK’s joint vision-language processing is realized by integrating ViT-extracted dense patch embeddings via linear projection, interleaving these with text tokens in the Transformer stream. There are no special cross-modal adapters or gating; both modalities rely on the core decoder (Team, 3 Jan 2026).

The pre-training data pipeline includes Korean-script OCR, web-crawled multimodal corpora, and manually-assembled multimodal instruction datasets, explicitly targeting visual question answering (QA), image captioning, and video understanding.

Agentic behaviors are instilled and benchmarked via supervised tool-using trajectories and RL, covering shell command execution, error-driven self-correction, and composite reasoning chains. Caching and “thinking-mode fusion” strategies are employed in RL to enable context-adaptive, multi-step planning (Team, 3 Jan 2026).

5. Empirical Performance and Benchmark Results

THINK establishes state-of-the-art or near-SOTA performance across Korean text, vision-language, and agent tasks for its size. Key evaluations are summarized below (selected from (Team, 3 Jan 2026, Team, 27 Jun 2025)):

Task Type	Benchmark	THINK 32B	EXAONE 32B	Qwen3-VL 32B	InternVL3.5
Korean Text-to-Text	KMMLU	71.3	75.2	47.4	61.2
	KoBALT	50.6	48.3	21.1	31.9
	CLIcK	75.2	73.1	62.4	62.7
	HAERAE-1.0	87.4	64.3	51.5	43.0
English Text-to-Text	MMLU	87.7	89.9	71.5	76.7
	HellaSwag	57.2	65.7	62.1	59.0
Korean Vision-to-Text	KoNET	75.1	-	-	-
	K-MMBench	88.1	-	-	-
Agent Tasks	Tau²-Airline	58.0	45.6	54.4	-
	TerminalBench	21.8	10	11.3	-
	TerminalBench Hard	9.9	3.6	6.9	-

A vision-augmented THINK variant achieves 46.4% on the KCSAT STEM (Korean SAT) vision-language benchmark, exceeding GPT-4.1 (40.3%) and approaching GPT-o1 (50.9%). Disabling explicit “think-mode” reasoning halves this performance, attesting to the model’s transfer of chain-of-thought skills to visual domains (Team, 27 Jun 2025). In global cross-lingual consistency (Global-MMLU-Lite), the model yields symmetric agreement rates of 74.5%, with error rates nearly matching Qwen3 32B. On machine translation, Flores xCOMET-XL scores reach 90.3 (Ko→En) and 85.8 (En→Ko) (Team, 27 Jun 2025).

Compute efficiency is a central outcome: THINK achieves 30–40% lower training cost than similarly sized baselines, by leveraging width-centric design and μP zero-shot hyperparameter porting, minimizing resource-intensive grid searches (Team, 27 Jun 2025).

6. Qualitative Reasoning, Agentic Workflows, and Limitations

In vision-language QA, THINK correctly identifies geological features, differentiates samples via stepwise CoT reasoning embedded in > ... format, and explains both high-level and low-level visual distinctions (Team, 3 Jan 2026). For agentic workflows, THINK is capable of issuing multi-step shell command sequences, observing errors, self-correcting, and reaching specified goals with explicit error feedback and service caching.

Principal strengths include outstanding Korean language/cultural reasoning, best-in-class Korean-centric multimodal performance, and advanced agentic ability for the 32B scale. Documented limitations involve degraded English-centric benchmark performance due to training-focus trade-offs, and catastrophic forgetting of text-only reasoning during sequential multimodal SFT (Team, 3 Jan 2026).

7. Open-Source Roadmap and Future Directions

A phased open-source and business-friendly deployment is planned via progressive magnitude pruning and multi-stage knowledge distillation. The initial “SEED 0.5B” distilled variant (pruned to 50% sparsity) demonstrates 39× reduced training cost relative to open model comparators and high Korean accuracy; analogous pipelines are intended for THINK 32B, aiming for sub-10B deployment with ≥95% retention of reasoning performance (Team, 27 Jun 2025).

Future research is directed toward joint omnimodal (text, vision, audio) pre-training to avoid catastrophic forgetting, explicit contrastive losses for vision–language alignment, and expanded curation of high-quality Korean multimodal data. These directions address core bottlenecks in model scalability, robustness, and cross-modal generalization as identified in the technical literature (Team, 3 Jan 2026).

---

References: HyperCLOVA X 32B Think (Team, 3 Jan 2026); HyperCLOVA X THINK Technical Report (Team, 27 Jun 2025); HyperCLOVA X Technical Report (Yoo et al., 2024).