K-EXAONE: Multilingual MoE Foundation Model

• K-EXAONE is a large-scale multilingual foundation model with 236B parameters and a 256K-token window, enabling efficient reasoning, code generation, and cross-lingual comprehension.
• It employs a sparse MoE architecture with hybrid attention, combining 36 sliding-window local attention layers and 12 full global attention layers to enhance processing efficiency.
• The model sets new benchmarks in safety, agentic tool use, and multilingual tasks, outperforming predecessors in Korean-specific and global evaluations.

K-EXAONE is a large-scale multilingual foundation model developed by LG AI Research, designed to achieve frontier-level performance in reasoning, code generation, agentic tool use, multilingual understanding, and safety-sensitive applications. As a direct descendant and substantial extension of the EXAONE 4.0 family, K-EXAONE establishes a new benchmark for mixture-of-experts (MoE) Transformer architectures in both the Korean and global LLM ecosystems, supporting a 256K-token context window and six languages: Korean, English, Spanish, German, Japanese, and Vietnamese (Choi et al., 5 Jan 2026).

1. Model Architecture

K-EXAONE adopts a sparse MoE Transformer comprising 48 layers: 36 layers implement sliding-window local attention (SWA), while 12 layers utilize full global attention (GA). Each MoE layer comprises 128 experts, implemented as independent FFNs. During inference, for each token, the router selects the top-8 scoring experts plus a shared expert, resulting in 9 active experts per token. Token routing is computed via a softmax over expert logits derived from the per-token hidden state: $$s = W_{\mathrm{gate}} h, \quad g = \mathrm{softmax}(s)$$ The top-8 (plus shared) indices of $g$ are selected and the output is normalized so that their weights sum to one.

The architecture includes an initial dense Transformer layer for stabilization. In total, K-EXAONE has 236B parameters, primarily in the MoE FFN modules ($\approx$235.5B), with a small dense Multi-Token Prediction (MTP) block ($\approx$0.52B). At inference, only 23B parameters are active per token—reflecting the sparsity efficiency of the MoE design. The MTP block augments next-token generation by predicting multiple future tokens, increasing decoding throughput by ~1.5× over standard autoregressive decoding.

Hybrid attention supports the 256K-token context window. SWA layers use a local window size of 128, dramatically reducing KV-cache storage (>10×) in these layers. GA layers apply full $O(n^2)$ attention to extend global awareness but limit memory/computation cost by restricting this mode to 12/48 layers.

Rotary Positional Embeddings (RoPE) are selectively applied to query/key pairs in SWA layers to convey positional information without interfering with globally attended tokens. The context window extension from 8K to 256K tokens follows a staged curriculum (8K→32K→256K), leveraging rehearsal, synthetic reasoning, and long-document datasets.

The multilingual tokenizer embodies 150K tokens, preserving 70% of the most frequent tokens from EXAONE 4.0 while adding tokens specific to new languages and technical domains. The SuperBPE strategy allocates 20% of the vocabulary to common multi-token (superword) sequences, distributed in a 2:3:1 ratio across English, Korean, and multilingual tokens, respectively. Unicode NFC normalization is applied to enhance code and notation handling, resulting in ~30% fewer tokens (bytes/token) in key domains (Choi et al., 5 Jan 2026).

2. Training Regimen and Objectives

Pretraining for K-EXAONE utilized a corpus of 11 trillion tokens, comprising high-quality web data in English, Korean, and Spanish; new language data for German, Japanese, and Vietnamese; as well as synthetic corpora including “thinking-augmented” stepwise reasoning samples. Cross-lingual transfer is promoted through the joint corpus and synthetic reasoning data.

The primary objective is causal language modeling with a token-level cross-entropy loss ($\mathcal{L}_{\rm LM}$). Two auxiliary objectives are incorporated: MoE load-balancing loss ($L_{\rm aux}$) with $\lambda_{\rm aux} = 1\times10^{-4}$ to mitigate expert collapse, and the MTP auxiliary loss ($L_{\rm MTP}$; $\lambda_{\rm MTP} = 0.05$) to enhance multi-token prediction performance: $g$0 Training is performed with quantization-aware FP8 arithmetic, using the Muon optimizer and WSD scheduler. The total computational budget is $g$1 FLOPs (Choi et al., 5 Jan 2026).

For context extension, a staged curriculum ensures preservation of short-context skills (via rehearsal data and NIAH monitoring), while long-context and synthetic reasoning data incrementally introduce longer dependencies.

3. Reasoning, Agentic, and Multilingual Capabilities

K-EXAONE’s design specifically targets agentic workflows, extended reasoning, and multilingual comprehension. The model supports a dual-mode inference paradigm inherited from EXAONE 4.0: Reasoning mode, optimized for stepwise logic, math, and code tasks, and Non-Reasoning mode, oriented toward instruction-following and high-throughput usability (Research et al., 15 Jul 2025).

The model is benchmarked across world knowledge (MMLU-Pro, GPQA-Diamond, Humanity’s Last Exam), mathematical reasoning (AIME, IMO-AnswerBench), agentic coding (LiveCodeBench, SWE-Bench), tool use ($g$2-Bench), instruction-following (IFBench), long-context understanding (AA-LCR), and both Korean-specific (KoBALT, KMMLU-Pro) and multilingual tasks (MMMLU, WMT24++).

Performance results demonstrate that K-EXAONE matches or exceeds open-weight models of similar scale on the majority of these benchmarks. For instance, on KoBALT (Korean language comprehension), K-EXAONE scores 61.8%, compared to EXAONE 4.0’s 25.4% and Qwen3-235B’s 56.1%. On LiveCodeBench v6 (coding/problem solving), K-EXAONE attains 80.7% (reasoning mode), compared with 66.7% for EXAONE 4.0 and 74.1% for Qwen3-235B. On safety benchmarks such as KGC-Safety, K-EXAONE achieves a safe-rate of 96.1% (Choi et al., 5 Jan 2026).

The model’s extended context window enables applications in long-document QA, summarization, and retrieval, covering entire legal documents, technical reports, or manuscripts without truncation.

4. Industrial Applications and Impact

The agentic features and multilingual capabilities of K-EXAONE are leveraged in real-world workflows, including software engineering agents, customer service solutions, web-search assistants, and document analysis pipelines. For example, a two-subagent system (summarizer plus trajectory compressor) enables web-search agents to manage hundreds of tool calls with maintained coherence, while the 256K-token window supports comprehensive analysis of long documents.

Sovereign AI objectives are met through strong Korean support and coverage of benchmarks such as KMMLU-Pro and KGC-Safety, facilitating adoption in national, industrial, and regulated settings where data sovereignty and cultural nuance are essential.

Inference efficiency is improved both by the MTP decoding pipeline and hybrid attention, supporting scalable deployment on high-end clusters for large models and (as inherited from EXAONE 4.0) on-device feasibility for smaller variants (Research et al., 15 Jul 2025, Choi et al., 5 Jan 2026).

5. Comparative Performance and Research Significance

K-EXAONE establishes new open-weight and proprietary baselines across reasoning, code, tool-use, and multilingual tasks. In mathematics (AIME 2025), K-EXAONE’s 92.8% matches frontier-class models. In multilingual understanding (MMMLU), it achieves 85.7%, comparable to DeepSeek-V3.2’s 88.0%. In instruction following, K-EXAONE achieves 67.3% (IFBench), surpassing EXAONE 4.0’s 36.0%. The model demonstrates robustness in both high- and low-resource languages, furthering the reach of large-scale LLM technology outside English contexts (Choi et al., 5 Jan 2026).

A plausible implication is that K-EXAONE’s approach to MoE scaling, hybrid attention, and staged context extension provides a model for future open and proprietary LLM development aiming to compete with state-of-the-art closed models.

6. Future Directions and Research Outlook

K-EXAONE’s success validates the viability of large-scale sparse MoE LLMs for broad cross-lingual, reasoning, and agentic tasks under strong efficiency, safety, and deployment constraints. Open questions remain regarding further efficiency at even larger parameter counts, robustness in low-resource domains, and the capacity for continual learning and adaptation, especially as deployment shifts increasingly to sovereign and edge environments.

The demonstrated benchmarking and engineering underpin a trend toward democratizing access to frontier LLM capability and establish K-EXAONE as a central foundation for subsequent research and industrial implementations in Korea and globally (Choi et al., 5 Jan 2026).