---
title: 'Llama-3-Motif: Korean-Enhanced 102B Transformer'
url: https://www.emergentmind.com/topics/llama-3-motif
type: topic
---

# Llama-3-Motif: Korean-Enhanced 102B Transformer

Searching arXiv for the primary paper and directly related methods.
arXiv search query: "Llama-3-Motif 102B Korean Moreh"
Llama-3-Motif is a 102-billion-parameter dense Transformer model developed by Moreh as a Korean-focused extension of Meta’s Llama 3 family. Built on the Llama 3 architecture, it is not a new architectural family but an enlarged and further trained version of Llama 3 70B, designed to improve Korean capability substantially while preserving strong English performance. The model is presented as a case study in expanding foundational language capabilities in open-source LLMs for a relatively under-served language through progressive scaling, bilingual data curation, and hyperscale training infrastructure [2509.03972].

## 1. Scope, motivation, and design objective

Llama-3-Motif is motivated by a specific gap in the open-source LLM landscape: Korean remains under-served by frontier LLMs because high-quality Korean data are scarcer than English data, and efficient ways to scale models for low-resource languages are still needed. The project therefore targets a bilingual operating point with Korean emphasis rather than a Korean-only specialization. Its stated objective is to enhance Korean capabilities while retaining strong performance in English, and the continual pretraining corpus is organized around a roughly 9:1 Korean-to-English ratio [2509.03972].

A common misconception would be to treat Llama-3-Motif as an architectural departure from Llama 3. The paper explicitly states the opposite: the core Transformer architecture is unchanged. The novelty lies in progressive scaling, continued pretraining, and post-training alignment rather than in redesigning the base model family.

The paper also frames the model as an infrastructure demonstration. Korean capability is treated not only as a data problem but also as a systems problem: training very large models is expensive and operationally complex, so the work emphasizes methods that reuse a pretrained checkpoint and reduce the need to train a 100B-scale model from scratch.

## 2. Architecture and progressive scaling path

Architecturally, Llama-3-Motif remains a standard dense Transformer. The final reported configuration for Llama-3-Motif-102B is as follows [2509.03972].

| Component | Value |
|---|---:|
| Parameters | 102B |
| Layers | 96 |
| Model dimension | 9,216 |
| FFN dimension | 30,720 |
| Attention heads | 72 |
| Key/Value heads | 8 |
| Vocabulary size | 128,000 |

The model is grown from a Llama 3 70B checkpoint rather than trained from scratch. The paper distinguishes two forms of expansion. “Depth” refers to adding layers, whereas “width” refers to enlarging hidden sizes, intermediate sizes, embeddings, and attention-related dimensions. The selected recipe combines **LlamaPro** for depth expansion with **Masked Structure Growth (MSG)** for width expansion [2509.03972].

The operational role of these methods is described at a high level. LlamaPro is presented as a progressive depth-expansion method that reuses existing weights so that the enlarged model can continue training while preserving the pretrained function as much as possible. MSG is described as a masking mechanism for width expansion in which newly added parameters are initially masked and then progressively unmasked step by step for each layer. The paper reports that MSG was used for width expansion, increasing the number of layers by 20% while preserving head dimension and resizing hidden/intermediate dimensions accordingly.

Method selection was preceded by preliminary experiments on Qwen 1.8B. The compared depth-expansion methods included initialization of a larger model from a normal distribution based on base weights, staged expansion, LlamaPro, and depth-up scaling. LlamaPro was selected from those trials. The paper does not provide a full ablation table for these comparisons, so the rationale is reported qualitatively rather than through a comprehensive numerical decomposition.

## 3. Corpus construction and bilingual data balance

The continual pretraining corpus contains approximately 194 billion tokens and is curated to maintain a roughly 9:1 Korean-to-English ratio. This mixture is central to the model’s stated objective: Korean is prioritized to improve Korean understanding and reasoning, while English is retained so the model remains strong in bilingual and English-only settings [2509.03972].

The Korean portion is largely web-crawled and includes news articles, blog posts, professional documents, patents, academic papers, and publicly available research reports. The inclusion of patents, papers, and reports is notable because it indicates that the corpus is not limited to informal or generic web text.

The paper describes a substantial text-extraction, filtering, cleaning, and deduplication pipeline. Two quantitative outcomes are reported:

| Processing outcome | Reduction |
|---|---:|
| Sample count | 83.59% |
| Total text volume | 40.13% |

This suggests that the pipeline removed many short, low-information, or duplicate samples while preserving a comparatively large token mass. That inference is consistent with the paper’s emphasis on data quality, although the paper does not formalize the retained distribution beyond the source categories and the reduction statistics.

The corpus design is thus bilingual but deliberately asymmetric. Korean receives most of the token budget, yet the English component is preserved as a control against catastrophic degradation of English capability during specialization.

## 4. Training infrastructure and post-training procedure

A substantial portion of the work concerns training infrastructure. Llama-3-Motif was trained on the **MoAI Platform**, described as Moreh’s AI infrastructure system for large-scale deep learning across thousands of GPUs. The platform’s highlighted features are automatic parallelization, GPU virtualization, and dynamic GPU allocation. The practical effect, according to the paper, is that many GPUs can be treated as a single virtual device, reducing manual distributed-systems configuration and allowing researchers to focus on model and data choices [2509.03972].

The paper states that extensive experiments were run on hundreds of AMD MI250 GPUs. These experiments included hyperparameter tuning, alignment adjustments, exploration of scaling techniques, and NEFTune-based embedding experiments. Exact cluster topology and node counts per run are not specified, but the work is explicitly situated in a hyperscale training regime.

After continual pretraining, post-training proceeds in two stages. First, supervised fine-tuning uses **NEFTune** with **NEFT-alpha = 8**. The paper characterizes NEFTune as adding noise to embeddings during instruction tuning in order to improve robustness and downstream instruction-following behavior. Second, preference optimization uses **Kahneman-Tversky Optimization (KTO)** rather than PPO or DPO. The stated rationale is operational: PPO is memory-heavy and requires policy-model training overhead, whereas DPO needs paired preference data and can be expensive to collect, especially for Korean.

KTO is described as using unpaired preference data with binary signals indicating whether an output is desirable or undesirable. The reported KTO settings are:

- **Batch size**: 128  
- **Learning rate**: 1e-6  
- **NEFT alpha**: 0  
- **KTO lambda values**: Desired 1.375, Undesired 1

The paper also reports an alignment complication: naive KTO hyperparameter tuning increased toxicity. To mitigate this, policy-model logits were cached across multiple hyperparameter-tuning rounds, improving computational efficiency and making repeated preference-alignment experiments more feasible.

## 5. Benchmark profile and reported performance

The evaluation concentrates on Korean benchmarks, specifically **KMMLU** for broad knowledge and reasoning and **KorMedMCQA** for Korean medical question answering. KMMLU contains 35,030 questions across 45 subjects drawn from Korean standardized exams such as PSAT, certification tests, and CSAT, and the paper uses 5-shot evaluation. KorMedMCQA is built from Korean medical licensing exam questions for doctors, nurses, and pharmacists; the paper reports 5-shot results on the doctor set and on the average across subdomains [2509.03972].

The reported headline results are summarized below.

| Benchmark / Model | Reported score |
|---|---:|
| KMMLU-direct (5-shot), Llama-3-Motif-102B+ | 64.74 |
| KMMLU-direct (5-shot), GPT-4o-2024-05-13 | 64.11 |
| KMMLU-direct (5-shot), Qwen2-72B-Instruct | 64.1 |
| KMMLU-direct (5-shot), Llama-3-70B-Instruct | 54.5 |
| KorMedMCQA Doctor (5-shot), Llama-3-Motif-102B | 77.19 |
| KorMedMCQA Average (5-shot), Llama-3-Motif-102B | 83.34 |
| KorMedMCQA Doctor (5-shot), GPT-4-base-0125 | 76.49 |
| KorMedMCQA Average (5-shot), GPT-4-base-0125 | 83.06 |

On KMMLU, the paper reports that Llama-3-Motif achieves **64.74**, slightly above GPT-4o’s reported **64.11** and above Qwen2-72B-Instruct’s **64.1**. Relative to Llama-3-70B-Instruct, the improvement is from **54.5** to **64.74**. On KorMedMCQA, the reported result is **77.19** on the doctor subset and **83.34** on the average, slightly above GPT-4-base-0125 at **76.49** and **83.06**, respectively.

These are the paper’s strongest empirical claims. They indicate competitive performance not only on Korean general reasoning but also on a specialized professional benchmark. At the same time, the evidence base presented in the paper is concentrated on Korean evaluations, so broader multilingual generalization is less extensively documented.

## 6. Interpretation, limitations, and relation to broader open-source LLM development

The main interpretation offered by the paper is that frontier-level Korean language capability can be achieved in open-source LLMs when scaling, data curation, alignment, and infrastructure are coordinated carefully. Llama-3-Motif embodies that claim through a specific combination: progressive model growth from Llama 3 70B, bilingual continual pretraining on a 194B-token corpus, SFT with NEFTune, KTO-based alignment, and large-scale execution on MoAI across AMD GPU clusters [2509.03972].

Several limitations are also explicit or directly implied. First, the paper provides limited technical detail on the internals of LlamaPro, MSG, and KTO beyond their high-level usage. Second, it does not include an exhaustive ablation suite isolating the individual contributions of scaling method, data mixture, SFT strategy, and alignment method. Third, the evaluation is concentrated on Korean benchmarks, so non-Korean specialization and broader multilingual behavior are less thoroughly characterized. Fourth, the reported toxicity sensitivity during KTO tuning indicates that alignment remains nontrivial even when the optimization framework is computationally attractive.

A further caution concerns comparison methodology. The paper notes that some claims are based on a mix of author-measured results and community-reported numbers, so cross-paper comparisons may not always be perfectly apples-to-apples. This does not negate the reported benchmark outcomes, but it does qualify how directly those outcomes should be interpreted as definitive rankings across model families.

In that context, Llama-3-Motif is best understood not as a novel Transformer architecture, but as a Korean-centric scaling and alignment program built on Llama 3. Its significance lies in showing that open-weight models can be extended toward strong Korean performance through checkpoint reuse, carefully curated bilingual data, and infrastructure that makes 100B-scale experimentation operationally feasible.

Source: https://www.emergentmind.com/topics/llama-3-motif