---
title: LLM-jp-4 Overview | Large Japanese Language Models
url: https://www.emergentmind.com/topics/llm-jp-4
type: topic
---

# LLM-jp-4 Overview | Large Japanese Language Models

LLM-jp-4 is the fourth-generation family of Japanese sovereign large language models developed by the cross-organizational LLM-jp project. It comprises an 8B dense Transformer, LLM-jp-4-8B, and a sparse mixture-of-experts model, LLM-jp-4-32B-A3B, with approximately 32.1B stored parameters and approximately 3.8B active parameters per token. Both models were trained from scratch on approximately 12T tokens and released under the Apache License 2.0. The family emphasizes Japanese-centered capability, multilingual training, transparency, reproducibility, legal clarity, and practical compatibility with open software ecosystems [2610.19331].

## 1. Development context and objectives

LLM-jp-4 was developed by the LLM-jp cross-organizational project, whose earlier publicly documented generations included LLM-jp model suite v1.0, released on 20 October 2023, and v2.0, released on 30 April 2024. The earlier project involved more than 1,500 participants from academia and industry and sought to develop open Japanese LLMs independently and openly [2407.03963]. That paper did not describe LLM-jp-4; the fourth-generation family is documented in a later technical report [2610.19331].

The central objective of LLM-jp-4 is the development of Japanese-centered sovereign LLMs without dependence on a closed foreign provider for the model, data-construction pipeline, or training software. The project identifies three related motivations:

1. **Sovereign capability**: reducing dependence on a small number of external providers and enabling models suited to Japanese language, culture, laws, and social priorities.
2. **Scientific transparency**: documenting data selection, preprocessing, optimization, alignment, and sources of model capability and failure.
3. **Reproducibility and legal clarity**: restricting datasets, external models, and software to resources that third parties can inspect, acquire, reconstruct, or replace with equivalent resources under compatible licenses.

The models are intended both as open Japanese foundation models and as reference implementations for transparent sovereign-AI development. The project favors architectures compatible with Megatron-LM and vLLM rather than highly specialized designs. It distinguishes legal permission from responsible use: Apache License 2.0 provides the stated licensing framework, while users remain responsible for intellectual-property, human-rights, safety, professional-use, and regulatory risks.

Both models share a broad training curriculum consisting of approximately 10.5T pre-training tokens, approximately 1.26T mid-training tokens, and post-training with supervised fine-tuning followed by Direct Preference Optimization. The total exposure is approximately 11.76T tokens, generally reported as approximately 12T tokens.

## 2. Model family and architecture

The LLM-jp-4 family contains two pretrained and post-trained models.

| Model | Architecture and parameters | Context |
|---|---|---:|
| LLM-jp-4-8B | Dense Transformer; 8.59B total and active parameters | 65,536 tokens |
| LLM-jp-4-32B-A3B | Sparse MoE; 32.14B total and 3.83B active parameters | 65,536 tokens |

Both models are decoder-only Transformers with 32 layers, RMSNorm, SwiGLU activation, rotary positional embeddings, grouped-query attention, no linear-layer biases, untied input-embedding and output-projection weights, bfloat16 computation, float32 attention softmax, and a maximum context length of 65,536 tokens after mid-training. Pre-training uses the causal language-model objective

$$
\mathcal{L}_{\mathrm{LM}} =
-\sum_{t=1}^{T}\log p_\theta(x_t\mid x_{<t}).
$$

### LLM-jp-4-8B

LLM-jp-4-8B follows a LLaMA-2-style dense Transformer configuration:

| Property | Value |
|---|---:|
| Layers | 32 |
| Hidden dimension | 4,096 |
| Query heads | 32 |
| Key/value heads | 8 |
| Attention | GQA |
| FFN dimension | 14,336 |
| Normalization | RMSNorm, $\epsilon=10^{-6}$ |
| RoPE base | $\theta=500{,}000$ |
| Parameters | 8.59B |
| Initialization standard deviation | 0.02 |

The 32 query heads share 8 key/value heads, so each key/value head is shared by four query heads. This reduces key/value-cache memory relative to ordinary multi-head attention. QK normalization is disabled.

### LLM-jp-4-32B-A3B

LLM-jp-4-32B-A3B follows a Qwen3-like general architecture and uses a wide, relatively shallow design intended to improve GPU matrix-multiplication efficiency. Its configuration is:

| Property | Value |
|---|---:|
| Layers | 32 |
| Hidden dimension | 2,560 |
| Query heads | 40 |
| Key/value heads | 4 |
| Attention | GQA |
| Routed experts | 128 |
| Experts activated per token | 8 |
| Expert FFN dimension | 960 |
| Total parameters | 32.14B |
| Active parameters | 3.83B |
| Normalization | RMSNorm, $\epsilon=10^{-6}$ |
| RoPE base | $\theta=500{,}000$ |
| Initialization standard deviation | 0.006 |
| Router precision | float32 |

The designation A3B refers to approximately 3B active parameters, not total stored parameters. Eight of 128 routed experts are selected for each token. For token representation $h$, the selected expert set is

$$
\mathcal{S}(h)=\operatorname{TopK}(r(h),K),
$$

where $K=8$ and $r(h)$ denotes router logits. The MoE output is conceptually

$$
\operatorname{MoE}(h)=
\sum_{e\in\mathcal{S}(h)}g_e(h)F_e(h),
$$

where $F_e$ is expert $e$ and $g_e(h)$ is its normalized routing weight.

The model uses auxiliary load-balancing and router $z$-loss coefficients of $\lambda_{\mathrm{LB}}=10^{-2}$ and $\lambda_z=10^{-3}$. Router logits are computed in float32. Grouped matrix multiplication and fused token permutation are used for efficient expert computation. Output-projection weights immediately before residual additions are scaled by the inverse square root of the total number of residual connections.

## 3. Tokenizer and training data

### Tokenizer

LLM-jp-4 introduces a tokenizer trained from scratch using SentencePiece Unigram. It has 196,608 tokens and emphasizes English, Japanese, Chinese, and Korean.

| Setting | Configuration |
|---|---|
| Algorithm | SentencePiece Unigram |
| Vocabulary | 196,608 tokens |
| Character coverage | 1.0 |
| Byte fallback | 256 byte tokens |
| Normalization | Identity; no NFKC |
| Whitespace | Standard pre-tokenization; repeated whitespace preserved |
| Cross-whitespace tokens | Not permitted |
| Maximum piece length | 16 |
| Unknown tokens | None for UTF-8 input because of byte fallback |

The tokenizer-training mixture consisted of approximately 68% English and code, 18% Japanese, 9% Chinese, and 5% Korean from approximately 22 GB of text. Documents were sampled uniformly within each language. The mixture was selected using a grid search optimizing held-out compression measured by bytes per token and characters per token.

The vocabulary contains 256 byte-fallback tokens, special and control tokens including `<unk>`, `<s>`, `</s>`, `<pad>`, `<mask>`, `<CLS>`, `<SEP>`, and `<EOD>`, 344 reserved slots, and the remaining learned subword units. Whitespace and indentation are preserved because source code is included in the training data. SuperBPE-style cross-whitespace tokens improved raw compression but degraded downstream performance and were rejected. Individual digit splitting improved mathematical performance in an ablation.

### Pre-training corpus

The pre-training corpus covers Japanese, English, Chinese, Korean, source code, mathematics, academic and educational material, books, government documents, patents, news, encyclopedic text, web pages, and question-answer data.

The initial Phase 1 corpus was constructed at a target size of approximately 20T tokens, although the models consumed approximately 6.3T Phase 1 tokens. Japanese sources included Aozora Bunko, Common Crawl, Ceek.jp News, e-Gov, FineWeb2, KAKEN, National Diet Minutes, NWC2010, NWJC, patents, SIP Web Corpus, WARP, and Wikipedia. English sources included Project Gutenberg, PeS2o, Reddit, Wikipedia and Wikibooks, Stack Exchange, FineWeb-Rescored, GSM8K, MathPile, AlgebraicStack, arXiv, OpenWebMath, and other mathematical material.

Reported source-corpus sizes include Japanese Common Crawl at 217.4B tokens, Japanese FineWeb2 at 216.1B, Japanese patents at 60.8B, English PeS2o at 60.0B, English Reddit at 82.3B, FineWeb-Rescored high-quality data at 2.0T, FineWeb-Rescored medium-quality data at 6.3T, MegaMath-Web-Pro-Max-OSS at 67.4B, StarCoder at 106.9B, The Stack at 117.9B, The Stack v2 at 720.3B, Chinese FineWeb2 at 566.5B, and Korean FineWeb2 at 48.0B. These are source-corpus counts rather than necessarily consumed training-token counts.

The sampling strategy upsampled Japanese and curated material relative to raw corpus proportions. Curated material such as books, Wikipedia, academic text, mathematics, and code generally received weight 8; major Japanese web data received weight 4; FineWeb-Rescored high-, medium-, and low-quality subsets received weights 4, 1, and 0; and Chinese and Korean web data received weight 1.

A baseline equal-weight mixture would have produced 91.1% English, 3.5% Japanese, and 97.3% unfiltered web data. To optimize the mixture, the authors trained 7.7B proxy models for 400B tokens and evaluated them with `llm-jp-eval`, using the macro-average across evaluation categories as the primary metric.

### Phase 2 and reconstructed datasets

Phase 2 began after approximately 6T tokens and targeted mathematics and coding while avoiding degradation in general-language performance. The principal changes were replacement of The Stack v1 with The Stack v2 and addition of MegaMath-Web-Pro-Max-OSS.

Eight configurations were evaluated using a $2^{5-2}=8$ Resolution III fractional-factorial design. Each configuration continued an 8B model for 100,000 steps, processing approximately 210B tokens from a checkpoint that had already consumed 3.8T tokens. MegaMath-Web-Pro-Max-OSS was the only factor satisfying the predefined positive-effect criterion on Swallow, with $\alpha=0.0079$, $p=0.025$, and Cohen’s $f=4.39$. No factor met the criterion on `llm-jp-eval`.

The project rejected datasets whose construction depended on proprietary or license-incompatible models. FineWeb-Rescored was reconstructed using Qwen3-32B rather than Llama 3 70B, and MegaMath-Web-Pro-Max-OSS was reconstructed using gpt-oss-120b rather than Llama 3.1 70B Instruct. The FineWeb reconstruction used Qwen3-32B to annotate a 450K-document subset, a BERT-based embedding classifier trained on those annotations, and classification of all 24.5B FineWeb documents using 128 H200 GPUs for approximately 1.5 days.

The project states that the entire Japanese corpus was deduplicated. Reported safeguards include licensing and provenance review, release-level screening, educational-quality scoring, corpus-mixture experiments, held-out benchmark splits, separate training and test splits, mechanical chat-format filtering, LLM-as-a-Judge filtering of synthetic reasoning data, removal of model-self-reference samples from general SFT data, and mathematical correctness filtering of generated LLMC Math examples. A complete token-level contamination audit, all near-duplicate thresholds, and all personally identifiable-information filtering procedures are not reported.

## 4. Training procedures

### Pre-training

Pre-training used Megatron-LM on the ABCI H200 GPU cluster with Transformer Engine attention kernels, bfloat16 computation, distributed optimization, overlapped gradient reduction and parameter gathering, tensor, expert, pipeline, and data parallelism, grouped matrix multiplication, fused token permutation, and float32 router logits.

| Setting | Dense 8B | MoE 32B-A3B |
|---|---:|---:|
| Phase 1 tokens | 6.3T | 6.3T |
| Phase 2 tokens | 4.2T | 4.2T |
| Phase 1 batch | 512 sequences | 1,024 sequences |
| Phase 2 batch | 2,048 sequences | 2,048 sequences |
| Sequence length | 4,096 | 4,096 |
| Phase 1 steps | 3,000,000 | 1,500,000 |
| Phase 2 steps | 500,000 | 500,000 |
| Peak learning rate | $3\times10^{-4}$ | $4\times10^{-4}$ |
| Optimizer | AdamW | AdamW |
| Weight decay | 0.1 | 0.1 |
| Gradient clipping | 1.0 | 1.0 |
| Precision | bfloat16 | bfloat16 |
| Attention softmax | float32 | float32 |
| Dropout | 0 | 0 |

Both models use AdamW with $\beta_1=0.9$, $\beta_2=0.95$, and Adam epsilon $10^{-8}$. Warmup lasts 2,000 steps. Phase 1 decay lasts 100,000 steps for the dense model and 50,000 steps for the MoE model; Phase 2 decay lasts 50,000 steps for both.

The project uses a warmup-stable-decay schedule with one-minus-square-root decay. After warmup, the learning rate remains constant until a checkpoint branch is created; only the branch is decayed. For normalized decay progress $u$, a representative schedule is

$$
\eta(u)=\eta_{\max}(1-\sqrt{u}).
$$

Proxy experiments found WSD superior to cosine decay at tested long-run settings, with longer decay intervals generally improving loss and downstream performance.

### Mid-training

Mid-training combines the pre-training mixture with the curated `LLM-jp-Corpus-Midtraining-v2`. Its objectives are instruction following, dialogue, reasoning, mathematics, coding, and long-context retention. The final mixture assigns sampling weight 0.5 to the pre-training corpus and 0.5 to the mid-training corpus.

| Stage | Sequence length | Steps | Global batch | Tokens |
|---|---:|---:|---:|---:|
| 1 | 4,096 | 100,000 | 2,048 | 0.84T |
| 2 | 16,384 | 25,000 | 512 | 0.21T |
| 3 | 65,536 | 25,000 | 128 | 0.21T |

Mid-training therefore processes approximately 1.26T tokens. Dense-model learning rates are $1.5\times10^{-4}$ and MoE learning rates are $2.0\times10^{-4}$. Stage 1 uses 2,000 warmup steps; Stage 2 continues without another warmup; and Stage 3 decays to zero using the WSD one-minus-square-root schedule. Context parallelism is enabled in Stages 2 and 3.

The corpus includes regenerated LLM-jp Midtraining-v1 data, FLAN, DolminoSynthMath, GSM8K, Stack Exchange, Nemotron datasets, Swallow Code, synthetic coding data, Wikipedia rewriting, and the LLMC Math Dataset. Synthetic examples were regenerated with gpt-oss-120b. A Resolution IV fractional-factorial design evaluated candidate corpus additions across 16 configurations. Seven of eight candidate additions were selected; Synthetic-Transpilation was excluded because its post-SFT effects were negative on several coding and conversational measures.

### Post-training

Post-training creates “thinking” models through supervised fine-tuning followed by DPO. The project explicitly does not use reinforcement learning in post-training.

The models use an adapted OpenAI Harmony format with assistant channels including `analysis` and `final`. The `analysis` channel represents reasoning, while `final` represents the user-visible response. The described system template identifies the model as LLM-jp-4 and supports configurable knowledge cutoff, current date, and reasoning effort; its default knowledge cutoff is 2025-12.

SFT data covers English and Japanese dialogue, instruction following, mathematics, coding, reasoning, knowledge extraction, and safety. Synthetic reasoning and answer fields were generated with gpt-oss-120b at low, medium, and high reasoning effort. Filtering removes invalid Harmony-format outputs, self-referential model questions except in specially constructed system-identity datasets, and samples not judged 5/5 on accuracy, fluency, detail, relevance, and overall quality.

The final SFT mixture contains 82,615 packed 64K-token chunks, approximately 5.414B tokens per epoch, and two epochs, totaling approximately 10.83B training tokens. AnswerCarefully is upsampled by a factor of 16, System Prompt Questions by 8, and Self-System Questions by 5. SFT uses AdamW, global batch size 64, $\beta_1=0.9$, $\beta_2=0.98$, weight decay 0.1, peak learning rate $4.0\times10^{-5}$, 5% warmup, a 20% stable phase, and one-minus-square-root decay over the remaining 75%. Loss is computed only on assistant reasoning and final-answer tokens.

For DPO, four candidates are sampled per prompt at temperature 1.0 and top-$p=0.95`. gpt-oss-120b judges the candidates; the highest-scoring response is selected as “chosen” and the lowest as “rejected.”

| Model | Preference pairs |
|---|---:|
| LLM-jp-4-8B | 154,561 |
| LLM-jp-4-32B-A3B | 144,131 |

DPO uses the SFT model as reference, $\beta=0.5$, AdamW, global batch size 256, microbatch size 1, peak learning rate $9.0\times10^{-7}$, 5% warmup, 20% stable training, one-minus-square-root decay, and two epochs. Candidate pairs are retained according to helpfulness or safety score gaps, answer-length ratio no greater than 1.5, and a maximum sequence length of 64K tokens.

## 5. Transparency, licensing, and availability

LLM-jp-4 models are released under the Apache License 2.0. The release is accompanied by training scripts and configurations, the tokenizer, SFT data where redistribution is permitted, separate DPO datasets for both models, the chat template, data-construction resources and source links, evaluation software and configurations, modified software components, and final model weights on Hugging Face.

The project classifies datasets according to whether they may be used for internal training, public training, retrieval, and redistribution. Only datasets classified L1–L3 may be used for publicly released model training. L1 resources permit internal and public training, retrieval, and redistribution; L2 resources permit training and retrieval but not redistribution; L3 resources permit training but not retrieval or redistribution. LX resources are restricted to internal training, while LZ resources are not permitted.

The external-model policy excludes proprietary models or models whose terms restrict training competing models on outputs, downstream model distribution, commercial use, model naming, or user-scale conditions. Open-source or Apache-licensed models such as Qwen3-32B and gpt-oss-120b were used for data construction. The project requires training and data-processing software to be open source, inspectable, modifiable, and redistributable, and states that its modifications are also released.

The openness framework does not make exact reproduction trivial. The corpus contains approximately 12T tokens, some source data cannot be redistributed, web snapshots may change, some datasets depend on license agreements, full-scale H200 access is expensive, external synthetic-data generation is not bitwise reproducible, and some preprocessing details and contamination audits are not specified. The paper also does not report the precise number of H200 GPUs used for full pre-training, total GPU-hours, wall-clock duration, energy consumption, or monetary cost.

## 6. Evaluation and reported results

Evaluation uses `llm-jp-eval v2.1.3`, Japanese and English subsets, MMLU, JMMLU, GPQA and GPQA Diamond, AIME 2024 and 2025, HLE, and BBH. LLM-as-a-Judge evaluation uses `llm-jp-judge` on Japanese and English MT-Bench, AnswerCarefully v2.0 with 336 questions, and `llm-jp-instructions` with 400 Japanese questions. The judge is GPT-5.4-2026-03-05, and each result is averaged over three independent evaluation runs.

### Aggregate benchmark performance

| Model | Japanese average | English average |
|---|---:|---:|
| llm-jp-3.1-8x13b-instruct4 | 0.307 | 0.354 |
| LLM-jp-4-8B | 0.590 | 0.634 |
| LLM-jp-4-32B-A3B | 0.605 | 0.650 |
| Qwen3-8B | 0.608 | 0.714 |
| gpt-oss-20B | 0.604 | 0.704 |

Relative to LLM-jp-3.1-8x13b-instruct4, LLM-jp-4-8B improves the Japanese average from 0.307 to 0.590 and the English average from 0.354 to 0.634. LLM-jp-4-32B-A3B reaches 0.605 in Japanese and 0.650 in English. The MoE model is close to Qwen3-8B and gpt-oss-20B on Japanese performance while remaining behind both external models on English aggregate performance.

### Knowledge and reasoning benchmarks

| Benchmark | LLM-jp-3.1 | LLM-jp-4-32B-A3B | Comparator |
|---|---:|---:|---|
| JMMLU | 0.590 | 0.820 | Substantial improvement |
| Japanese GPQA Diamond | 0.310 | 0.540 | Substantial improvement |
| English GPQA Diamond | Not stated | 0.520 | Qwen3-8B: 0.560; gpt-oss-20B: 0.670 |
| AIME 2024 | Not stated | 0.400 | Below Qwen3-8B and gpt-oss-20B |
| AIME 2025 | Not stated | 0.300 | Below Qwen3-8B and gpt-oss-20B |
| HLE | Not stated | Low | Low across compared models |

These results indicate strong progress on Japanese knowledge and reasoning, particularly JMMLU and Japanese GPQA Diamond. The reported results also identify remaining weaknesses in English graduate-level science, advanced mathematics, and expert-level broad knowledge. Exact 8B scores for JMMLU, GPQA, AIME, and HLE are not reported in the supplied technical report.

### Conversational and safety evaluation

| Model | MT-Bench JA | MT-Bench EN | AnswerCarefully | llm-jp-instructions |
|---|---:|---:|---:|---:|
| gpt-4o | 7.29 | 7.69 | 4.00 | 4.07 |
| GPT-5.4 medium | 8.87 | 8.89 | 4.43 | 4.82 |
| gpt-oss-20B medium | 7.33 | 7.85 | 3.55 | 3.16 |
| LLM-jp-4-8B medium | 7.54 | 7.79 | 3.69 | 3.54 |
| LLM-jp-4-32B-A3B medium | 7.82 | 7.86 | 3.70 | 3.61 |

At medium reasoning effort, LLM-jp-4-32B-A3B exceeds gpt-oss-20B on Japanese MT-Bench, AnswerCarefully, and `llm-jp-instructions`, while achieving approximately equal English MT-Bench performance. It exceeds LLM-jp-4-8B on all four reported judge metrics at medium effort. Increasing reasoning effort from low to medium improves both LLM-jp-4 models, particularly on MT-Bench and AnswerCarefully.

These scores are produced by an LLM judge rather than human raters. The report does not provide inter-rater agreement, judge calibration, confidence intervals, or systematic judge-bias analysis.

### Long-context, coding, and safety limitations

The training schedule extends sequence length from 4,096 tokens during pre-training to 65,536 tokens during mid-training, with stated goals of long-document understanding and context retention in multi-turn conversations. However, the supplied report does not provide an independent numerical long-context benchmark, retrieval-at-length curve, needle-in-a-haystack result, or degradation analysis by context length. The 65,536-token context therefore describes architectural and training support rather than a measured long-context capability.

The corpus and ablations target coding and mathematics. The Stack v2 improved selected MMLU and code-related subtasks, MegaMath-OSS produced a statistically meaningful positive Swallow effect, digit-level tokenization improved mathematical performance, and several mid-training code additions improved JHumanEval and HumanEval after SFT. Synthetic-Code-Review produced a large positive HumanEval effect, Synthetic-Student-Teacher improved MT-Bench, and Synthetic-Transpilation was excluded because it harmed several post-SFT measures. Full final HumanEval, JHumanEval, and coding scores for the two released models are not provided in the final-evaluation section.

Safety evaluation remains limited relative to the range of possible misuse, robustness, privacy, bias, and multilingual risks. Separate research indicates that language can alter safety-relevant behavior in some multilingual models: Japanese reasoning reduced launch decisions in certain high-stakes strategic vignettes, although the effect was model-specific and LLM-jp-4 was not evaluated [2608.12373]. This finding does not establish that LLM-jp-4 is safer in Japanese; it supports evaluating Japanese as a potentially distinct behavioral and safety-relevant reasoning mode.

LLM-jp-4 therefore combines substantial generation-over-generation gains, particularly on Japanese knowledge and reasoning, with documented limitations in English performance relative to leading open-weight comparators, advanced mathematics, expert-level general knowledge, independent long-context validation, exact reproducibility, and broad safety evaluation. Its broader methodological significance lies in treating the corpus, external-model policy, software stack, optimization recipe, evaluation process, and legal framework as components of the model artifact rather than undisclosed infrastructure surrounding released weights.

Source: https://www.emergentmind.com/topics/llm-jp-4