LLM-jp-4 Technical Report

Abstract: This paper presents LLM-jp-4, a family of Japanese sovereign LLMs developed with an emphasis on transparency, reproducibility, and practical usability. LLM-jp-4 comprises an 8B dense model and a 32B-A3B mixture-of-experts model, both trained from scratch on approximately 12 trillion tokens. To support independent verification and reproduction, we establish explicit criteria for selecting training data, external models, and software, and make the development process publicly available. Evaluation results demonstrate substantial improvements over previous LLM-jp models and competitive performance against open-weight models of similar scale. By documenting the development principles, training procedures, and empirical findings of LLM-jp-4, we aim to provide a reference for future sovereign AI development.

• The paper successfully develops a competitive Japanese language model in a transparent, legally compliant manner for better reproducibility.
• The LLM-jp-4 32B-A3B model achieved industry-leading performance of 0.605 on Japanese-optimized llm-jp-eval and outperformed gpt-oss-20B in technical and natural language tasks when compared to expected measures in conversational contexts.
• Use of open, consistent, and vigorously tested training pipelines is recommended to achieve similar results.

Scope and development objective

“LLM-jp-4 Technical Report” presents the construction of a Japanese sovereign LLM family designed around three simultaneous requirements: competitive capability, legal and licensing compliance, and end-to-end transparency (2610.19331). The released family contains an 8.6B-parameter dense Transformer and a 32.1B-parameter sparse MoE model with 3.83B active parameters per token. Both models were trained from scratch on approximately 10.5T pre-training tokens, followed by 1.26T mid-training tokens and post-training with SFT and DPO.

The report’s central claim is organizational as much as technical: a model can remain reproducible and permissively licensed without sacrificing competitive performance. This claim is operationalized rather than treated as a general principle. The authors restrict training datasets, external models used for data construction, and software dependencies to resources whose provenance, licensing, and processing procedures can be inspected or reconstructed by third parties. The final model weights are released under Apache 2.0, while training scripts, data-construction procedures, evaluation methodology, and selected SFT and DPO datasets are also made available.

The project therefore treats openness as a property of the development pipeline, not merely of the final checkpoint. This distinction is important because data filtering, synthetic-data generation, tokenizer construction, optimization schedules, and post-training objectives can materially determine model behavior. The report follows the broader fully open-model methodology exemplified by Pythia and OLMo, but applies it to a multilingual model with Japanese as a primary target (Tarmidi, 2023, Groeneveld et al., 2024).

Training pipeline and model family

LLM-jp-4 uses a three-stage curriculum: pre-training, mid-training, and post-training. Pre-training establishes broad linguistic and factual competence; mid-training introduces instruction-response distributions and extends context length; post-training converts the models into thinking conversational systems.

Figure 1: LLM-jp-4 training proceeds through two pre-training phases, three mid-training context stages, and SFT/DPO post-training.

Pre-training Phase 1 uses 6.3T tokens from LLM-jp Corpus v4. Phase 2 adds 4.2T tokens from an updated mixture with increased mathematical and coding content. Mid-training contributes approximately 0.84T tokens at 4K context, 0.21T tokens at 16K context, and 0.21T tokens at 64K context. The models consequently receive a curriculum that changes both data distribution and sequence length while preserving a common underlying architecture.

The dense model follows a LLaMA-style Transformer with 32 layers, hidden dimension 4,096, 32 attention heads, and 8 KV heads. The MoE model contains 128 routed experts, activates 8 experts per token, and uses a relatively wide and shallow configuration with hidden dimension 2,560 and expert hidden dimension 960. Both models use GQA, SwiGLU, RMSNorm, untied embeddings and output projections, RoPE with base $500{,}000$, and a maximum context length of 65,536 tokens. The MoE design is motivated primarily by GPU efficiency: the authors report that larger GEMM operations improved throughput in preliminary experiments.

The MoE training recipe uses a parameter initialization standard deviation of 0.006, FP32 router computation, router $z$-loss, and auxiliary load-balancing loss. The corresponding coefficients are $10^{-3}$ and $10^{-2}$. These choices target routing stability rather than architectural novelty. The practical design objective is compatibility with widely used ecosystems such as Megatron-LM and vLLM, reducing the engineering burden for subsequent fine-tuning and deployment.

Tokenization and multilingual representation

LLM-jp-4 introduces a SentencePiece Unigram tokenizer with 196,608 vocabulary entries. The vocabulary was trained from scratch on a balanced mixture of English and code, Japanese, Chinese, and Korean rather than being expanded sequentially from an existing tokenizer. The resulting language proportions were approximately 0.68 English and code, 0.18 Japanese, 0.09 Chinese, and 0.05 Korean.

The tokenizer preserves text more faithfully than conventional normalization-heavy pipelines. It disables NFKC normalization, retains repeated whitespace, permits whitespace-only pieces, and includes byte fallback for complete UTF-8 coverage. These decisions are particularly relevant for source code, where indentation and whitespace alignment are semantically meaningful. Digits are split into individual tokens; ablations indicated an improvement in mathematical performance from this choice.

The report also evaluates superword-style tokens that cross whitespace boundaries. Although these tokens improved raw compression, they reduced downstream performance in the authors’ experiments and were rejected. This result illustrates the report’s broader methodology: compression efficiency is treated as an intermediate metric, not as a sufficient criterion for tokenizer selection.

Data construction and licensing constraints

The corpus design gives Japanese data a substantially higher sampling weight than its raw web-scale availability would produce. The authors deduplicate the Japanese collection and upsample curated sources such as books, academic papers, government documents, mathematics, and code. In contrast, English web data are filtered according to educational quality, and low-quality FineWeb-Rescored material is excluded from the sampling mixture.

A notable contribution is the reproduction of datasets whose original construction depended on models with restrictive licenses. FineWeb-Rescored replaces Llama 3.1 70B Instruct with Qwen3-32B for document annotation. Similarly, MegaMath-Web-Pro-Max-OSS replaces the original model used in MegaMath-Web-Pro-Max with gpt-oss-120b. This approach does not reproduce the original dataset exactly: the reconstructed mathematical corpus contains 83M documents and 67.4B tokens, compared with 69M documents and 79.4B tokens in the original. The difference is explicitly reported rather than obscured.

The corpus-mixture studies use small-scale continued-pretraining experiments and fractional factorial designs. For Phase 2, adding MegaMath-Web-Pro-Max-OSS is the only modification satisfying the predefined positive-effect criteria on the Swallow evaluation, with an estimated effect coefficient of 0.0079, $p=0.025$, and Cohen’s $f=4.39$. None of the tested modifications meets the same criteria on LLM-jp-eval. The all-modification configuration nevertheless achieves the highest average score among the eight tested configurations, and the authors report no clear degradation in general-task performance.

This evidence supports adding mathematical and coding data, but the experimental design imposes an important qualification: the Resolution III design aliases main effects with two-factor interactions. The estimated effects therefore cannot be interpreted as fully isolated causal contributions. The report acknowledges this and assumes that interactions are relatively small.

Optimization and long-run training

LLM-jp-4 uses AdamW and a warmup-stable-decay schedule rather than a conventional cosine schedule. The WSD design allows the principal training trajectory to remain at a stable learning rate while separate branches undergo terminal decay to produce finalized checkpoints. This is useful for a long-running training program because the final token budget can change without invalidating the stable trajectory.

In a 400B-token proxy study, reducing the dense-model global batch size from 1,024 to 512 improved downstream performance per token, while a reduction to 256 was more sample-efficient but did not scale efficiently to the target GPU allocation. The production dense Phase 1 therefore uses a batch size of 512. Other tested modifications—including changing AdamW $\beta_2$, using reduced-precision attention softmax, resetting position identifiers at document boundaries, and changing the RoPE base—did not produce robust improvements.

The training curves show steady gains during pre-training, followed by sharp improvements when terminal learning-rate decay begins. The effect is particularly visible on reasoning-oriented tasks such as BBH chain-of-thought and GSM8K. This result is practically consequential: the final checkpoint is not simply an arbitrary intermediate state on a stable trajectory, and terminal decay materially affects benchmark performance.

Mid-training and context extension

Mid-training combines the pre-training corpus with a newly selected instruction-response corpus containing general dialogue, STEM, mathematics, coding, and reasoning data. Candidate-source selection uses a Resolution IV fractional factorial design, followed by evaluation both before and after SFT.

The analysis indicates that STEM data from Nemotron Post-Training Dataset v1 improve multiple benchmarks before SFT. Code-related data generally improve JHumanEval and HumanEval after SFT, although their effects on general-purpose benchmarks vary. Synthetic Code Review has a particularly large positive effect on HumanEval, while Synthetic Student-Teacher improves MT-Bench. Synthetic Transpilation produces negative effects on several coding and conversational metrics and is excluded from the final mixture.

The final mid-training schedule maintains a constant number of tokens per optimizer update while increasing sequence length from 4K to 16K and then 64K. The corresponding global batch sizes decrease from 2,048 to 512 and then 128 sequences. This preserves approximately 8.39M tokens per update and allows context extension without changing the basic optimization scale.

Figure 2: Evaluation trajectories across pre-training and mid-training show broad gains but also task-specific regressions during distributional adaptation.

Mid-training improves NLI, BBH chain-of-thought, GSM8K, and SQuAD2 early in training, with later gains on MR and MMLU. However, GSM8K, XWinograd-en, and OpenBookQA temporarily decline. The temporary GSM8K regression is consistent with specialization to the mid-training distribution, but the report does not establish whether the effect reflects representation interference, calibration changes, or altered decoding behavior. Recovery during post-training indicates that the degradation is not necessarily permanent.

Post-training as a thinking-model alignment procedure

The post-training pipeline uses OpenAI’s Harmony-style response format, including distinct analysis and final channels. The models are trained with SFT followed by DPO; reinforcement learning is not used.

SFT data combine English and Japanese sources covering dialogue, instruction following, mathematics, coding, reasoning, knowledge extraction, and safety. The authors generate reasoning traces and final answers using gpt-oss-120b at low, medium, and high reasoning-effort settings. Samples are filtered for format compliance, self-referential content, and quality. The final filter retains only examples rated 5 out of 5 on accuracy, fluency, detail, relevance, and overall quality by an LLM-as-a-Judge procedure.

The resulting SFT mixture contains 82,615 fixed-length 64K-token chunks, approximately 5.41B training tokens per epoch, and is trained for two epochs. The Japanese LLMC Math Dataset is a prominent component: it begins with 2,243 manually constructed problems and is expanded to approximately 150,000 filtered examples using synthetic generation and mathematical correctness checks.

DPO data are generated from multiple responses sampled from each SFT model. gpt-oss-120b scores the candidates for helpfulness or safety, and pairs are retained when the score gap is sufficiently large. The final datasets contain 154,561 preference pairs for the 8B model and 144,131 for the 32B-A3B model. The DPO objective uses the SFT checkpoint as reference and $\beta=0.5$.

This procedure produces models whose benchmark behavior depends not only on the pretrained representations but also on explicit reasoning-format supervision and judge-model selection. The use of a single open model for synthetic reasoning generation and candidate judging improves licensing consistency, but it also introduces a shared-model dependence between data generation, filtering, and preference construction.

Benchmark performance

The strongest numerical evidence concerns Japanese capability. On LLM-jp-eval v2.1.3, the Japanese average increases from 0.307 for LLM-jp-3.1-8x13b-instruct4 to 0.590 for LLM-jp-4-8B and 0.605 for LLM-jp-4-32B-A3B. The MoE model is therefore close to Qwen3-8B at 0.608 and gpt-oss-20B at 0.604, despite having only 3.83B active parameters per token.

Figure 3: Japanese LLM-jp-eval category scores show large gains over the previous LLM-jp generation and competitive performance with external open-weight models.

English performance also improves substantially, from 0.354 for the previous LLM-jp model to 0.634 for the 8B dense model and 0.650 for the 32B-A3B model. However, the English gap to the external baselines remains material: Qwen3-8B scores 0.714 and gpt-oss-20B scores 0.704.

Figure 4: English category scores improve strongly relative to LLM-jp-3.1 but remain below Qwen3-8B and gpt-oss-20B.

The most pronounced gains occur on challenging Japanese knowledge and reasoning tasks. JMMLU rises from 0.590 for LLM-jp-3.1-8x13b-instruct4 to 0.820 for LLM-jp-4-32B-A3B-thinking. Japanese GPQA Diamond increases from 0.310 to 0.540. These improvements establish that the new training pipeline is not limited to broad average-score gains; it materially improves difficult Japanese evaluation tasks.

At the same time, LLM-jp-4 does not match the strongest baselines on difficult English science and mathematics tasks. English GPQA Diamond reaches 0.520 for the 32B-A3B model, below Qwen3-8B at 0.560 and gpt-oss-20B at 0.670. AIME 2024 and AIME 2025 scores are 0.400 and 0.300, respectively, and HLE scores remain low across models.

Figure 5: Challenging-benchmark results reveal substantial Japanese gains alongside persistent English science, mathematics, and expert-knowledge gaps.

Conversational evaluation produces a more favorable comparison for Japanese use. At medium reasoning effort, LLM-jp-4-32B-A3B-thinking scores 7.82 on Japanese MT-Bench, 7.86 on English MT-Bench, 3.70 on AnswerCarefully, and 3.61 on LLM-jp-instructions. The corresponding gpt-oss-20B scores are 7.33, 7.85, 3.55, and 3.16. Thus, LLM-jp-4-32B-A3B exceeds gpt-oss-20B on Japanese conversational quality and safety-oriented evaluation while remaining comparable on English MT-Bench.

These results should be interpreted with care because the conversational scores are obtained using gpt-5.4 as judge and are averaged over only three evaluation runs. LLM-as-a-Judge results are also sensitive to prompt design, judge calibration, and model-family interactions.

Limitations and open questions

The report’s primary limitation is that several empirical conclusions rely on small proxy studies and fractional factorial designs. The Phase 2 corpus experiment uses a Resolution III design, so reported main effects may be confounded with pairwise interactions. The mid-training design provides stronger resolution but still evaluates only a limited number of configurations. Consequently, the data do not establish a unique optimal corpus mixture or isolate the independent contribution of every dataset.

The use of synthetic data generated and judged by gpt-oss-120b is another substantive dependency. Although this choice is compatible with the project’s licensing policy, it may transmit the generator’s stylistic, factual, and evaluative biases into SFT and DPO. The report does not provide a systematic human audit of the full synthetic corpus or an ablation separating manually authored from model-generated reasoning traces.

Evaluation coverage is also asymmetric. Japanese performance is competitive with the selected open-weight baselines, while English performance remains lower on several difficult benchmarks. The report evaluates safety primarily through AnswerCarefully and does not present a broad red-team analysis, multilingual safety audit, privacy evaluation, memorization study, or systematic contamination analysis. The Apache 2.0 license removes use restrictions, but the report correctly notes that licensing cannot reliably prevent misuse; the practical safety consequences of this release model remain an open empirical question.

Finally, the comparison between the dense and MoE models does not constitute a controlled architecture ablation. They differ in parameterization, active computation, initialization, routing, and optimization settings. The results demonstrate the practical value of the 32B-A3B model, but they do not quantify the causal contribution of sparsity independently of total parameter count or training dynamics.

Conclusion

LLM-jp-4 is a technically detailed attempt to combine sovereign model development with reproducibility, licensing discipline, and competitive Japanese-language performance. Its principal contributions are the public specification of the training pipeline, reconstruction of externally sourced datasets under compatible licensing conditions, systematic though limited corpus-mixture studies, and release of dense and MoE models trained on approximately 10.5T pre-training tokens.

The empirical results are strongest for Japanese capability: the 32B-A3B model reaches 0.605 on Japanese LLM-jp-eval, 0.820 on JMMLU, and outperforms gpt-oss-20B on several Japanese conversational evaluations. The same model remains behind Qwen3-8B and gpt-oss-20B on English average performance and difficult mathematical and scientific benchmarks. The report therefore supports a specific conclusion rather than a universal one: transparent, legally constrained training can produce a strong Japanese-focused open-weight model, while the extent to which the same recipe closes remaining English reasoning gaps is left unresolved (2610.19331).