---
title: 'PLaMo 2: Japanese LLM Evolution'
url: https://www.emergentmind.com/topics/plamo-2
type: topic
---

# PLaMo 2: Japanese LLM Evolution

Searching arXiv for the PLaMo 2 technical report and closely related PLaMo papers.
PLaMo 2 is a series of Japanese-focused large language models developed by Preferred Networks as the successor to PLaMo-100B. It is characterized by a hybrid Samba-based architecture, continual pre-training that transitions to full attention for 32K-token contexts, extensive use of synthetic corpora to address Japanese data scarcity, and an efficiency-oriented pipeline based on weight reuse, structured pruning, post-training with supervised fine-tuning and direct preference optimization, and inference optimization with vLLM and quantization [2509.04897]. The term also encompasses an expanding model ecosystem built on the same base language models, including contextualized speech recognition with PLaMo-100B as a precursor and the multimodal PLaMo 2.1-VL family for visual question answering and visual grounding [2408.08027] [2604.19324].

## 1. Origin, scope, and positioning

PLaMo 2 is explicitly presented as a family of Japanese-focused large language models centered around two main base scales: **PLaMo 2.0–31B** and **PLaMo 2.0–8B / 2.1–8B**. The 31B line is positioned as a high-end open model and successor to PLaMo-100B in capability, whereas the 8B line is designed to reach or surpass PLaMo-100B performance at a fraction of the size and training compute [2509.04897].

The design goals are threefold. First, PLaMo 2 targets strong Japanese-centric capability, including Japanese web text, translation, Japanese-specific cultural and knowledge tasks, and instruction following. Second, it emphasizes efficiency and scalability through weight reuse, pruning, and inference-oriented systems engineering. Third, it supports long-context processing, with the final models configured for 32K tokens after continual pre-training and rotary-position reconfiguration [2509.04897].

This positioning marks a transition from the earlier PLaMo-100B regime to a smaller but more optimized model family. In the speech recognition work based on PLaMo-100B, the 100-billion-parameter model served as a decoder-only transformer LLM that could be adapted to automatic speech recognition through a pre-trained Whisper encoder, a linear adapter, and QLoRA fine-tuning, without changing the model architecture [2408.08027]. PLaMo 2 generalizes the underlying design philosophy: preserve a strong text LLM core, then extend it through efficient architectural modification, synthetic data, and downstream specialization.

A plausible implication is that “PLaMo 2” should be understood less as a single checkpoint than as a platform architecture: a text-first Japanese LLM family from which smaller, instruction-tuned, speech-adapted, and vision-language variants can be derived while preserving a consistent deployment and adaptation strategy.

## 2. Core architecture and long-context transition

The base PLaMo 2 models adopt a **Samba-based hybrid architecture** in which **Mamba / SSM** layers alternate with **Sliding-Window Attention (SWA)** layers. This design is intended to provide linear-time sequence processing and constant memory characteristics for the SSM component, while retaining localized self-attention via a fixed-size attention window [2509.04897].

The report identifies a central limitation of this hybrid design: long-context exact retrieval degrades sharply once the relevant token lies beyond the SWA window. On **Phonebook** and **Passkey Retrieval**, a 1B PLaMo 2 model collapses to near-zero accuracy when the relevant information is more than 2048 tokens away, which the report attributes to the lossy compression of past information into a fixed-size hidden state and the locality constraint of SWA [2509.04897].

To address this, PLaMo 2 uses **Continual Pre-training (CPT)** in which the SWA window is enlarged to the full target context length, effectively converting SWA into full attention. At the same time, the models adopt **Adjustable Base Frequency RoPE (ABF-RoPE)** with base frequency $\theta = 1{,}000{,}000$, enabling stable rotary positional encoding up to 32K tokens. CPT is performed on a subset of the original pre-training corpus, and the final checkpoint is chosen to balance long-context retrieval improvements with preservation of overall Japanese text generation quality [2509.04897].

This architectural evolution is central to the identity of PLaMo 2. The “hybrid-to-full-attention” transition is not an incidental finetuning step but a structural response to a measured failure mode. This suggests that PLaMo 2 occupies an intermediate point between sequence-model efficiency and transformer-style random-access retrieval: its pretraining pipeline exploits hybrid efficiency, while its deployed long-context capability depends on later full-attention adaptation.

## 3. Data strategy, synthetic corpora, and model compression

A defining feature of PLaMo 2 is its reliance on synthetic corpora to compensate for the relative scarcity of high-quality large-scale Japanese data. The training pipeline combines CommonCrawl-based Japanese and English web corpora, code data extracted from public sources and filtered CommonCrawl pages, and large synthetic datasets for translation, paraphrasing, code, and math [2509.04897].

For Japanese web data, the report combines an **educational value filter** in the style of FineWeb-Edu with **categorization + downsampling** in the style of DataComp-LM. For coding data, it uses public code datasets and CommonCrawl pages filtered by HTML tags such as `<pre>` and `<code>`, then classified as coding-oriented with a fastText classifier trained on LLM-labeled data. For synthetic Japanese text, it translates high-quality English corpora into Japanese, paraphrases Japanese corpora, and augments code and math through multi-stage LLM generation pipelines [2509.04897].

The 8B line, especially **PLaMo 2.1–8B**, is strongly shaped by **structured pruning** and **knowledge distillation**. The report states that PLaMo 2.1–8B is derived from the fully trained **PLaMo 2–31B** using a **Minitron**-based structured pruning procedure, followed by retraining with a KL-based distillation loss from the 31B teacher. For memory efficiency, teacher logits are truncated to the top 128 tokens, and the resulting 8B model inherits much of the teacher’s capability while requiring far less compute [2509.04897].

The compute-performance comparison reported for JMMLU is a core result:

| Model | Training tokens / FLOPs | JMMLU |
|---|---:|---:|
| PLaMo 2.0–8B | 6T tokens, 288,000×10¹⁸ FLOPs | 0.572 |
| PLaMo 2.1–8B | 500B tokens, 55,000×10¹⁸ FLOPs | 0.672 |
| PLaMo 2.0–31B | 2T tokens, 372,000×10¹⁸ FLOPs | 0.635 |

These numbers support the report’s central claim that pruning plus distillation is more effective than scaling tokens at fixed size in this regime [2509.04897]. A plausible implication is that PLaMo 2 should be interpreted not only as a model family but also as a training doctrine: synthetic Japanese augmentation and teacher-derived compression are treated as primary levers, not auxiliary optimizations.

## 4. Post-training, chat formatting, and inference optimization

PLaMo 2’s post-training stack consists of **supervised fine-tuning (SFT)**, **model merging**, and **direct preference optimization (DPO)**. The report describes two SFT branches: a general Japanese instruction-following branch and a math/coding-heavy branch. These are subsequently merged in weight space using a weighted average akin to model soups, after which DPO is applied using synthetic preference pairs scored by **Nemotron4-340B-reward** [2509.04897].

The DPO stage includes two stated enhancements: **length regularization** to discourage overly long outputs and an **SFT loss on chosen responses** to retain likelihood on preferred completions. Preference data are constructed by generating multiple candidate responses per prompt, selecting the maximum- and minimum-reward responses, and discarding pairs with too small a reward gap [2509.04897].

For conversational formatting, the models use a **ChatML-like** format with a dedicated separator token `<|plamo:op|>`. This choice replaces Alpaca-style headers and is described as consistent with pre-training exposure to QA and conversational data. For structured benchmarks that separate instruction and input, the report represents instructions as the main user message and inputs as a separate “input” role [2509.04897].

Inference engineering is equally central. PLaMo 2.0–31B is integrated into **vLLM** with custom handling for both attention blocks and Mamba/SSM blocks. The deployment stack includes **pipeline parallelism**, **tensor parallelism**, **chunked prefill**, and selective use of **torch.compile** with TorchInductor. Quantization uses **GPTQ INT4** for weights and **FP8** for the KV cache. The reported model size shrinks from approximately **63GB** in BF16 to **17GB** in INT4, and FP8 KV quantization approximately halves KV memory per token, increasing the threshold before recomputation from around **128 concurrent requests** to around **256 concurrent requests** [2509.04897].

The report further states that quantization causes only marginal degradation on key evaluations. For example, on **Jaster**, the average score changes from **0.665** with BF16/BF16 to **0.662** with INT4/FP8(E4M3), while on **M-IFEval Japanese** the INT4/FP8(E4M3) configuration slightly improves the average from **0.677** to **0.684** [2509.04897]. These results are important because they indicate that the “PLaMo 2” identity includes not only model weights and training recipes but also a concrete, low-overhead inference profile.

## 5. Evaluation profile and Japanese-language performance

PLaMo 2 is evaluated on a broad Japanese-centric suite spanning pre-training and post-training benchmarks. On **JMMLU** and **MMLU**, the report shows that **PLaMo 2.1–8B** reaches **0.635** on both metrics, surpassing **PLaMo-100B** on JMMLU, where the latter records **0.575**. On **pfgen-bench** pre-training evaluation, **PLaMo 2.0–31B** records **0.817**, which the report identifies as the best among the compared models at that scale, while **PLaMo 2.1–8B** records **0.725** [2509.04897].

The strongest claims concern post-training Japanese performance. On **Jaster**, **PLaMo 2.1–8B** reaches **0.626**, outperforming **Qwen2.5–7B-Instruct** at **0.501**, **Qwen3–8B** at **0.593**, and **Llama-3.1–8B-Instruct** at **0.547**. On the 31B scale, **PLaMo 2.0–31B** reaches **0.665**, exceeding **PLaMo 1.0 Prime (100B)** at **0.620**, **Qwen2.5–32B-Instruct** at **0.659**, and **gpt-4o-mini** at **0.635** [2509.04897].

On **M-IFEval Japanese**, **PLaMo 2.1–8B** records **0.630**, compared with **0.463** for **Qwen2.5–7B-Instruct** and **0.342** for **Llama-3.1–8B-Instruct**. At 31B, **PLaMo 2.0–31B** reaches **0.677**, compared with **0.628** for **Qwen2.5–32B-Instruct**, **0.629** for **Qwen3–32B (thinking)**, and **0.610** for **gpt-4o-mini** [2509.04897].

On **pfgen-bench** post-training evaluation, the reported totals are **0.893** for **PLaMo 2.1–8B** and **0.890** for **PLaMo 2.0–31B**, both exceeding the compared Qwen and PLaMo 1.0 baselines. The 8B model is reported with **Fluency 0.964**, **Truthfulness 0.960**, and **Helpfulness 0.755**; the 31B model with **Fluency 0.951**, **Truthfulness 0.952**, and **Helpfulness 0.767** [2509.04897].

These results justify describing PLaMo 2 as state-of-the-art on Japanese benchmarks among similarly sized open models, but they also reveal a specific performance profile. The family is especially strong on Japanese instruction compliance, fluency, and Japanese-specific knowledge, while remaining less dominant on general multilingual code and reasoning benchmarks. The report explicitly notes that **Qwen3-8B** remains stronger on code, and that PLaMo 2 still lags on some reasoning-heavy tasks [2509.04897].

## 6. Extensions across modalities: speech and vision-language variants

Although PLaMo 2 is fundamentally a text LLM family, the broader PLaMo line now includes multimodal extensions that illuminate how the architecture is meant to be used in practice.

In speech recognition, the earlier PLaMo-100B work demonstrates a decoder-only ASR pipeline in which a pre-trained **Whisper large-v3 encoder** produces frame-level audio features, a learned **linear adapter** maps stacked audio features into the text embedding space, and the decoder consumes a single sequence containing `<bos>`, audio embeddings, language tags, optional keywords, and transcription tokens. Crucially, the paper states that **the model architecture of PLaMo-100B is not changed**, and ASR is achieved through prompting and lightweight adaptation with **QLoRA** [2408.08027].

That work reports that only about **0.21% of the decoder’s parameters** are learnable during ASR fine-tuning for PLaMo-100B, with **218,234,880** trainable parameters out of **103,098,001,920**. On Japanese ASR without keyword use, the PLaMo-based system achieves **5.83%** CER on **CommonVoice v8.0 test**, **13.30%** CER on **CommonVoice v16.1 test**, and **10.67%** CER on **YODAS test**, outperforming Whisper on those Japanese evaluations. On English **LibriSpeech test**, Whisper remains better at **2.95%** WER versus **3.63%** for PLaMo [2408.08027].

The same paper shows the effect of prompt-based contextualization for rare and ambiguous words. On YODAS, training with keywords and using keywords at inference improves PLaMo-100B from **12.45%** to **11.47%** CER on dev and from **10.67%** to **9.48%** CER on test. For unseen rare keywords, **KWER** drops from **63.77%** to **24.29%** on dev and from **52.74%** to **10.45%** on test [2408.08027]. This suggests continuity with PLaMo 2’s broader philosophy: contextual conditioning should be accomplished by prompt construction and adapter modules rather than invasive architectural specialization.

In vision-language modeling, **PLaMo 2.1-VL** extends the **PLaMo 2.1** text LLMs into a family of lightweight VLMs for local and edge deployment, with **8B** and **2B** variants. The architecture is **LLaVA-style**, using a **SigLIP2** image encoder, a simple **MLP adapter**, and **dynamic tiling**, with the underlying PLaMo 2.1 LLM already instruction-tuned via **DPO** [2604.19324].

The VLM report emphasizes two core capabilities: **Visual Question Answering (VQA)** and **Visual Grounding / REC**. On **JA-VG-VQA-500**, **PLaMo 2.1-8B-VL** reaches **61.5 ROUGE-L** and **72.4%** by an LLM judge, while **PLaMo 2.1-2B-VL** reaches **60.7 ROUGE-L** and **71.6%**. On **Japanese Ref-L4**, the 8B model reaches **85.2** REC accuracy and the 2B model **82.4**, both outperforming several open baselines on Japanese [2604.19324].

The industrial evaluations are equally notable. On **factory task analysis**, **PLaMo 2.1-8B-VL** achieves **53.9%** zero-shot accuracy, exceeding the compared Qwen baselines. On **infrastructure anomaly detection**, the zero-shot **bbox + label** macro F1 is **39.3** for the 8B model, and fine-tuning improves the corresponding score from **39.7** to **64.9** on the specified evaluation set [2604.19324]. These results situate PLaMo 2 as a foundation for Japanese-first multimodal systems, not merely a monomodal LLM family.

## 7. Limitations, controversies, and broader significance

The PLaMo 2 technical report is explicit about several limitations. It states that the family still underperforms some competitors on reasoning and math-heavy tasks, and that it is **not tuned for explicit chain-of-thought** outputs. It also identifies a structural trade-off in the original Samba-based design: the architecture is efficient but fundamentally limited for long-range exact retrieval until full-attention continual pre-training is introduced [2509.04897].

Another stated concern is data bias. Because training relies heavily on synthetic corpora and Japanese web sources, the models may encode biases specific to Japanese internet subcultures. Safety evaluation on **BFCL V2** remains strong but not perfect, and quantization slightly reduces the safety score from **87.5%** to **85.25%** in the reported comparison between BF16/BF16 and INT4/FP8(E4M3) configurations [2509.04897].

The speech recognition study identifies a different kind of limitation: contextual prompts are highly effective, but prompt distribution can interact with dataset composition in undesirable ways. In particular, when keyword-bearing and keyword-free datasets are mixed asymmetrically, the model can learn dataset-specific biases, such as associating missing keywords with incomplete transcriptions and elevated deletion errors [2408.08027]. The vision-language report likewise stresses that industrial deployment requires human-in-the-loop validation, clear operational assumptions, and awareness of false positives and false negatives, especially for small anomalies and state-driven anomalies [2604.19324].

Taken together, these reports indicate that PLaMo 2 is best understood as a technically coherent Japanese-centric model ecosystem built around four commitments: aggressive efficiency engineering, synthetic-data amplification, strong instruction tuning, and prompt-compatible multimodal extensibility. Its significance lies less in maximal parameter scale than in demonstrating that an 8B-class Japanese model can match or exceed the earlier 100B line on many Japanese tasks through architectural redesign, pruning, distillation, and post-training [2509.04897]. A plausible implication is that the broader PLaMo trajectory represents a shift from brute-force scale to domain-focused systems optimization, with Japanese language performance and local deployability as the primary constraints.

Source: https://www.emergentmind.com/topics/plamo-2