PLaMo 2: Japanese LLM Evolution
- PLaMo 2 is a Japanese-focused LLM family featuring a hybrid Samba-based architecture and continual pre-training to enable full attention over 32K-token contexts.
- It leverages synthetic corpora, structured pruning, and knowledge distillation to overcome data scarcity and optimize model efficiency.
- The ecosystem expands into speech recognition and vision-language variants, demonstrating strong instruction tuning and scalable inference optimization.
Searching arXiv for the PLaMo 2 technical report and closely related PLaMo papers. PLaMo 2 is a series of Japanese-focused LLMs developed by Preferred Networks as the successor to PLaMo-100B. It is characterized by a hybrid Samba-based architecture, continual pre-training that transitions to full attention for 32K-token contexts, extensive use of synthetic corpora to address Japanese data scarcity, and an efficiency-oriented pipeline based on weight reuse, structured pruning, post-training with supervised fine-tuning and direct preference optimization, and inference optimization with vLLM and quantization (Networks et al., 5 Sep 2025). The term also encompasses an expanding model ecosystem built on the same base LLMs, including contextualized speech recognition with PLaMo-100B as a precursor and the multimodal PLaMo 2.1-VL family for visual question answering and visual grounding (Nozawa et al., 2024, Kerola et al., 21 Apr 2026).
1. Origin, scope, and positioning
PLaMo 2 is explicitly presented as a family of Japanese-focused LLMs centered around two main base scales: PLaMo 2.0–31B and PLaMo 2.0–8B / 2.1–8B. The 31B line is positioned as a high-end open model and successor to PLaMo-100B in capability, whereas the 8B line is designed to reach or surpass PLaMo-100B performance at a fraction of the size and training compute (Networks et al., 5 Sep 2025).
The design goals are threefold. First, PLaMo 2 targets strong Japanese-centric capability, including Japanese web text, translation, Japanese-specific cultural and knowledge tasks, and instruction following. Second, it emphasizes efficiency and scalability through weight reuse, pruning, and inference-oriented systems engineering. Third, it supports long-context processing, with the final models configured for 32K tokens after continual pre-training and rotary-position reconfiguration (Networks et al., 5 Sep 2025).
This positioning marks a transition from the earlier PLaMo-100B regime to a smaller but more optimized model family. In the speech recognition work based on PLaMo-100B, the 100-billion-parameter model served as a decoder-only transformer LLM that could be adapted to automatic speech recognition through a pre-trained Whisper encoder, a linear adapter, and QLoRA fine-tuning, without changing the model architecture (Nozawa et al., 2024). PLaMo 2 generalizes the underlying design philosophy: preserve a strong text LLM core, then extend it through efficient architectural modification, synthetic data, and downstream specialization.
A plausible implication is that “PLaMo 2” should be understood less as a single checkpoint than as a platform architecture: a text-first Japanese LLM family from which smaller, instruction-tuned, speech-adapted, and vision-language variants can be derived while preserving a consistent deployment and adaptation strategy.
2. Core architecture and long-context transition
The base PLaMo 2 models adopt a Samba-based hybrid architecture in which Mamba / SSM layers alternate with Sliding-Window Attention (SWA) layers. This design is intended to provide linear-time sequence processing and constant memory characteristics for the SSM component, while retaining localized self-attention via a fixed-size attention window (Networks et al., 5 Sep 2025).
The report identifies a central limitation of this hybrid design: long-context exact retrieval degrades sharply once the relevant token lies beyond the SWA window. On Phonebook and Passkey Retrieval, a 1B PLaMo 2 model collapses to near-zero accuracy when the relevant information is more than 2048 tokens away, which the report attributes to the lossy compression of past information into a fixed-size hidden state and the locality constraint of SWA (Networks et al., 5 Sep 2025).
To address this, PLaMo 2 uses Continual Pre-training (CPT) in which the SWA window is enlarged to the full target context length, effectively converting SWA into full attention. At the same time, the models adopt Adjustable Base Frequency RoPE (ABF-RoPE) with base frequency , enabling stable rotary positional encoding up to 32K tokens. CPT is performed on a subset of the original pre-training corpus, and the final checkpoint is chosen to balance long-context retrieval improvements with preservation of overall Japanese text generation quality (Networks et al., 5 Sep 2025).
This architectural evolution is central to the identity of PLaMo 2. The “hybrid-to-full-attention” transition is not an incidental finetuning step but a structural response to a measured failure mode. This suggests that PLaMo 2 occupies an intermediate point between sequence-model efficiency and transformer-style random-access retrieval: its pretraining pipeline exploits hybrid efficiency, while its deployed long-context capability depends on later full-attention adaptation.
3. Data strategy, synthetic corpora, and model compression
A defining feature of PLaMo 2 is its reliance on synthetic corpora to compensate for the relative scarcity of high-quality large-scale Japanese data. The training pipeline combines CommonCrawl-based Japanese and English web corpora, code data extracted from public sources and filtered CommonCrawl pages, and large synthetic datasets for translation, paraphrasing, code, and math (Networks et al., 5 Sep 2025).
For Japanese web data, the report combines an educational value filter in the style of FineWeb-Edu with categorization + downsampling in the style of DataComp-LM. For coding data, it uses public code datasets and CommonCrawl pages filtered by HTML tags such as <pre> and <code>, then classified as coding-oriented with a fastText classifier trained on LLM-labeled data. For synthetic Japanese text, it translates high-quality English corpora into Japanese, paraphrases Japanese corpora, and augments code and math through multi-stage LLM generation pipelines (Networks et al., 5 Sep 2025).
The 8B line, especially PLaMo 2.1–8B, is strongly shaped by structured pruning and knowledge distillation. The report states that PLaMo 2.1–8B is derived from the fully trained PLaMo 2–31B using a Minitron-based structured pruning procedure, followed by retraining with a KL-based distillation loss from the 31B teacher. For memory efficiency, teacher logits are truncated to the top 128 tokens, and the resulting 8B model inherits much of the teacher’s capability while requiring far less compute (Networks et al., 5 Sep 2025).
The compute-performance comparison reported for JMMLU is a core result:
| Model | Training tokens / FLOPs | JMMLU |
|---|---|---|
| PLaMo 2.0–8B | 6T tokens, 288,000×10¹⁸ FLOPs | 0.572 |
| PLaMo 2.1–8B | 500B tokens, 55,000×10¹⁸ FLOPs | 0.672 |
| PLaMo 2.0–31B | 2T tokens, 372,000×10¹⁸ FLOPs | 0.635 |
These numbers support the report’s central claim that pruning plus distillation is more effective than scaling tokens at fixed size in this regime (Networks et al., 5 Sep 2025). A plausible implication is that PLaMo 2 should be interpreted not only as a model family but also as a training doctrine: synthetic Japanese augmentation and teacher-derived compression are treated as primary levers, not auxiliary optimizations.
4. Post-training, chat formatting, and inference optimization
PLaMo 2’s post-training stack consists of supervised fine-tuning (SFT), model merging, and direct preference optimization (DPO). The report describes two SFT branches: a general Japanese instruction-following branch and a math/coding-heavy branch. These are subsequently merged in weight space using a weighted average akin to model soups, after which DPO is applied using synthetic preference pairs scored by Nemotron4-340B-reward (Networks et al., 5 Sep 2025).
The DPO stage includes two stated enhancements: length regularization to discourage overly long outputs and an SFT loss on chosen responses to retain likelihood on preferred completions. Preference data are constructed by generating multiple candidate responses per prompt, selecting the maximum- and minimum-reward responses, and discarding pairs with too small a reward gap (Networks et al., 5 Sep 2025).
For conversational formatting, the models use a ChatML-like format with a dedicated separator token <|plamo:op|>. This choice replaces Alpaca-style headers and is described as consistent with pre-training exposure to QA and conversational data. For structured benchmarks that separate instruction and input, the report represents instructions as the main user message and inputs as a separate “input” role (Networks et al., 5 Sep 2025).
Inference engineering is equally central. PLaMo 2.0–31B is integrated into vLLM with custom handling for both attention blocks and Mamba/SSM blocks. The deployment stack includes pipeline parallelism, tensor parallelism, chunked prefill, and selective use of torch.compile with TorchInductor. Quantization uses GPTQ INT4 for weights and FP8 for the KV cache. The reported model size shrinks from approximately 63GB in BF16 to 17GB in INT4, and FP8 KV quantization approximately halves KV memory per token, increasing the threshold before recomputation from around 128 concurrent requests to around 256 concurrent requests (Networks et al., 5 Sep 2025).
The report further states that quantization causes only marginal degradation on key evaluations. For example, on Jaster, the average score changes from 0.665 with BF16/BF16 to 0.662 with INT4/FP8(E4M3), while on M-IFEval Japanese the INT4/FP8(E4M3) configuration slightly improves the average from 0.677 to 0.684 (Networks et al., 5 Sep 2025). These results are important because they indicate that the “PLaMo 2” identity includes not only model weights and training recipes but also a concrete, low-overhead inference profile.
5. Evaluation profile and Japanese-language performance
PLaMo 2 is evaluated on a broad Japanese-centric suite spanning pre-training and post-training benchmarks. On JMMLU and MMLU, the report shows that PLaMo 2.1–8B reaches 0.635 on both metrics, surpassing PLaMo-100B on JMMLU, where the latter records 0.575. On pfgen-bench pre-training evaluation, PLaMo 2.0–31B records 0.817, which the report identifies as the best among the compared models at that scale, while PLaMo 2.1–8B records 0.725 (Networks et al., 5 Sep 2025).
The strongest claims concern post-training Japanese performance. On Jaster, PLaMo 2.1–8B reaches 0.626, outperforming Qwen2.5–7B-Instruct at 0.501, Qwen3–8B at 0.593, and Llama-3.1–8B-Instruct at 0.547. On the 31B scale, PLaMo 2.0–31B reaches 0.665, exceeding PLaMo 1.0 Prime (100B) at 0.620, Qwen2.5–32B-Instruct at 0.659, and gpt-4o-mini at 0.635 (Networks et al., 5 Sep 2025).
On M-IFEval Japanese, PLaMo 2.1–8B records 0.630, compared with 0.463 for Qwen2.5–7B-Instruct and 0.342 for Llama-3.1–8B-Instruct. At 31B, PLaMo 2.0–31B reaches 0.677, compared with 0.628 for Qwen2.5–32B-Instruct, 0.629 for Qwen3–32B (thinking), and 0.610 for gpt-4o-mini (Networks et al., 5 Sep 2025).
On pfgen-bench post-training evaluation, the reported totals are 0.893 for PLaMo 2.1–8B and 0.890 for PLaMo 2.0–31B, both exceeding the compared Qwen and PLaMo 1.0 baselines. The 8B model is reported with Fluency 0.964, Truthfulness 0.960, and Helpfulness 0.755; the 31B model with Fluency 0.951, Truthfulness 0.952, and Helpfulness 0.767 (Networks et al., 5 Sep 2025).
These results justify describing PLaMo 2 as state-of-the-art on Japanese benchmarks among similarly sized open models, but they also reveal a specific performance profile. The family is especially strong on Japanese instruction compliance, fluency, and Japanese-specific knowledge, while remaining less dominant on general multilingual code and reasoning benchmarks. The report explicitly notes that Qwen3-8B remains stronger on code, and that PLaMo 2 still lags on some reasoning-heavy tasks (Networks et al., 5 Sep 2025).
6. Extensions across modalities: speech and vision-language variants
Although PLaMo 2 is fundamentally a text LLM family, the broader PLaMo line now includes multimodal extensions that illuminate how the architecture is meant to be used in practice.
In speech recognition, the earlier PLaMo-100B work demonstrates a decoder-only ASR pipeline in which a pre-trained Whisper large-v3 encoder produces frame-level audio features, a learned linear adapter maps stacked audio features into the text embedding space, and the decoder consumes a single sequence containing <bos>, audio embeddings, language tags, optional keywords, and transcription tokens. Crucially, the paper states that the model architecture of PLaMo-100B is not changed, and ASR is achieved through prompting and lightweight adaptation with QLoRA (Nozawa et al., 2024).
That work reports that only about 0.21% of the decoder’s parameters are learnable during ASR fine-tuning for PLaMo-100B, with 218,234,880 trainable parameters out of 103,098,001,920. On Japanese ASR without keyword use, the PLaMo-based system achieves 5.83% CER on CommonVoice v8.0 test, 13.30% CER on CommonVoice v16.1 test, and 10.67% CER on YODAS test, outperforming Whisper on those Japanese evaluations. On English LibriSpeech test, Whisper remains better at 2.95% WER versus 3.63% for PLaMo (Nozawa et al., 2024).
The same paper shows the effect of prompt-based contextualization for rare and ambiguous words. On YODAS, training with keywords and using keywords at inference improves PLaMo-100B from 12.45% to 11.47% CER on dev and from 10.67% to 9.48% CER on test. For unseen rare keywords, KWER drops from 63.77% to 24.29% on dev and from 52.74% to 10.45% on test (Nozawa et al., 2024). This suggests continuity with PLaMo 2’s broader philosophy: contextual conditioning should be accomplished by prompt construction and adapter modules rather than invasive architectural specialization.
In vision-language modeling, PLaMo 2.1-VL extends the PLaMo 2.1 text LLMs into a family of lightweight VLMs for local and edge deployment, with 8B and 2B variants. The architecture is LLaVA-style, using a SigLIP2 image encoder, a simple MLP adapter, and dynamic tiling, with the underlying PLaMo 2.1 LLM already instruction-tuned via DPO (Kerola et al., 21 Apr 2026).
The VLM report emphasizes two core capabilities: Visual Question Answering (VQA) and Visual Grounding / REC. On JA-VG-VQA-500, PLaMo 2.1-8B-VL reaches 61.5 ROUGE-L and 72.4% by an LLM judge, while PLaMo 2.1-2B-VL reaches 60.7 ROUGE-L and 71.6%. On Japanese Ref-L4, the 8B model reaches 85.2 REC accuracy and the 2B model 82.4, both outperforming several open baselines on Japanese (Kerola et al., 21 Apr 2026).
The industrial evaluations are equally notable. On factory task analysis, PLaMo 2.1-8B-VL achieves 53.9% zero-shot accuracy, exceeding the compared Qwen baselines. On infrastructure anomaly detection, the zero-shot bbox + label macro F1 is 39.3 for the 8B model, and fine-tuning improves the corresponding score from 39.7 to 64.9 on the specified evaluation set (Kerola et al., 21 Apr 2026). These results situate PLaMo 2 as a foundation for Japanese-first multimodal systems, not merely a monomodal LLM family.
7. Limitations, controversies, and broader significance
The PLaMo 2 technical report is explicit about several limitations. It states that the family still underperforms some competitors on reasoning and math-heavy tasks, and that it is not tuned for explicit chain-of-thought outputs. It also identifies a structural trade-off in the original Samba-based design: the architecture is efficient but fundamentally limited for long-range exact retrieval until full-attention continual pre-training is introduced (Networks et al., 5 Sep 2025).
Another stated concern is data bias. Because training relies heavily on synthetic corpora and Japanese web sources, the models may encode biases specific to Japanese internet subcultures. Safety evaluation on BFCL V2 remains strong but not perfect, and quantization slightly reduces the safety score from 87.5% to 85.25% in the reported comparison between BF16/BF16 and INT4/FP8(E4M3) configurations (Networks et al., 5 Sep 2025).
The speech recognition study identifies a different kind of limitation: contextual prompts are highly effective, but prompt distribution can interact with dataset composition in undesirable ways. In particular, when keyword-bearing and keyword-free datasets are mixed asymmetrically, the model can learn dataset-specific biases, such as associating missing keywords with incomplete transcriptions and elevated deletion errors (Nozawa et al., 2024). The vision-language report likewise stresses that industrial deployment requires human-in-the-loop validation, clear operational assumptions, and awareness of false positives and false negatives, especially for small anomalies and state-driven anomalies (Kerola et al., 21 Apr 2026).
Taken together, these reports indicate that PLaMo 2 is best understood as a technically coherent Japanese-centric model ecosystem built around four commitments: aggressive efficiency engineering, synthetic-data amplification, strong instruction tuning, and prompt-compatible multimodal extensibility. Its significance lies less in maximal parameter scale than in demonstrating that an 8B-class Japanese model can match or exceed the earlier 100B line on many Japanese tasks through architectural redesign, pruning, distillation, and post-training (Networks et al., 5 Sep 2025). A plausible implication is that the broader PLaMo trajectory represents a shift from brute-force scale to domain-focused systems optimization, with Japanese language performance and local deployability as the primary constraints.
