Benchmarking Large Language Models for Grapheme-to-Phoneme Conversion: A Japanese Case Study

Abstract: Grapheme-to-phoneme (G2P) conversion is essential for controllable and robust text-to-speech, and LLMs, with broad linguistic knowledge, offer a promising approach. We benchmarked over 30 LLMs on Japanese G2P, comparing them with conventional morphological analyzers on 3000 manually annotated sentences. We evaluated two prompting strategies: a parse mode, where the LLM performs morphological analysis followed by rule-based kana conversion, and a direct mode, where the LLM directly predicts kana readings. The results show that model size, version, and Japanese-specialized training are key factors, with the best LLMs achieving kana character error rate below 0.52\% vs. the best conventional tool (1.03\%). Parse mode outperforms direct mode for most models, as rule-based post-processing relieves the LLM of handling complex pronunciation rules. We also show that feeding LLM-predicted kana into a kana-input TTS yields better pronunciation than end-to-end TTS.

• The paper's main finding is that LLMs, especially Japanese-specialized variants, dramatically lower kana Character Error Rate compared to traditional G2P systems.
• Experimental results demonstrate that parse mode generally outperforms direct mode, with larger models achieving superior accuracy and robustness.
• Combining LLM-based G2P with TTS systems boosts pronunciation accuracy, offering a practical advantage over end-to-end TTS pipelines.

Benchmarking LLMs for Japanese Grapheme-to-Phoneme Conversion

Introduction and Motivation

Explicit grapheme-to-phoneme (G2P) conversion is critical for Japanese TTS, mainly due to the complexities associated with kanji polyphony, context-dependent readings, and irregular numeral–counter combinations. End-to-end TTS models have reduced reliance on explicit G2P modules, but their inherent opaqueness and lack of controllability compromise robustness, especially for proper nouns and out-of-vocabulary terms. LLMs, given their extensive linguistic priors and prompt-based controllability, present new opportunities to address these challenges. The paper systematically benchmarks LLMs on Japanese G2P, evaluating both proprietary APIs and open-weight models, and compares them to established morphological analyzers.

Methodological Framework

Two LLM prompting approaches are examined:

• Parse Mode: The LLM performs morphological analysis, yielding a sequence of segmented words with associated kana readings. A deterministic rule-based post-processing module subsequently enforces context-sensitive pronunciation transformations, such as particle conversion and long vowel normalization.

Figure 1: Parse mode pipeline, leveraging LLMs for segmentation and reading estimation, followed by rule-based pronunciation post-processing.

• Direct Mode: The LLM directly maps raw Japanese text to its kana reading, subsuming segmentation, reading estimation, and pronunciation rule handling within a single model execution.

Figure 2: Direct mode pipeline, with LLMs directly outputting kana readings in one step.

A manually annotated reference corpus (3000 sentences from JVS) ensures rigorous evaluation. Accuracy is assessed via kana Character Error Rate (CER), which correlates closely with phoneme error rate due to the composite nature of Japanese kana.

Experimental Findings

Model Scaling and CER Dynamics

The study benchmarks over 30 LLMs, spanning proprietary APIs (Claude Opus 4.6, Gemini 3.1 Pro, GPT-5.2) and open-weight models of varying architectural families and parameter counts (Gemma, Qwen, Llama, GLM, Kimi, gpt-oss). For open-weight models, CER performance consistently improves with increasing parameter count, confirming scaling law applicability to the G2P domain. Within the same family, larger models yield lower CER; e.g., Gemma3-27B markedly outperforms Gemma3-4B and Gemma3-12B.

Figure 3: Relationship between model size and kana CER for open-weight LLMs in parse mode.

Notably, proprietary models achieve strong results: Claude Opus 4.6 reaches 0.52% CER in parse mode, Gemini 3.1 Pro achieves 0.53% in direct mode, both significantly outperforming OpenJTalk (1.03%) and other conventional analyzers.

Impact of Japanese-Specialized Training

Japanese-specialized variants (e.g., Swallow pre-training) systematically reduce CER compared to their generic counterparts. Llama3.3-Swallow-70B yields 2.85% versus the base Llama3.3-70B's 6.58%. Swallow-enhanced Qwen and Gemma models exhibit similar improvements, underscoring the efficacy of targeted linguistic adaptation for G2P.

Parse Mode vs. Direct Mode

Parse mode is advantageous for most models, particularly smaller LLMs, due to the offloading of deterministic pronunciation rule enforcement to the post-processing step. Direct mode's performance degrades under instruction complexity and rule entanglement, with pronounced failures observed for smaller or less specialized models (e.g., LLM-jp-3.1-13b: 86.02% CER). Exceptions occur with select proprietary models (Gemini 3.1 Pro, GPT-5.2), where direct mode slightly outperforms, attributed to their prompt-following reliability and superior contextual understanding.

LLMs vs Conventional Tools

LLMs demonstrate superior performance on out-of-vocabulary and loanwords, areas where dictionary-dependent methods fail. In contrast, LLMs occasionally generate plausible but incorrect readings for high-frequency terms, reflecting their probabilistic nature. Conventional analyzers are consistently outperformed by LLMs with specialized and sufficiently large architectures.

Effects of Reasoning Modes

Higher reasoning effort ("thinking mode") marginally improves segmentation and reading accuracy for high-performing models, as shown by Gemini 3 Flash’s CER reduction from 0.94% to 0.54%. For most models, differences are negligible, indicating that error sources are dominated by the base linguistic knowledge rather than reasoning depth.

Implications for TTS Performance

Combining LLM-based G2P with kana-input TTS models yields better pronunciation accuracy than state-of-the-art E2E TTS systems while maintaining similar levels of naturalness. G2P-driven synthesis with Gemini 3.1 Pro (direct) achieves a CER of 2.38%, approaching the oracle (true kana input, 2.10%), compared to the best E2E TTS system at 3.96%. This demonstrates not only a practical advantage in accuracy but also validates the retention of naturalness metrics (UTMOS) in hybrid pipelines.

Theoretical and Practical Implications

The results provide strong empirical evidence for scaling laws in G2P, validate targeted pre-training as an effective strategy, and highlight the continuing relevance of explicit G2P modules for robust and controllable TTS in Japanese. Furthermore, LLM-based systems facilitate out-of-vocabulary and loanword adaptation, which is crucial for dynamic, user-facing applications. Direct mode's vulnerability in smaller models suggests prompt engineering and hybrid rule-based strategies remain necessary for high-integrity deployments.

Anticipated future work includes optimization of prompt design, ensemble utilization, and replacing deterministic post-processing with LLM-based rule application, potentially further enhancing pronunciation accuracy.

Conclusion

Comprehensive benchmarking reveals that large, Japanese-specialized LLMs dramatically surpass conventional morphological analyzers for G2P in Japanese, with proprietary API models achieving the lowest CER. Parse mode remains optimal for most models, and explicit G2P improves downstream TTS pronunciation accuracy without compromising naturalness. Continued development of LLM-based G2P modules will further mitigate errors on proper nouns, loanwords, and context-sensitive readings, laying the groundwork for robust, high-fidelity Japanese TTS pipelines.