Evaluating Japanese Dialect Robustness Across Speech and Text-based Large Language Models

Abstract: Dialogue systems based on LLMs have advanced significantly in recent years. However, dialectal variation remains a major challenge, particularly for systems that process spoken input. LLM-based speech LLMs (SLMs), which integrate LLMs with speech processing components, show promise for spoken language tasks, yet their ability to comprehend dialects has not been sufficiently studied. Moreover, it remains unclear how the dialectal understanding of the base LLM affects SLM performance. This study investigates the dialectal robustness of both LLMs and SLMs using Japanese dialects as a test case. We define robustness as the ratio of performance on dialectal versus standard inputs, enabling fair comparisons. Our experiments show that SLM robustness correlates with that of their text-based counterparts. Furthermore, training with dialectal data and fine-tuning the speech encoder each improves robustness in SLMs.

• The paper introduces a novel robustness metric comparing dialectal versus standard BLEU scores across LLMs and SLMs.
• It leverages the CPJD corpus of 20 dialects and applies LoRA fine-tuning and adapter training to improve translation accuracy.
• Experimental results reveal that speech models face additional robustness penalties, emphasizing the need for fine-tuning and dialect-specific training.

Evaluation of Japanese Dialect Robustness in Speech and Text-based LLMs

Introduction

The paper "Evaluating Japanese Dialect Robustness Across Speech and Text-based LLMs" (2606.25436) presents a comprehensive study of dialectal robustness within both text- and speech-capable LLMs (LLMs/SLMs), focusing particularly on Japanese dialects. The authors identify major gaps in spoken dialogue systems, where dialectal variation leads to exacerbated performance drops, especially for speech-based systems. They introduce a robustness metric—defined as the ratio of performance on dialectal versus standard inputs—enabling fair and interpretable comparisons across modalities and model architectures.

Figure 1: An example of robustness evaluation using a translation task, comparing LLM outputs for standard and dialectal inputs using BLEU.

Dataset and Evaluation Framework

Central to this study is the Crowdsourced Parallel Speech Corpus of Japanese Dialects (CPJD), which spans 20 regional dialects. Each region is associated with one of two sentence sets, thus restricting overlap and enabling controlled evaluation.

Figure 2: Geographic distribution of the 20 dialects in the CPJD corpus, with color-coding by sentence set assignment.

To facilitate cross-modality evaluation, the primary task is translation from Japanese (standard or dialectal) to English, measured via BLEU and BLEURT metrics. Where the CPJD lacked English references, the authors generated them using LLMs, and for missing audio, synthesized speech using TTS. The robustness metric allows normalization relative to each model's baseline, minimizing confounds due to model strength or evaluation difficulty.

Model Architectures

The paper evaluates both instruction-tuned LLMs (text-only translation) and speech-to-text SLMs constructed by connecting Whisper-Large-V3 encoders to LLMs, mediated via downsampling and linear projection adapters. For text translation, LoRA fine-tuning is applied to suppress prompt-following chatter and achieve focused output alignment.

Experimental Results

RQ1: Dialectal Competence Transfer from LLM to SLM

Across all architectures and dialects, absolute scores for dialectal input are consistently lower than for standard Japanese, a deficit magnified in audio-based SLMs. Robustness scores—computed as the ratio of BLEU for dialectal versus standard input—are shown to be highly correlated between LLMs and their speech-based SLM counterparts (Pearson coefficients: 0.848 BLEU, 0.910 BLEURT), indicating that dialectal competence is partially transferred from text to speech modality. However, SLMs incur an additional robustness penalty, likely due to acoustic phenomena (intonation, phonetic variability) not addressed by base LLMs.

Figure 3: BLEU robustness for text and audio models, showing correlation and decline in robustness for SLMs.

RQ2: Impact of Dialectal Data in SLM Training

Training SLMs with CPJD dialectal data markedly improves robustness for nearly all dialects as evidenced by most points lying above the diagonal in a BLEU robustness scatter plot. This indicates that adapter-level training on dialectal data effectively enhances generalization, enabling the model to map dialectal lexicon and constructs to standard translation targets.

Figure 4: Robustness comparison showing positive impact of CPJD dialectal data in SLM training, as measured by BLEU.

RQ3: Speech Encoder Fine-tuning

Fine-tuning the Whisper encoder—previously fixed in baseline SLMs—yields substantial gains in dialectal robustness, especially for dialects exhibiting initial deficits (e.g., Morokata, Tsugaru). This is strong evidence for insufficient coverage of regional Japanese in Whisper, which benefits from targeted adaptation.

Figure 5: Robustness comparison for audio models with frozen vs. fine-tuned speech encoders, highlighting improved dialectal handling.

Implications and Future Directions

The findings underscore that SLM dialectal robustness is strongly dependent on (1) the underlying LLM's performance on dialectal text, (2) targeted dialectal adapter training, and (3) speech encoder adaptation. This reveals key pathways for mitigating fairness concerns and broadening inclusivity in dialogue systems. Practically, applications in public services, voice assistants, and conversational agents would benefit from dialectal coverage; theoretically, research in multi-dialect and low-resource adaptation could leverage robustness-normalized evaluation frameworks.

Further, since dialect competence is only partially transferred from text to speech, multimodal training—which explicitly builds dialectal parity across modalities—remains an open technical frontier. Future work could exploit massively multilingual corpora and self-supervised learning for acoustic dialect representation, or augment LLMs with cross-lingual/cross-dialectal prompts.

Conclusion

This paper provides an integrated evaluation of Japanese dialect robustness in both text and speech modalities using LLMs and SLMs. Dialectal competence in SLMs reflects the strength of the underlying LLM, and can be significantly improved by dialectal adapter training and encoder fine-tuning. The study's robustness metric enables normalized, interpretable comparison across systems, offering a technical foundation for inclusive multilingual and multi-dialectal speech technology development.