Can LLM Agents Identify Spoken Dialects like a Linguist?

Abstract: Due to the scarcity of labeled dialectal speech, audio dialect classification is a challenging task for most languages, including Swiss German. In this work, we explore the ability of LLMs as agents in understanding the dialects and whether they can show comparable performance to models such as HuBERT in dialect classification. In addition, we provide an LLM baseline and a human linguist one. Our approach uses phonetic transcriptions produced by ASR systems and combines them with linguistic resources such as dialect feature maps, vowel history, and rules. Our findings indicate that, when linguistic information is provided, the LLM predictions improve. The human baseline shows that automatically generated transcriptions can be beneficial for such classifications, but also presents opportunities for improvement.

• The paper demonstrates that agentic LLMs, augmented with explicit dialectological knowledge, improve classification accuracy by over 10% compared to standard LLMs.
• It employs a modular LangGraph framework that integrates IPA charts, vowel/consonant analysis, and regional dialect maps to enable chain-of-thought reasoning.
• Results highlight HuBERT's superior performance, underscoring the need for multimodal adaptation and specialized acoustic input for effective dialect identification.

LLM Agents for Dialect Identification: Replicating Linguistic Expertise in Swiss German

Introduction

The paper "Can LLM Agents Identify Spoken Dialects like a Linguist?" (2603.29541) systematically investigates the capacity of current LLM architectures, in both standalone and agentic forms, to classify spoken Swiss German dialects from phonetic transcriptions. The study directly benchmarks these LLM-based approaches against state-of-the-art acoustic encoders (HuBERT) and a human linguist, with a focus on tasks grounded in dialectology, where scarce annotation and complex regional phonology pose major challenges.

Swiss German Dialect Classification: Datasets and Label Mapping

The core datasets are SwissDial and STT4SG-350, both curated to represent the fine-grained socio-geographic variation among Swiss German dialects. SwissDial offers orthographically rich, parallel dialect data from eight canonical regions, each with a single speaker, while STT4SG-350 provides much broader demographic and regional coverage but with coarser region assignments. The label mapping process involves reconciling these ontologies, leveraging regional dialect atlases and geographic information systems. Special attention is given to constructing binary (High Alemannic vs. Highest Alemannic) and eight-way class splits that are both linguistically principled and practically tractable.

Figure 1: SwissDial class regions and the Innerschweiz dialect area as defined in the reference atlas, underpinning the region mapping strategy.

Methodological Architecture: ASR, Agentic LLMs, and Encoder Baselines

The pipeline begins with phonetic ASR using Wav2Vec2Phoneme (XLSR-53), generating IPA transcriptions with a non-negligible PER of 33.3%. Models receive as input both these transcriptions and parallel Standard German text.

The study introduces several system variants:

• HuBERT Encoder Baseline: Fine-tuned and evaluated on balanced train/test splits for both binary and multiclass tasks, leveraging self-supervised audio representations.
• LLM Baselines: Prompted with dialectological reasoning tasks, using only the ASR phonetic input, with standard LLM inference via GPT-4o mini.
• Agentic LLM using LangGraph: Implements a modular, multi-node agent architecture where nodes specialize in vowel/consonant analysis and feature-based reasoning. Nodes receive auxiliary knowledge, including dialect feature maps, historical vowel shifts, and IPA resources, to perform chain-of-thought inference and explicit class justification.

  Figure 2: Multi-node agentic framework for dialect analysis, enabling modular linguistic reasoning over phonetic input and dialect-geographic knowledge.

A formal human baseline is established by providing the same IPA transcriptions and linguistic documentation to an expert dialectologist, who is directed to disregard prior regional familiarity and rely strictly on the evidential record.

Experimental Findings

Comparative Performance

On the central binary classification task (High vs. Highest Alemannic), the following accuracies are observed:

• HuBERT: 66.3%
• Agentic LLM (LangGraph, GPT-4o mini): 58.0%
• Standalone LLM (GPT-4o mini): 47.8%
• Human Linguist: 72.5% (with clear evidence on 58/80 segments, 81% accuracy on certain predictions)

The HuBERT encoder maintains a strong advantage, reflecting the impact of direct acoustic input and fine-tuning. The agentic LLM configuration surpasses a simple LLM prompting baseline by over 10 percentage points, evidencing the contribution of explicit linguistic knowledge and decomposition of analytic reasoning. Human accuracy remains the gold standard, but the gap between agentic LLMs and humans is notably smaller than that between humans and standalone LLMs.

Figure 3: Example IPA transcription input featuring salient Highest Alemannic features, which acts as input for model and human analysis.

Qualitative Analysis: Agentic Reasoning and Alignment

Agentic LLM configurations leverage feature maps, IPA charts, and dialect change histories to guide their inference, partially compensating for the limited prior dialect knowledge of commercial LLMs. Chain-of-thought analyses and explicit confidence scores further aid in interpretability and error analysis. However, prediction imbalance is observed, particularly an over-prediction of Highest Alemannic, suggesting persistent acquisition bottlenecks in the current LLMs' linguistic generalization.

Figure 4: Phone alignments between the human baseline and GPT models; GPT-5 achieves near-perfect alignment, whereas GPT-4o mini exhibits hallucinations and misalignments.

Supplementary comparisons with GPT-5 (via ChatGPT) show that scaling LLM capacity increases the robustness of phonetic reasoning and phone-to-lemma alignment, but even state-of-the-art LLMs fail in scenarios requiring expert handling of ambiguous or underspecified phonological cues.

Practical and Theoretical Implications

Model Robustness and Generalization

The HuBERT encoder, due to direct access to spectral features and large-scale audio pretraining, continues to outperform text-only LLMs and most agentic systems for dialect classification. The performance ceiling for LLMs is only elevated once explicit dialectological knowledge and modular reasoning are introduced via the LangGraph agentic approach. These results indicate that general-purpose LLMs remain suboptimal for dialect identification absent explicit guidance or specialized adaptation.

Limitations and Opportunities

One key limiting factor is the reliance on ASR with high phone error rates, which constrains both model and human accuracy. Furthermore, LLMs fundamentally lack inductive biases for phonetic-geographic inference, necessitating agentic frameworks and rich auxiliary resources.

The study carefully documents that the absence of direct audio input (textual IPA input only for LLMs) constitutes a major difference in representational access between the HuBERT encoder and LLMs, precluding direct architectural comparisons but illuminating the importance of multimodal adaptation. Scalability tests with GPT-5 indicate potential, but significant model and infrastructure advances will be required for parity with human analysis—especially on ambiguous, mixed, or loanword-rich inputs.

Future Directions

The work highlights the future necessity of combining strong speech models with agentic, knowledge-grounded LLM architectures. Promising avenues include:

• Multimodal Agentic LLMs: Extending agent-based reasoning over both audio and phonetic representations using full speech-LLMs
• Few-shot/Fine-tuning Adaptation: Investigating whether LLMs can acquire dialect inference with limited, well-curated exemplars
• Dynamic Resource Integration: Equipping LLM agents with expert-curated dialectological databases, regional sound change tables, and isogloss vector maps for real-time dialectology

Conclusion

The paper delivers a precise assessment of the current limits of LLMs and agentic frameworks for dialect identification tasks where human expertise is rooted in phonological and geographic theory. Agentic LLM approaches using explicit linguistic information can bridge part of the performance gap between standard LLMs and humans, but cannot yet match the leading acoustic encoders or expert judgment in this domain. The findings establish a rigorous baseline for future work at the intersection of computational dialectology, agentic NLP, and multimodal language modeling.