PhonologyBench: Evaluating Phonological Skills of Large Language Models (2404.02456v2)

Abstract: Phonology, the study of speech's structure and pronunciation rules, is a critical yet often overlooked component in LLM research. LLMs are widely used in various downstream applications that leverage phonology such as educational tools and poetry generation. Moreover, LLMs can potentially learn imperfect associations between orthographic and phonological forms from the training data. Thus, it is imperative to benchmark the phonological skills of LLMs. To this end, we present PhonologyBench, a novel benchmark consisting of three diagnostic tasks designed to explicitly test the phonological skills of LLMs in English: grapheme-to-phoneme conversion, syllable counting, and rhyme word generation. Despite having no access to speech data, LLMs showcased notable performance on the PhonologyBench tasks. However, we observe a significant gap of 17% and 45% on Rhyme Word Generation and Syllable counting, respectively, when compared to humans. Our findings underscore the importance of studying LLM performance on phonological tasks that inadvertently impact real-world applications. Furthermore, we encourage researchers to choose LLMs that perform well on the phonological task that is closely related to the downstream application since we find that no single model consistently outperforms the others on all the tasks.

• The paper introduces PhonologyBench, a benchmark assessing LLM phonological skills through three tasks: grapheme-to-phoneme conversion, syllable counting, and rhyme word generation.
• The paper reveals a significant performance gap between LLMs and humans, particularly in syllable counting and rhyme generation.
• The paper highlights the role of word frequency and orthography in performance and advocates for future research with phonologically enriched training data.

PhonologyBench: A New Benchmark to Assess Phonological Awareness in LLMs

Introduction to PhonologyBench

PhonologyBench represents a novel benchmark designed to rigorously evaluate the phonological skills of LLMs across three diagnostic tasks in English: grapheme-to-phoneme conversion, syllable counting, and rhyme word generation. This benchmark emerges in response to the wide application of LLMs in text-based tasks that inherently require an understanding of both written and spoken language forms, such as poetry generation and educational tools. Despite their extensive training on textual data, LLMs' capabilities in phonological tasks, which are crucial for numerous real-world applications, remain underexplored.

Methodology and Task Design

PhonologyBench introduces three tasks, each serving to test a different aspect of phonological awareness:

1. Grapheme-to-Phoneme Conversion: Evaluates a model's ability to translate written language into phonetic script.
2. Syllable Counting: Examines how accurately a model can enumerate syllables in a sentence.
3. Rhyme Word Generation: Tests a model's proficiency in identifying words that rhyme with a given word.

The benchmark encompasses a dataset with 4,000 data points spread across these tasks, providing a comprehensive framework for understanding how well various LLMs grasp phonological concepts.

Evaluation Across Six LLMs

The paper evaluates the performance of six LLMs: three closed-source models (GPT-4, Claude-3-Sonnet, and GPT-3.5-Turbo) and three open-source models (LLaMA-2-13B-Chat, Mistral-7B, and Mixtral-8X7B) on the PhonologyBench tasks. This evaluation highlights the existence of a performance gap between human capabilities and that of the LLMs, with significant variances observed in tasks like rhyme word generation and syllable counting. It is noted that no single model consistently outperforms others across all tasks, underscoring the necessity for a careful selection of LLMs based on the phonological demands of specific downstream applications.

Insights and Implications

The findings from PhonologyBench underline several critical insights:

• Performance Gap and Task Difficulty: There is a noticeable performance gap between LLMs and humans, especially prominent in syllable counting and rhyme word generation. This gap reveals the inherent difficulty LLMs face in understanding complex phonological tasks without explicit training on speech data.
• Impact of Word Frequency and Orthography: The paper sheds light on the influence of word frequency and the role of orthography in LLM performance on phonological tasks. High-frequency words and those preserved during tokenization tend to yield better results than their counterparts.
• Complexity and Real-World Application: The variance in performance across different tasks implicates the complexity of phonological understanding and its significant impact on the practical utility of LLMs in real-world applications.

Future Directions

PhonologyBench opens avenues for future research focused on improving the phonological capabilities of LLMs. Proposed directions include augmenting LLM training with phonologically rich data and exploring new models specifically designed to understand and generate phonetic and phonological patterns. Furthermore, the distinct performance patterns observed across models highlight the potential for tailored model selection and optimization based on the phonological requirements of specific applications.

Conclusion

PhonologyBench contributes significantly to our understanding of LLMs' phonological skills, offering a robust benchmark for comparative assessments. The insights gained from this research not only reveal existing limitations but also chart pathways for future developments aimed at enhancing the phonological reasoning capabilities of LLMs, thereby broadening their applicability in linguistically sophisticated domains.