PhonologyBench: Evaluating Text-based Phonological Reasoning

• PhonologyBench is a benchmark for evaluating English phonological competence in LLMs through text-based tasks.
• It includes three tasks—grapheme-to-phoneme conversion, syllable counting, and rhyme generation—with performance measured via exact match and success rate.
• Findings indicate that while LLMs show notable phonological reasoning, they lag behind human performance, especially on syllable counting.

PhonologyBench is an English phonology benchmark for LLMs that evaluates whether text-only systems can handle sound-structured linguistic tasks despite having no access to speech. It was introduced to make phonological competence an explicit object of LLM evaluation through three diagnostic tasks—grapheme-to-phoneme conversion, syllable counting, and rhyme word generation—and it has since also served as a standardized testbed for prompting studies on latent phonological reasoning in text-based models (Suvarna et al., 2024, Jang et al., 22 Jul 2025).

1. Definition and research motivation

PhonologyBench was introduced against the background that phonology is “a critical yet often overlooked component in LLM research,” even though downstream uses such as educational tools, poetry generation, and speech-adjacent applications depend on pronunciation-sensitive behavior (Suvarna et al., 2024). Its central premise is that LLMs trained only on orthography may still acquire “imperfect associations between orthographic and phonological forms” from text corpora, and that these associations should be measured directly rather than inferred from broader language benchmarks (Suvarna et al., 2024).

The benchmark is designed specifically for English phonology in a text-only setting. In a later study, it is described as “the central evaluation suite used to probe phonological reasoning in text-only LLMs,” functioning simultaneously as a diagnostic of latent phonological knowledge and a testbed for comparing prompting methods without modifying the dataset itself (Jang et al., 22 Jul 2025). That later framing is significant because it recasts PhonologyBench from a one-time evaluation suite into a reusable benchmark for controlled experimentation on phonological competence in LLMs.

2. Task structure and phonological representation

PhonologyBench targets three core tasks: Rhyme Word Generation, Grapheme-to-Phoneme (G2P) Conversion, and Syllable Counting (Suvarna et al., 2024, Jang et al., 22 Jul 2025). These were selected because they require non-trivial access to English phonology rather than simple lexical semantics. In the later formalization, they are associated with stress, syllable structure, phonotactics, and phoneme–grapheme mapping, and can be expressed as text-based mappings over orthographic, phonemic, or sentence inputs (Jang et al., 22 Jul 2025).

For G2P, the task is to map an orthographic string to an IPA phoneme sequence, formalized as $f_{\text{g2p}}: G^* \to P^*$. For syllable counting, the task is to map a sentence to an integer, formalized as $f_{\text{syll}}: W^* \to \mathbb{N}$. For rhyme generation, the task is to produce words whose rhyme nucleus plus coda match the target, with evaluation against a gold rhyme set derived from pronunciation dictionaries (Jang et al., 22 Jul 2025).

The phonological representation is explicitly IPA-based, using General American English or American English conventions. The benchmark is monolingual, and later documentation emphasizes that its gold phonological forms are dictionary-based, not guessed: IPA pronunciations are used for G2P, rhyme sets are phonologically defined from those pronunciations, and syllable counts are derived from phonemic representations (Jang et al., 22 Jul 2025). In this design, rhyme is not treated as orthographic resemblance, and syllables are linked to the definition that “a syllable is a unit of pronunciation having exactly one vowel sound” (Jang et al., 22 Jul 2025).

3. Dataset composition and coverage

The original benchmark paper describes PhonologyBench as containing “about 4,000 items,” with approximately 3,000 words for G2P, 1,000 sentences for syllable counting, and 300 words for rhyme generation (Suvarna et al., 2024). A later paper that adopts PhonologyBench “as is” reports more granular counts and category splits, while noting a minor inconsistency in one table’s high/low labeling for G2P (Jang et al., 22 Jul 2025).

Task	Original description	Later adopted report
G2P conversion	3,000 words	3,126 words total
Rhyme word generation	300 words	309 words total
Syllable counting	1,000 sentences	993 sentences

The later report further divides the dataset into frequency-based subsets. For G2P, it reports 2,084 high-frequency and 1,042 low-frequency words. For rhyme generation, it reports 199 common and 110 rare target words, with each target word associated with a gold reference set of around 1,200 phonologically-valid rhymes. For syllable counting, it reports 993 sentences (Jang et al., 22 Jul 2025). The original paper also includes a tokenization-based split inside high-frequency G2P words, distinguishing whole-word tokens from split-word tokens under OpenAI’s tiktoken tokenizer (Suvarna et al., 2024).

The benchmark is explicitly monolingual: American English. Its underlying resources are pronunciation dictionaries and related corpora, and later documentation states that the benchmark does not directly annotate feature matrices, allophones, or detailed rule systems, although such knowledge is implicitly required for high performance (Jang et al., 22 Jul 2025).

4. Evaluation protocol and original benchmark findings

PhonologyBench uses task-specific metrics that make phonological correctness explicit. For G2P conversion, the metric is Exact Match (EM) accuracy over full IPA transcriptions. For syllable counting, the metric is Exact Match accuracy over the predicted sentence-level integer. For rhyme word generation, the metric is Success Rate (SR), defined for each target word $w$ as

$$\text{SR}(w) = \frac{|R_{\text{model}}(w) \cap R_{\text{gold}}(w)|}{|R_{\text{model}}(w)|},$$

with overall SR computed by averaging over target words (Jang et al., 22 Jul 2025).

The original paper evaluates six LLMs in zero-shot settings: GPT‑4, GPT‑3.5‑Turbo, Claude‑3‑Sonnet, LLaMA‑2‑13B‑Chat, Mistral‑7B‑Instruct, and Mixtral‑8x7B‑Instruct, alongside a dedicated g2pE baseline for G2P and human baselines for rhyme and syllable counting (Suvarna et al., 2024). The main result is that LLMs show “notable performance” despite having no speech input, but remain well below humans or specialized systems on the hardest tasks (Suvarna et al., 2024).

On G2P, the g2pE baseline reaches 62.4% accuracy on high-frequency words and 52.8% on low-frequency words. Among LLMs, Claude‑3‑Sonnet reaches 52.7% / 40.2%, GPT‑4 reaches 51.9% / 38.1%, and GPT‑3.5‑Turbo reaches 47.6% / 34.4%, while smaller open models lag far behind (Suvarna et al., 2024). The benchmark also reveals a tokenization effect: high-frequency words preserved as single tokens are transcribed more accurately than words split into subword pieces (Suvarna et al., 2024).

On syllable counting, the benchmark reports a large human–model gap. Humans reach 90.0% overall accuracy, while Claude‑3‑Sonnet reaches 55.3%, GPT‑4 23.3%, GPT‑3.5‑Turbo 19.6%, and the open models cluster near or below a simple vowel-count baseline (Suvarna et al., 2024). The paper highlights this as an approximately 45% gap relative to human performance (Suvarna et al., 2024).

On rhyme generation, humans reach 86.4% SR on common words and 60.4% on rare words. GPT‑4 is the strongest model at 69.1% / 46.1%, followed by GPT‑3.5‑Turbo at 66.5% / 42.7% and Claude‑3‑Sonnet at 62.4% / 39.6% (Suvarna et al., 2024). The paper summarizes this as an approximately 17% gap to humans on rhyme word generation and emphasizes that no single model consistently outperforms the others on all tasks, making model choice application-specific (Suvarna et al., 2024).

5. Reuse in prompting research and the P-CoT results

A later paper, “P-CoT: A Pedagogically-motivated Participatory Chain-of-Thought Prompting for Phonological Reasoning in LLMs,” reuses PhonologyBench unchanged as a standardized external benchmark and broadens the experimental frame to 12 LLMs, including open models such as Llama‑3.3‑70B‑Instruct, Qwen2.5‑72B‑Instruct, and gemma‑2‑27b‑it, and closed models such as gpt‑4o, Claude 3.5 Sonnet, and Claude 3.5 Haiku (Jang et al., 22 Jul 2025).

That study compares zero-shot, few-shot, and P-CoT prompting. P-CoT is a prompt-only method that restructures the same example content used in few-shot learning into a dialogic teacher–student chain-of-thought with explicit scaffolding and discovery steps. The paper states that P-CoT is “not additional fine-tuning or external tools; it’s purely prompt design” (Jang et al., 22 Jul 2025).

Across all three PhonologyBench tasks, the later study reports that few-shot learning offers inconsistent gains, whereas P-CoT consistently enhances performance (Jang et al., 22 Jul 2025). On rhyme generation, weaker open models show gains of roughly 50 percentage points in common-word SR, and strong proprietary models approach human performance. On G2P, the paper reports statistically significant improvements over baseline as measured by Mann–Whitney U tests, with $p = 5.72 \times 10^{-6}$ for high-frequency words and $p = 6.44 \times 10^{-12}$ for low-frequency words (Jang et al., 22 Jul 2025). On syllable counting, P-CoT sharply reduces the high-error tail, decreasing the “4+ syllables off” category and increasing exact matches (Jang et al., 22 Jul 2025).

In that later interpretation, PhonologyBench provides evidence that LLMs encode latent phonological abilities that are not reliably deployed under standard prompting. The benchmark thus becomes not only an evaluative instrument but also an experimental substrate for studying how prompt structure modulates phonological reasoning (Jang et al., 22 Jul 2025).

6. Position in the benchmark landscape and stated limitations

PhonologyBench occupies a specific niche in the larger evaluation ecosystem. In its own positioning, later work contrasts it with task-specific phonology-related systems such as DeepRapper for rap lyric generation and TwistList for tongue-twister generation, arguing that PhonologyBench offers a multi-task, unified test of phonological reasoning in text-only LLMs (Jang et al., 22 Jul 2025). It is also distinguished from broader linguistic benchmarks that concentrate more heavily on syntax, semantics, morphology, or discourse (Jang et al., 22 Jul 2025).

At the same time, subsequent work has clarified the limits of this scope. The Chinese benchmark “Phun-Bench” explicitly contrasts itself with benchmarks such as PhonologyBench, arguing that existing phonology evaluations are “either solvable through rote memorization or intertwined with other abilities,” and proposing instead a benchmark organized around Homophony, Rhyme, and Phonetic Similarity in Chinese (Yue et al., 5 Jun 2026). Speech-centered benchmarks such as “The Perceptimatic English Benchmark for Speech Perception Models” and “PRiSM: Benchmarking Phone Realization in Speech Models” address phonetic discrimination and phone realization from audio rather than text-only LLM reasoning, which underscores that PhonologyBench is a benchmark of text-mediated phonological competence, not acoustic phonology or speech perception (Millet et al., 2020, Bharadwaj et al., 20 Jan 2026).

The limitations stated around PhonologyBench are correspondingly narrow but consequential. It is limited to American English; it covers only three tasks; it does not test cross-linguistic phonological phenomena such as tone, rich morphology, or different syllable structures; and it omits areas such as stress placement prediction, allophony, assimilation, coarticulation, and phonotactic acceptability judgments (Jang et al., 22 Jul 2025). The original paper likewise notes that its rhyme gold sets, g2pE-based syllable counts, and text-only setup all constrain what is being measured (Suvarna et al., 2024). A consistent implication across the later literature is that PhonologyBench is best understood as a focused benchmark for segmental and prosodic reasoning that can be expressed in text, rather than a comprehensive benchmark for phonology as a whole (Jang et al., 22 Jul 2025).