Phoneme Addition in Speech Processing

• Phoneme addition is a task-dependent intervention in speech processing that incorporates extra phonetic representations or composite units to enhance pronunciation sensitivity.
• It encompasses various implementations, including ASR error correction with phoneme-augmented encoders, vocabulary expansion in phoneme recognition, and phoneme inventory enlargement in TTS fine-tuning.
• Empirical studies demonstrate reductions in word error rates and improvements in naturalness, while also highlighting challenges like vocabulary sparsity and effective transfer of newly added phonemes.

Phoneme addition is a task-dependent intervention in speech and language processing in which a model is supplied with additional phoneme-structured information beyond its original representation or inventory. In recent arXiv literature, the term denotes at least three distinct operations: the explicit incorporation of phonetic representations alongside wordpiece embeddings in context-aware ASR error correction, the expansion of a phoneme recognizer’s output vocabulary by adding frequent phoneme bigrams, and the enlargement of a pre-trained TTS model’s phoneme inventory during fine-tuning by appending unseen target phonemes (He et al., 31 May 2025, Park et al., 2022, Murata et al., 18 Jun 2026). This suggests a broader technical theme: phoneme addition is not a single algorithm, but a family of mechanisms for increasing pronunciation sensitivity, representational granularity, or symbol coverage.

1. Terminological scope and formal definitions

The literature uses “phoneme addition” in different but structurally related senses. In "PMF-CEC: Phoneme-augmented Multimodal Fusion for Context-aware ASR Error Correction with Error-specific Selective Decoding" (He et al., 31 May 2025), phoneme addition refers to the explicit incorporation of phonetic, phoneme-level representations alongside ordinary wordpiece embeddings so that an ASR post-processor can distinguish rare or out-of-vocabulary terms that sound alike. In "German Phoneme Recognition with Text-to-Phoneme Data Augmentation" (Park et al., 2022), the analogous operation is vocabulary augmentation: the model’s basic phoneme vocabulary is extended with selected phoneme bigrams. In "Exploring Pre-training Benefits on Phoneme Addition through Fine-tuning in Speech Synthesis" (Murata et al., 18 Jun 2026), phoneme addition is defined as the expansion of a pre-trained model’s phoneme inventory during fine-tuning.

Paper	Task	Meaning of phoneme addition
(He et al., 31 May 2025)	Context-aware ASR error correction	Add phoneme-level representations alongside wordpiece embeddings
(Park et al., 2022)	German phoneme recognition	Add selected phoneme bigrams to the output vocabulary
(Murata et al., 18 Jun 2026)	Transfer-learning TTS	Add unseen target phonemes to the pre-trained phoneme inventory

The TTS formulation is explicit. If a pre-trained model has phoneme inventory $V_{\text{pre}}$ and a set of new target phonemes $\Delta V$ is introduced during fine-tuning, then phoneme addition yields

$V' = V_{\text{pre}} \cup \Delta V.$

The model must then generate high-quality speech for every symbol in $V'$ (Murata et al., 18 Jun 2026).

The ASR formulation is representational rather than inventory-based. If the ASR output is

$S = (s_1, s_2, \dots, s_m),$

and each $s_i$ is mapped to a phoneme sequence

$P_s = (p_1^i, p_2^i, \dots, p_{q_i}^i),$

then the phoneme encoder produces a contextual phoneme representation

$E^{(P_s)} = \text{XPhoneBERT}(TE(P_s) + PE(P_s)),$

with shared hidden dimension $d_h=768$ and phoneme vocabulary size $d_{vp}=1\,960$ (He et al., 31 May 2025).

The German phoneme-recognition formulation is token-based. Let $\Delta V$0 be the set of atomic phonemes, with $\Delta V$1 in the baseline. A phoneme bigram is any ordered pair $\Delta V$2 occurring contiguously in the corpus, and the model extends its output vocabulary by inserting new tokens for selected frequent bigrams (Park et al., 2022).

2. Phoneme addition as multimodal fusion in ASR post-processing

In PMF-CEC, phoneme addition is motivated by a specific failure mode of end-to-end ASR: rare words such as proper names and technical terms are often misrecognized as more frequent words with similar acoustic realizations. The preceding ED-CEC method already leveraged contextual information such as named entities and technical terms, but its accuracy remained low when rare words had similar pronunciations but different spellings. The phoneme-augmented extension addresses this homophone problem by introducing a phoneme encoder in parallel with the wordpiece encoder, thereby supplying the correction model with a pronunciation-aware signal (He et al., 31 May 2025).

The phoneme pathway begins by converting each word’s phoneme string into a sequence of discrete tokens and embedding it with a pretrained Transformer, XPhoneBERT. In abbreviated notation,

$\Delta V$3

The text pathway uses a frozen BERT to embed wordpieces. The core fusion module then aligns the wordpiece encoder output $\Delta V$4 with the phoneme encoder output $\Delta V$5 through cross-attention:

$\Delta V$6

followed by residual fusion:

$\Delta V$7

The resulting $\Delta V$8 carries both textual context and phonetic cues (He et al., 31 May 2025).

Error detection is formulated over $\Delta V$9 with edit labels $V' = V_{\text{pre}} \cup \Delta V.$0:

$V' = V_{\text{pre}} \cup \Delta V.$1

Because the previous ASR error detection module suffered from overdetection, PMF-CEC introduces a retention probability mechanism. Let $V' = V_{\text{pre}} \cup \Delta V.$2 be the max softmax score for the chosen label at position $V' = V_{\text{pre}} \cup \Delta V.$3. Then

$V' = V_{\text{pre}} \cup \Delta V.$4

If $V' = V_{\text{pre}} \cup \Delta V.$5, the predicted label is overridden with the prior label, such as “Keep,” filtering low-confidence edits (He et al., 31 May 2025).

Training jointly optimizes the error-detection and context-aware correction heads. The detection loss is

$V' = V_{\text{pre}} \cup \Delta V.$6

the correction loss $V' = V_{\text{pre}} \cup \Delta V.$7 is the sum of token-generation cross-entropy and context-selection cross-entropy, and the final objective is

$V' = V_{\text{pre}} \cup \Delta V.$8

The implementation pipeline reported for contextual ASR correction is: tokenize ASR output into wordpieces plus dummy gaps; generate phoneme sequences per word and embed via XPhoneBERT; embed wordpieces via a frozen BERT; fuse both modalities by cross-attention and residual addition; predict Keep/Delete/Change with a light AED head and apply the retention probability mechanism; for tokens marked Change, invoke a small partially autoregressive CEC decoder that either generates new wordpieces or points into a rare-word list via context-attention gating; and train jointly with $V' = V_{\text{pre}} \cup \Delta V.$9 (He et al., 31 May 2025).

3. Empirical behavior in homophone-aware ASR correction

The empirical claim of PMF-CEC is not merely that phoneme information is useful in the abstract, but that it specifically improves rare-word correction under homophone ambiguity. Experiments on five public datasets—ATIS, SNIPS, LibriSpeech clean/other, DATA2, and PRLVS—showed that adding phoneme fusion to ED-CEC yielded an average relative biased word error rate reduction of $V'$0, including $V'$1 on test-clean and $V'$2 on test-other; U-WER also dropped by approximately $V'$3; and on the homophone-heavy PRLVS set, biased WER fell from $V'$4 (He et al., 31 May 2025).

The paper also reports that the proposed method maintains reasonable inference speed while outperforming other contextual biasing methods, and remains valuable compared with LLM-based methods in terms of faster inference and better robustness under large biasing lists. On a synthesized homophone test involving “tsavo” versus “savo,” PMF-CEC recovers the correct phoneme sequence where text-only methods fail. This directly anchors the phoneme-addition mechanism to its intended use case: correction of acoustically similar but orthographically distinct rare words (He et al., 31 May 2025).

Ablation results further isolate the contribution of the added phoneme pathway. Removing the phoneme encoder raises B-WER on LibriSpeech test-clean from $V'$5, which the paper presents as confirmation of the phoneme pathway’s value. Omitting the retention probability mechanism lowers AED accuracy from $V'$6 and slightly degrades WER from $V'$7. The recommended hyperparameters are the XPhoneBERT-derived phoneme vocabulary size $V'$8, retention threshold $V'$9, and detection-correction weight $S = (s_1, s_2, \dots, s_m),$0 (He et al., 31 May 2025).

These results delimit a common misconception. Phoneme addition in this setting is not a generic replacement for contextual biasing; rather, it is an augmentation that becomes especially relevant when lexical context alone cannot distinguish candidates with similar pronunciations. The reported gains are strongest precisely where homophone confusion is salient.

4. Vocabulary augmentation by phoneme bigrams in phoneme recognition

In the German phoneme recognition study by Park and Park, phoneme addition takes the form of inserting frequent phoneme bigrams into the model vocabulary (Park et al., 2022). Let $S = (s_1, s_2, \dots, s_m),$1 be the set of atomic phonemes. For a sentence with phoneme transcription

$S = (s_1, s_2, \dots, s_m),$2

the bigram multiset is

$S = (s_1, s_2, \dots, s_m),$3

Corpus frequency is defined as

$S = (s_1, s_2, \dots, s_m),$4

and the empirical probability of bigram $S = (s_1, s_2, \dots, s_m),$5 is

$S = (s_1, s_2, \dots, s_m),$6

Selection is constrained by phonological category. The phoneme set is partitioned into vowels $S = (s_1, s_2, \dots, s_m),$7 and consonants $S = (s_1, s_2, \dots, s_m),$8. For vowel-only models, the top-$S = (s_1, s_2, \dots, s_m),$9 members of $s_i$0 are chosen; for consonant-only models, the top-$s_i$1 members of $s_i$2 are chosen; and for total-all models, the top-$s_i$3 members of $s_i$4 are chosen without distinction. If $s_i$5 is sorted so that $s_i$6, then

$s_i$7

The baseline output vocabulary is $s_i$8 with $s_i$9 single phonemes; the augmented models extend this vocabulary by inserting new tokens for selected bigrams, and at scoring time bigram tokens are expanded back into two phonemes (Park et al., 2022).

The data pipeline starts from the CSS10 German speech-text corpus with $P_s = (p_1^i, p_2^i, \dots, p_{q_i}^i),$0 utterances. Epitran is used to convert each text sentence into its phoneme sequence, and the corpus is split into $P_s = (p_1^i, p_2^i, \dots, p_{q_i}^i),$1 train, $P_s = (p_1^i, p_2^i, \dots, p_{q_i}^i),$2 validation, and $P_s = (p_1^i, p_2^i, \dots, p_{q_i}^i),$3 test instances. The model is an encoder-decoder Transformer, explicitly “Attention is All You Need,” with hidden size $P_s = (p_1^i, p_2^i, \dots, p_{q_i}^i),$4, feed-forward inner dimension $P_s = (p_1^i, p_2^i, \dots, p_{q_i}^i),$5, $P_s = (p_1^i, p_2^i, \dots, p_{q_i}^i),$6 encoder layers, $P_s = (p_1^i, p_2^i, \dots, p_{q_i}^i),$7 decoder layer, and $P_s = (p_1^i, p_2^i, \dots, p_{q_i}^i),$8 attention heads. All variants are trained for $P_s = (p_1^i, p_2^i, \dots, p_{q_i}^i),$9 epochs, with checkpointing and BLEU measurement every $E^{(P_s)} = \text{XPhoneBERT}(TE(P_s) + PE(P_s)),$0 epochs, and early overfitting is observed after approximately $E^{(P_s)} = \text{XPhoneBERT}(TE(P_s) + PE(P_s)),$1 epochs (Park et al., 2022).

Evaluation uses corpus-level BLEU over phoneme tokens from $E^{(P_s)} = \text{XPhoneBERT}(TE(P_s) + PE(P_s)),$2-gram through $E^{(P_s)} = \text{XPhoneBERT}(TE(P_s) + PE(P_s)),$3-gram:

$E^{(P_s)} = \text{XPhoneBERT}(TE(P_s) + PE(P_s)),$4

where $E^{(P_s)} = \text{XPhoneBERT}(TE(P_s) + PE(P_s)),$5 and

$E^{(P_s)} = \text{XPhoneBERT}(TE(P_s) + PE(P_s)),$6

All reported BLEU scores are computed with NLTK’s corpus_bleu using uniform $E^{(P_s)} = \text{XPhoneBERT}(TE(P_s) + PE(P_s)),$7 weights (Park et al., 2022).

The best-epoch results show a sharply non-monotonic effect of phoneme addition.

Model	Best BLEU	Epoch
baseline	51.75	80
vowel30	53.33	80
const20	53.11	80
total30	28.54	70

The paper reports that vowel30 increases BLEU by $E^{(P_s)} = \text{XPhoneBERT}(TE(P_s) + PE(P_s)),$8, const20 by $E^{(P_s)} = \text{XPhoneBERT}(TE(P_s) + PE(P_s)),$9, and total30 decreases BLEU by $d_h=768$0. Definite-article accuracy over $d_h=768$1 follows the same trend: baseline $d_h=768$2, vowel30 $d_h=768$3, const20 $d_h=768$4, total30 $d_h=768$5 (Park et al., 2022).

The error analysis is important because it specifies why some forms of phoneme addition help while others hurt. Reported failure types include repetition or overfit errors, phoneme dropout, and substitution of similar phonemes such as predicting /ᴐ/ or /o/ in place of /U/. The positive explanation for vowel30 and const20 is that frequent vowel or consonant pairs become atomic tokens, are seen more reliably, and can be learned acoustically as single units. The negative explanation for total30 is that mixing vowels and consonants without phonological distinction yields many low-utility bigrams, expands the vocabulary from $d_h=768$6 to $d_h=768$7 tokens, and produces representation sparsity and decoder confusion on a training set of approximately $d_h=768$8 sentences (Park et al., 2022).

5. Phoneme inventory expansion in transfer-learning TTS

In transfer-learning TTS, phoneme addition is defined as the expansion of a pre-trained model’s phoneme inventory during fine-tuning, and the central question is whether the pre-trained ability to generate seen phonemes contributes to learning newly added ones (Murata et al., 18 Jun 2026). The acoustic model is Conformer-FastSpeech2. Let $d_h=768$9 be a phoneme sequence, $d_{vp}=1\,960$0 the learned embedding matrix for pre-trained phonemes, and $d_{vp}=1\,960$1 the remaining model parameters. To add $d_{vp}=1\,960$2 new phonemes, the embedding matrix is expanded as

$d_{vp}=1\,960$3

where $d_{vp}=1\,960$4 is randomly initialized. The acoustic model predicts mel-spectrogram frames by

$d_{vp}=1\,960$5

Training continues under the original FastSpeech2 loss, with no additional phoneme-specific term (Murata et al., 18 Jun 2026).

The paper identifies five technical challenges. New phoneme embeddings must be randomly initialized without pre-learned acoustic-linguistic priors; the model must preserve the original phonemes while learning $d_{vp}=1\,960$6; low-resource conditions provide only limited utterances containing the new symbols; fine-tuning can induce catastrophic forgetting or insufficient adaptation; and cross-lingual acoustic mismatch further complicates adaptation when $d_{vp}=1\,960$7 comes from another language, as in Japanese phonemes added to an English-pre-trained model (Murata et al., 18 Jun 2026).

Two experimental settings are used. The first is a simulation with LLM-generated phoneme-controlled corpora: pre-training on a Limited corpus of $d_{vp}=1\,960$8 IPA sentences synthesized into $d_{vp}=1\,960$9K utterances ($\Delta V$00 h) that exclude $\Delta V$01, followed by fine-tuning on subsets of a Full corpus of $\Delta V$02 IPA sentences synthesized into $\Delta V$03K utterances by the same speaker, with $\Delta V$04. The second is cross-lingual transfer from English to Japanese: pre-training on English VCTK ($\Delta V$05 h; $\Delta V$06 phonemes) and fine-tuning on the Japanese JSUT basic5000 subset while adding $\Delta V$07 Japanese-specific phonemes, again with $\Delta V$08 (Murata et al., 18 Jun 2026).

The evaluation metric for phoneme accuracy is phoneme error rate,

$\Delta V$09

where $\Delta V$10, $\Delta V$11, and $\Delta V$12 denote substitutions, deletions, and insertions in the recognized phoneme sequence. Target PER restricts these counts to occurrences of the newly added phonemes $\Delta V$13. Naturalness is measured by UTMOS, a pre-trained predictor of MOS in the range $\Delta V$14 (Murata et al., 18 Jun 2026).

The reported finding is counterintuitive but consistent across both settings. In the simulated experiments, scratch training achieves equal or lower Target PER at every data size for both plosive consonants and front vowels, while fine-tuning consistently yields higher UTMOS than scratch at all $\Delta V$15. In the English-to-Japanese transfer experiments, scratch training again yields lower Target PER for Japanese-specific phonemes at all $\Delta V$16, whereas UTMOS satisfies fine-tune $\Delta V$17 scratch in low-resource regimes with $\Delta V$18, and the gap diminishes as $\Delta V$19 grows (Murata et al., 18 Jun 2026).

The paper’s interpretation is that pre-training contributes mainly to naturalness improvement, but offers limited benefit for phoneme addition itself. Because the newly added embeddings are randomly initialized and fine-tuning must balance retention of $\Delta V$20 with acquisition of $\Delta V$21, the pre-trained model does not learn the new phonemes faster than scratch training. Suggested future directions include broader phoneme inventories during pre-training, auxiliary losses such as contrastive phoneme classification, meta-learning or continual-learning frameworks, improved initialization based on acoustic-phonetic similarity or language-universal representations, and unsupervised or semi-supervised adaptation (Murata et al., 18 Jun 2026).

6. Comparative patterns, limitations, and recurrent misconceptions

Across these studies, phoneme addition is beneficial only when its granularity and inductive bias are matched to the task. In PMF-CEC, adding phoneme representations improves discrimination among homophonous rare words because the objective is not to enlarge a symbol inventory but to inject pronunciation-aware evidence into a context-aware editor (He et al., 31 May 2025). In the German phoneme-recognition setting, adding selected vowel or consonant bigrams helps, but adding top mixed bigrams without phonological distinction can substantially degrade performance (Park et al., 2022). In transfer-learning TTS, adding unseen target phonemes during fine-tuning improves neither the data efficiency nor the accuracy of those phonemes relative to scratch training, even though it improves naturalness (Murata et al., 18 Jun 2026).

A recurrent misconception is that more phoneme structure necessarily yields better performance. The evidence is more conditional. The German study shows that vocabulary expansion can be strongly positive or strongly negative depending on how the new units are selected; total30 produces a large BLEU decrease, whereas vowel30 and const20 improve BLEU by more than one point (Park et al., 2022). The TTS study shows that pre-training on seen phonemes does not automatically transfer to newly added phonemes, because the embeddings for $\Delta V$22 are randomly initialized and must be learned under capacity and interference constraints (Murata et al., 18 Jun 2026). The ASR study shows that phoneme addition is most effective when paired with architecture and inference controls designed for its failure modes, such as cross-attention fusion and a retention probability mechanism for overdetection (He et al., 31 May 2025).

A plausible implication is that phoneme addition should be viewed less as a universal upgrade and more as a structured design choice. When the problem is homophone-sensitive post-correction, phoneme-side conditioning can reduce biased WER. When the problem is sequence modeling over phoneme tokens, carefully selected composite units may help, but indiscriminate vocabulary growth may induce sparsity. When the problem is cross-lingual TTS adaptation, inventory expansion alone is insufficient; initialization, continual-learning dynamics, and explicit phoneme-discrimination objectives may be required.

The current literature therefore presents phoneme addition as a technically precise intervention whose effects are architecture-dependent and task-dependent. Its success depends on what is being added—representations, bigrams, or new inventory symbols—how those additions are parameterized, and whether the downstream objective actually permits the model to exploit the additional phonological structure (He et al., 31 May 2025, Park et al., 2022, Murata et al., 18 Jun 2026).