PhonemeBERT: Phoneme-Aware Transformer Models

• PhonemeBERT is a family of Transformer models that integrate phoneme-level representations with contextual encoding to enhance speech and language processing.
• Different architectures merge text, phoneme, and acoustic streams using objectives like MLM and CTC to achieve improved robustness and performance in ASR, TTS, and polyphone disambiguation.
• Empirical results demonstrate significant gains in noise resistance, speaker recognition, and naturalness in TTS, highlighting its practical impact on speech technology.

PhonemeBERT refers to a family of Transformer-based models that incorporate phonemic representations into the language modeling pipeline. These systems combine phoneme-level information—either from raw acoustic features, forced-aligned labels, or phoneme token sequences—with contextual Transformer encoders, most frequently using BERT or its derivatives, to produce phonetically aware or robust contextual representations. Various architectures under the PhonemeBERT moniker have been developed to target distinct applications: robust language understanding under automatic speech recognition (ASR) noise, text-to-speech (TTS) synthesis, polyphone disambiguation, and multimodal spoken-language processing. Approaches differ in whether phoneme models are paired with word/subword streams, operate directly on acoustic features, or rely exclusively on phonemic tokenization, but all share the central goal of leveraging the bidirectional self-attention of Transformer encoders to produce high-level representations with explicit or implicit phonological structure.

1. Model Architectures

PhonemeBERT models typically instantiate variants of "BERT-base" architectures—12 Transformer encoder layers, hidden size 768, feed-forward size 3072, and 12 attention heads—but exhibit diversity in input organization:

• Joint Word/Phoneme Transformer: "Phoneme-BERT" (Sundararaman et al., 2021) concatenates ASR-derived subword tokens and phoneme sequence tokens:

$[CLS], w_1, ..., w_N, [SEP], p_1, ..., p_M, [SEP]$

with each token parameterized by token, positional, and modality-type embeddings. All input streams are processed through the same stack of 12 Transformer layers.

• Acoustic Feature-Based Encoder: "BERTphone" [editor's note: also described as PhonemeBERT in the text; (Ling et al., 2019)] operates on sequences of mean-normalized, 40-dimensional MFCC frames (stacked by 3), linearly projected to the embedding dimension. Learned positional embeddings are added. This purely acoustic interface distinguishes it from text-only token-based variants.
• Multimodal and Polyphone Encoding: g2pW (Chen et al., 2022) applies BERT to Mandarin grapheme sequences with explicit positional targeting for polyphonic characters, employing a unified encoder and a conditioning network for softmax weighting at the output.
• Text-to-Speech Variants: Multiple architectures including PL-BERT (Li et al., 2023), Mixed-Phoneme BERT (Zhang et al., 2022), PnG-BERT (Jia et al., 2021), and XPhoneBERT (Nguyen et al., 2023) process phoneme-level sequences (pure, or in combination with sup-phoneme, subword, or grapheme representations), with minor modifications to input embedding structure and head architecture for auxiliary prediction tasks.

2. Pretraining and Learning Objectives

PhonemeBERT models apply a range of pretraining objectives:

• Masked Language Modeling (MLM): The most common objective, applied at the token-level for both words and phonemes (Sundararaman et al., 2021, Jia et al., 2021, Li et al., 2023, Nguyen et al., 2023). Masking strategies often include 80% [MASK], 10% random, 10% unchanged token replacements. For phoneme+grapheme models, word-level consistent masking enforces contemporaneous masking of all tokens aligned to the same word (Jia et al., 2021).
• Span-Masking Acoustics: For acoustic models, BERTphone (Ling et al., 2019) zero-masks random short spans of input frames and minimizes an L1 reconstruction loss over all positions:

$$L_{recons} = \frac{1}{T} \sum_{t=1}^{T} | x_t - FFN(z_t) |$$

where $z_t$ is the Transformer output for frame $t$.

• Connectionist Temporal Classification (CTC): BERTphone jointly optimizes a frame-level phoneme labeling loss:

$L_{CTC} = -\log P(Y|X)$

where $Y$ is the unaligned phoneme sequence and $P(Y|X)$ is computed by summing over all valid CTC label paths.

• Joint or Multi-Task Losses: Many models optimize a weighted sum of losses, e.g.,

$$L = \lambda \cdot \sqrt{T} \cdot L_{recons} + (1-\lambda) L_{CTC}$$

in BERTphone, or $L_{total} = L_{phoneme} + \beta L_{POS}$ in g2pW (Chen et al., 2022), where $\beta$ weights POS tag supervision.

• Auxiliary Tasks: PL-BERT (Li et al., 2023) includes a secondary phoneme-to-grapheme head, and uses a joint MLM+P2G loss. Mixed-Phoneme BERT merges phoneme and BPE-derived sup-phoneme prediction (Zhang et al., 2022).

3. Applications and Downstream Usage

PhonemeBERT models demonstrate state-of-the-art or competitive results in several domains:

Model	Task(s)	Notable Results/Claims
BERTphone	Speaker/Language Rec.	$C_{avg}=6.16$ (3s LRE07), 18% EER reduction (Ling et al., 2019)
PhonemeBERT (orig)	ASR-robust NLU	+3.6% accuracy (TREC-50), +4.7% macro-F1 (ATIS) at WER > 30% (Sundararaman et al., 2021)
PnG BERT	Neural TTS	MOS 4.47$\pm$0.05, human parity in SxS pref. (Jia et al., 2021)
XPhoneBERT	Multilingual TTS Encode	MOS gain up to +1.8 (5% data), MCD/F0-RMSE drops (Nguyen et al., 2023)
Mixed-Phoneme BERT	TTS (+sup-phonemes)	+0.30 CMOS, $3\times$ faster than PnG BERT (Zhang et al., 2022)
g2pW	Mandarin Polyphone	99.08% phoneme acc. (CPP), outperforms prior art (Chen et al., 2022)

In language and speaker recognition, BERTphone features yield superior $C_{avg}$ and EER on LRE07 and Fisher/VoxCeleb compared to MFCC or bottleneck DNN front-ends. For robust NLU under ASR noise, PhonemeBERT outperforms strong RoBERTa and joint-MLM baselines, with gains growing as WER increases (Sundararaman et al., 2021). In Mandarin g2p/Polyphone, g2pW demonstrates superior per-character and overall phoneme disambiguation rates (Chen et al., 2022). For TTS, PnG BERT, XPhoneBERT, PL-BERT, and Mixed-Phoneme BERT all report MOS or CMOS gains over previous pipelines, with some (e.g., XPhoneBERT) demonstrating largest improvements under low-resource regimes.

4. Phoneme Representation Strategies

Different PhonemeBERT variants exploit a range of phonemic representation and alignment schemes:

• Token-based: Most models use a vocabulary of phoneme tokens, sometimes augmented with suprasegmental features (e.g., stress/diacritics) or sup-phoneme units (merges via BPE) (Zhang et al., 2022, Nguyen et al., 2023). For languages with polyphones, softmax masking/conditioning is applied (Chen et al., 2022).
• Frame-based/Acoustic: BERTphone directly processes learned frame embeddings projected from MFCC input (Ling et al., 2019).
• Multi-stream: Joint word/phoneme, phoneme/grapheme, or phoneme/subword pipelines allow multi-modal fusion via shared self-attention (Sundararaman et al., 2021, Jia et al., 2021).
• Multilingual: XPhoneBERT pretrains over 94 locales, using a 1960-token phoneme inventory.

Preprocessing typically requires grapheme-to-phoneme conversion engines (e.g., CharsiuG2P, Phonemizer), word alignment for consistent masking, and, for acoustic models, forced phoneme alignment.

5. Empirical Results and Analysis

Empirical evaluations consistently show that incorporating phoneme-level structure, especially via MLM-style pretraining or CTC objectives, provides robustness to ASR errors, improved disambiguation (polyphones), and enhanced naturalness/prosody in TTS. In "Phoneme-BERT" (Sundararaman et al., 2021), joint pretraining with both word and phoneme streams yields up to a 5.2% absolute accuracy gain on sentiment classification (SST-5) at WER≥30%. BERTphone achieves $C_{avg}=6.16$ (EER=4.63%) on LRE07 3s—much lower than MFCC baselines—and reduces speaker EER by 18% relative (Ling et al., 2019).

In TTS, XPhoneBERT improves MOS by up to 1.76 in 5% data settings for Vietnamese ((Nguyen et al., 2023), Table 2), while Mixed-Phoneme BERT achieves CMOS parity with PnG-BERT with a $3\times$ real-time factor speedup (Zhang et al., 2022). Ablation studies find strong dependence on multimodal input, consistent whole-word masking, and auxiliary tasks (e.g., P2G loss in PL-BERT (Li et al., 2023)).

6. Limitations, Extensions, and Future Directions

Practical limitations identified include:

• Input Requirements: Many PhonemeBERT pipelines require high-quality phoneme transcriptions and alignment, which may be unavailable for some languages or practical ASR setups.
• Frozen Encoders: Freezing the pretrained encoder, while enabling plug-and-play integration, can limit adaptation to domain/task-specific acoustic or linguistic phenomena (Ling et al., 2019).
• Masking and Multitask Tuning: The tradeoff between pure CTC and span-masking loss must be tuned per downstream task (Ling et al., 2019). Incorrect weightings can degrade the useful phonetic or prosodic information learned.

Proposed extensions include:

• Adapting models for multilinguality by expanding phoneme inventories.
• Integrating prosodic or suprasegmental cues (pitch, energy) alongside phonemes.
• Joint modeling of boundary detection for better word segmentation, as motivated by findings in phoneme-level BabyLMs (Goriely et al., 4 Apr 2025).
• Curriculum learning schedules or auxiliary boundary-prediction losses may further improve phoneme-based segmentation and representation learning.
• Fine-tuning or unfreezing pretrained encoders is hypothesized to yield additional downstream gains at the expense of universality.

7. Significance in Speech and Language Technology

PhonemeBERT and related models have established a paradigm for robust, phonetically informed modeling in speech and language applications. By leveraging self-attention architectures capable of contextual fusion across phoneme, word, and auxiliary streams, these models systematically outperform traditional feature extraction or classical bottleneck approaches under conditions of ASR noise, low resource, and cross-linguality. Their use in TTS directly enhances output prosody and intelligibility, and their applicability to language identification, speaker recognition, and disambiguation tasks demonstrates broad utility for speech processing. The integration of phoneme-level modeling into Transformer pipelines is now a prevailing approach in the design of robust, interpretable, and flexible spoken language representation systems (Ling et al., 2019, Sundararaman et al., 2021, Jia et al., 2021, Li et al., 2023, Nguyen et al., 2023, Zhang et al., 2022, Chen et al., 2022, Goriely et al., 4 Apr 2025).