XPhoneBERT: Multilingual Phoneme Transformer

• XPhoneBERT is a multilingual Transformer-based model that learns phoneme-level representations through large-scale pre-training on 330 million phoneme sentences.
• It integrates a 12-layer BERT encoder with dynamic masking and advanced alignment objectives to improve synthesis quality and cross-lingual transfer.
• Empirical evaluations show MOS improvements and significant gains in data efficiency across full and low-resource TTS settings for diverse languages.

XPhoneBERT is a class of multilingual Transformer-based pre-trained models designed to capture phonemic or phoneme-level representations for enhanced spoken language processing, particularly in text-to-speech (TTS) systems and cross-lingual tasks. It is distinguished by large-scale pre-training exclusively or jointly on phoneme sequences from a wide range of languages, integration of advanced alignment objectives, and empirical advances in both data efficiency and synthesis quality across typologically diverse languages (Nguyen et al., 2023, Nguyen et al., 2023).

1. Architectural Foundations

XPhoneBERT adopts the standard BERT-base Transformer encoder configuration but replaces word- or subword-level tokens with phoneme tokens. The architecture comprises:

• 12 Transformer encoder layers
• Hidden dimension: 768
• 12 self-attention heads
• Feed-forward dimension: 3072

Processing begins with an input state $X \in \mathbb{R}^{L \times d_\text{model}}$, with each Transformer block applying multi-head self-attention and a two-layer feed-forward network, including layer normalization and residual connections. Each head's queries, keys, and values are computed as:

$$Q_h = XW_h^Q, \quad K_h = XW_h^K, \quad V_h = XW_h^V,$$

with $$W_h^{Q,K,V} \in \mathbb{R}^{d_\text{model} \times d_k}$$, typically $d_k = d_\text{model}/H$. Scaled dot-product attention and projection are performed as in standard Transformers (Nguyen et al., 2023).

Phoneme vocabulary size is approximately 1,960 units, supporting stress marks, boundary tokens, and diacritics. Tokenization is based on whitespace-separation of segmented phoneme sequences, producing a simple yet expressive input format for cross-lingual transfer.

2. Pre-training Data and Objectives

XPhoneBERT's pre-training corpus spans 330 million phoneme-level sentences across 94 languages/locales. Sentence sources are derived from large-scale Wikipedia data (wiki40b and additional dumps), normalized and segmented, with grapheme-to-phoneme conversion performed via the CharsiuG2P toolkit. Each language's phonemic inventory and segmentation are preserved through species boundary markers and whitespace.

The pre-training objective follows the RoBERTa protocol:

• Dynamic masking of 15% of input phoneme tokens per mini-batch
• Masked Language Modeling (MLM) loss:

$$\mathcal{L}_\text{MLM} = -\sum_{i \in M} \log P(x_i | x_{[1...L]\setminus M})$$

• No next-sentence prediction (Nguyen et al., 2023)

This approach ensures the learning of rich, contextualized phoneme representations capable of supporting downstream TTS and language transfer tasks.

3. Integration with Neural TTS Pipelines

In downstream TTS, XPhoneBERT is used as the input phoneme encoder within state-of-the-art end-to-end models such as VITS. In this arrangement:

• The baseline VITS encoder (a small Transformer) is replaced directly by the 12-layer XPhoneBERT encoder.
• XPhoneBERT hidden states are linearly projected to match the posterior encoder input expected by VITS.

Fine-tuning involves:

• Freezing XPhoneBERT parameters for the first 25% of TTS training steps, then jointly training all components for the remaining 75%.
• AdamW optimizer: $\beta_1 = 0.8$, $\beta_2 = 0.99$, weight decay $= 0.01$, initial learning rate $= 2\times10^{-4}$, decayed per epoch.
• Both full-resource (e.g., 24 hours English, 18 hours Vietnamese) and low-resource (~1 hour) regimes are supported, with scaled training iterations for low-data settings (Nguyen et al., 2023).

4. Empirical Performance and Data Efficiency

XPhoneBERT delivers consistent gains in speech naturalness, prosodic richness, and spectral detail across full and low-resource TTS settings. Core empirical results for English (LJSpeech) and Vietnamese TTS are summarized below (MOS: Mean Opinion Score; MCD: Mel-cepstral distance; F0 RMSE: pitch error):

Model	MOS (↑)	MCD (↓)	F0 RMSE (↓)
English
Ground truth	4.39±0.08	0.00	0.00
VITS (full data)	4.00±0.08	7.04	377
VITS + XPhoneBERT (full)	4.14±0.07	6.63	348
VITS (5% data)	2.88±0.11	7.40	407
VITS + XPhoneBERT (5%)	3.22±0.11	7.15	383
Vietnamese
Ground truth	4.26±0.06	0.00	0.00
VITS (full data)	3.74±0.08	5.41	249
VITS + XPhoneBERT (full)	3.89±0.08	5.12	234
VITS (5% data)	1.59±0.05	6.20	291
VITS + XPhoneBERT (5%)	3.35±0.10	5.39	248

Notably, with only 5% of the training set, MOS gains are +0.34 (English) and +1.76 (Vietnamese). This establishes that large-scale multilingual phoneme pre-training yields strong data efficiency and cross-lingual robustness.

5. Broader Applications: Cross-Lingual and Robust Multimodal Representation

Extensions of the XPhoneBERT paradigm, including PhoneXL ("XPhoneBERT" in some sources), integrate phonemic transcriptions as a distinct modality for cross-lingual transfer learning (Nguyen et al., 2023). Core design features include:

• Parallel orthographic and phonemic input embeddings for each token position.
• Shared Transformer backbone (e.g., mBERT or XLM-R).
• Three unsupervised alignment losses:
  • Bidirectional local contrastive loss aligns per-token orthographic and phonemic embeddings.
  • Cross-modal masked language modeling recovers masked orthographic tokens from both modalities.
  • Code-switched MLM, leveraging token-level cross-lingual signal with small bilingual dictionaries.
• Training objective:

$$\mathcal{L} = \mathcal{L}_\text{task} + \alpha \mathcal{L}_\text{align} + \beta \mathcal{L}_\text{MLM} + \gamma \mathcal{L}_\text{multi}$$

• Gains in zero-shot token classification for script-divergent pairs: e.g., ZH→VI Named Entity Recognition (F₁ +2.34), JA→KO POS tagging (+3.12 F₁).

The principal empirical finding is that integration of phonemic information via explicit modeling and alignment substantially improves cross-lingual transfer, particularly when orthographic similarity is low but phonological correspondence is high (Nguyen et al., 2023).

6. Context within Phoneme-Based LLMs

Earlier work, such as PhonemeBERT (Sundararaman et al., 2021), introduced joint language modeling of ASR transcript and phoneme sequence for ASR robustness and phonetic-aware text understanding. The model employs shared Transformer stacks for different input modalities, multi-branch MLM losses, and demonstrates improved downstream classification (sentiment, question, intent) under high word error rates.

XPhoneBERT advances this direction by operating at substantially greater scale (330M phoneme sentences across nearly 100 languages), reframing the pre-training objective to pure phoneme-level MLM, and directly targeting TTS as the motivating application domain. Compared to models that fuse text and phoneme modalities for textual classification, XPhoneBERT establishes that large-scale phoneme-only pre-training suffices for robust phoneme sequence representation and yields generalization benefits in both TTS and cross-lingual tasks.

7. Limitations, Open Challenges, and Future Directions

Current limitations include:

• Reliance on subjective (MOS) and limited objective metrics (MCD, F0 RMSE); lacking fine-grained or listener-based prosody evaluations.
• Early freezing of XPhoneBERT parameters during fine-tuning avoids catastrophic forgetting but may impede full adaptation; exploration of adaptation modules (e.g., adapters) is warranted.
• While the model supports 94 languages/locales, empirical reporting is limited to a subset, and results on typologically distant or low-resource languages are pending.
• Real-time deployment challenges due to the size of the 12×768 model in latency-sensitive TTS engines.

A plausible implication is that XPhoneBERT's core approach—multilingual phonemic pre-training with flexible alignment—can serve as a foundation for future models that address these challenges via efficient architectural adaptations, expanded cross-lingual evaluation, and enhanced multi-modal representation learning. Potential directions include integration with audio-text pre-trained models, dynamic loss weighting, and advanced fusion strategies for orthographic–phonemic input (Nguyen et al., 2023, Nguyen et al., 2023, Sundararaman et al., 2021).