CraBERT: Efficient Phoneme Encoder Pre-Training via Cascade Fusion of Subword Representations for Text-to-Speech

Abstract: This paper introduces CraBERT, a pre-trained phoneme encoder (PPEnc) designed for efficient pre-training in text-to-speech (TTS). CraBERT employs a cascade-fusion architecture and a subword-phoneme alignment algorithm to integrate representations from a pre-trained subword-level BERT into a phoneme-level BERT. This design provides prior word- and sentence-level information, reducing the amount of pre-training required by the phoneme encoder. Subjective listening evaluations show that CraBERT achieves MOS values comparable to existing PPEncs after approximately one epoch of pre-training, whereas the baselines in our comparison are pre-trained for approximately ten epochs. These results demonstrate that CraBERT can efficiently learn representations suitable for improving the perceived naturalness and prosody of synthesized speech.

• The paper introduces a cascade-fusion architecture that integrates frozen subword-level DistilBERT with trainable phoneme-level BERT to enhance pre-training efficiency.
• It employs a dynamic time warping (DTW)-based alignment for optimal mapping of subword and phoneme sequences, reducing pre-training updates by nearly 15× while achieving competitive MOS scores.
• The method enables rapid convergence and scalable cross-linguistic adaptation, providing a cost-effective approach to improve TTS systems.

CraBERT: Efficient Cascade Fusion of Subword Representations for Phoneme Encoder Pre-Training in TTS

Introduction

CraBERT addresses the inefficiencies of phoneme encoder pre-training in text-to-speech (TTS) systems by leveraging a cascade-fusion architecture that integrates robust semantic priors from a pre-trained subword-level BERT into the phoneme encoder. Traditional BERT-based phoneme encoders (PPEncs), such as MP BERT and PL BERT, exhibit significant inefficiency due to the intrinsic limitations of phoneme-only pre-training: long sequence lengths, small vocabulary, and variable phonetic notations. The cascade-fusion approach proposed in CraBERT circumvents these challenges by conditioning phoneme representations on subword-level features, substantially reducing the required pre-training steps without sacrificing TTS quality.

Figure 1: CraBERT architecture featuring cascade fusion of fixed subword-level DistilBERT representations with trainable phoneme-level BERT, aligned via a data-driven DTW-based module for efficient pre-training.

CraBERT Architecture and Training

CraBERT employs a two-tier encoder system. The architecture contains a frozen, pre-trained DistilBERT model to encode subword representations and a phoneme-level BERT (PBERT) for phoneme tokens. The subword and phoneme sequences are aligned by a DTW-based aligner, which generates mapping indices for upsampling subword features to phoneme resolution. These upsampled subword features are combined with phoneme embeddings through element-wise addition prior to being processed by the PBERT blocks (see Figure 1).

This architecture enables the phoneme encoder to inherit rich contextual and semantic features from the subword BERT, transforming the MLM and phoneme-to-grapheme (P2G) prediction objectives. Importantly, only the PBERT and task-specific heads are updated during pre-training; DistilBERT remains frozen, minimizing computational overhead and irreducible redundancy.

The MLM objective exclusively applies whole-word dynamic masking to phoneme tokens (not the subword tokens), with masking strategies that closely follow the BERT convention but replace masked subword representations with learnable embeddings. The P2G objective predicts only the subword tokens aligned to masked phoneme tokens, avoiding inefficiency observed in global P2G targets. Weight tying and transfer initialization further stabilize and accelerate the pre-training process.

Data-Driven Subword-Phoneme Alignment

The alignment of subword and phoneme sequences is resolved via a data-driven algorithm based on dynamic time warping (DTW). Using a character-phoneme co-occurrence probability matrix trained from large text corpora, the algorithm determines optimal alignment paths between subword and phoneme sequences for each input. The method proceeds by building and normalizing the probability matrix, then constructing a cost and tracking matrix to derive the optimal alignment via forward and backward iterations. Each phoneme is then mapped to its corresponding subword through a letter bridge, supporting arbitrary subword tokenization schemes.

This procedure does not require human-labeled alignment, word-level tokenization, or rule-based heuristics; it generalizes across phoneme inventories and tokenization paradigms, preserving the information content in both subword and phoneme streams. It is a scalable and portable solution for the alignment problem.

Experimental Results

CraBERT is rigorously compared against MP BERT and PL BERT using a consistent backbone (12-layer BERT\textsubscript{BASE}), identical downstream TTS (VITS), corpus (BookCorpus + Wikipedia), and evaluation (multi-speaker LibriTTS-R, MOS test with 50 native listeners). The key results are:

• Subjective TTS quality (MOS): CraBERT, pretrained for only one epoch (9000 steps), achieves MOS values on par with MP BERT and PL BERT, each of which require ten epochs (90,000 steps). No further improvement of MOS is observed when CraBERT is trained for ten epochs, indicating rapid convergence and representation sufficiency.
• Pre-training efficiency: CraBERT exhibits a per-epoch training time advantage, and for matched downstream quality, it reduces total pre-training time by $14\times$--$15\times$ compared to the baselines. This is primarily attributable to freezing DistilBERT and to the lower number of total pre-training updates required for convergence.
• Masking rate: Conventional 15% masking, as in BERT, is suboptimal for CraBERT. With subword features providing strong conditional priors, higher masking rates up to 75% yield monotonic improvement in subjective MOS, after which quality degrades.

The results also demonstrate that parallel fusion of subword and phoneme representations (without cascade conditioning) is inferior to the cascade-fusion approach of CraBERT, further reinforcing the benefit of the proposed conditioning and training paradigm.

Implications and Future Directions

CraBERT reshapes the landscape of PPEnc design by decoupling word/sentence-level semantic pre-training from phoneme-level learning, allowing for rapid and computationally efficient training without degrading synthesis quality. The paradigm can be generalized to other language pairs, tokenization methods, or prosody modeling tasks, and the subword-to-phoneme alignment mechanism supports robust cross-linguistic adaptation. The method's empirical findings suggest that future research should focus on more precise delineation of what TTS-relevant phoneme-level features are not captured via language modeling alone, possibly leading to improved objectives or architectures.

Further, CraBERT’s architecture opens the possibility of plug-and-play enhancement by replacing DistilBERT with more powerful or domain-conditioned pre-trained subword encoders, and it provides a substrate for modular learning in multi-task or multi-lingual TTS systems. The improved sample efficiency also reduces the environmental and financial cost of TTS research and deployment, which is a practical advantage for the field.

Conclusion

CraBERT introduces an efficient, cascade-fusion-based approach to phoneme encoder pre-training for TTS, aligning and integrating frozen subword-level semantic priors with trainable phoneme representations. Subjective evaluations confirm that CraBERT dramatically reduces the required pre-training compute while achieving TTS quality comparable to state-of-the-art baselines. The architectural, algorithmic, and experimental contributions of CraBERT position it as a strong and generalizable methodology for future TTS and related speech-text modeling tasks.