CraBERT: Phoneme Encoder for Efficient TTS

• The paper introduces CraBERT, a cascade-fusion phoneme encoder that leverages frozen subword BERT to accelerate TTS pre-training and enhance speech naturalness.
• It employs dynamic time warping and a subword-phoneme alignment algorithm to effectively merge higher-level linguistic information into phoneme representations.
• Empirical results reveal that a 1-epoch pre-training setup achieves competitive MOS scores, showcasing efficient knowledge transfer for improved prosody.

CraBERT is a pre-trained phoneme encoder (PPEnc) for text-to-speech (TTS) that integrates representations from a pre-trained subword-level BERT into a phoneme-level BERT through a cascade-fusion architecture and a subword-phoneme alignment algorithm. Its stated objective is efficient phoneme-encoder pre-training: the design injects prior word- and sentence-level information into phoneme representations so that the phoneme encoder requires substantially fewer gradient updates while still producing representations suitable for improving the perceived naturalness and prosody of synthesized speech (Yang et al., 15 Jun 2026).

1. Motivation and problem setting

In modern neural TTS, the phoneme encoder converts input phoneme tokens into contextual representations that strongly influence the naturalness and prosody of the generated speech. CraBERT is motivated by three stated limitations of existing phoneme-only pre-trained encoders. First, phoneme sequences are long: a typical sentence in phoneme space is $2\text{–}3\times$ longer than in word/subword space, which makes it slow to capture word- and sentence-level phenomena from phoneme modeling alone. Second, the phoneme vocabulary is small, which weakens contextual representations and makes masked language modeling (MLM) comparatively easy. Third, multiple phonetic notations may exist for a language, requiring separate pre-training efforts per notation (Yang et al., 15 Jun 2026).

CraBERT addresses these issues by cascading already-trained subword-level BERT features into a phoneme-level BERT. The subword model is frozen, and its outputs are fused into the phoneme model. Under this arrangement, PBERT, the phoneme-level BERT used in CraBERT, is intended to spend fewer updates re-learning higher-level syntax and semantics and more updates on TTS-specific refinements.

This framing places CraBERT within a line of work that treats phoneme encoding not as an isolated sequence-modeling problem but as a transfer problem from subword language representations to phoneme-conditioned synthesis. A plausible implication is that CraBERT is less concerned with replacing language-model pre-training than with reusing it in a form compatible with phoneme-space TTS.

2. Cascade-fusion architecture

CraBERT consists of four sequential operations: a pre-trained subword BERT ingests the subword token sequence; a trainable aligner upsamples those subword embeddings to phoneme time steps; element-wise addition fuses the aligned subword representations into phoneme embeddings; and the fused sequence is processed by PBERT for contextual phoneme modeling (Yang et al., 15 Jun 2026).

Let $p = (p_1,\dots,p_M)$ denote the phoneme index sequence and $g = (g_1,\dots,g_N)$ the subword index sequence. The subword encoder output is

$$r_g = \theta_{\mathrm{DistilBERT}}(g) \in \mathbb{R}^{N \times d}.$$

An alignment module $A$ produces a mapping $A:\{1\dots N\}\to\{1\dots M\}$, and the upsampled subword representation satisfies

$\tilde r_{g,j} = r_{g,k} \quad \text{if } A(k)=j$

or an average if multiple $k$ map to the same phoneme position. Writing the upsampled matrix as $\tilde R_g \in \mathbb{R}^{M \times d}$, CraBERT forms the fused PBERT input by

$$H^{(0)}_j = E_{ph}(p_j) + \tilde r_{g,j}, \qquad j=1\dots M.$$

The phoneme-level encoder then applies a standard $p = (p_1,\dots,p_M)$0 forward pass for layers $p = (p_1,\dots,p_M)$1: $p = (p_1,\dots,p_M)$2

The architectural asymmetry is central. DistilBERT is frozen, while PBERT and the length regulator remain trainable. The paper attributes large speed-ups to this choice. The reported comparison between cascade fusion and parallel fusion is also significant: CraBERT-0e outperforms CraBERT$p = (p_1,\dots,p_M)$3-0e even without phoneme pre-training, indicating that the specific ordering and injection mechanism matter, not merely the presence of auxiliary subword features.

3. Subword-phoneme alignment by dynamic time warping

CraBERT uses a data-driven alignment procedure in two phases: construction of a character-phoneme cost matrix and dynamic-programming alignment of characters to phonemes, followed by lifting from characters to subwords (Yang et al., 15 Jun 2026).

In the first phase, a character-phoneme co-occurrence matrix

$p = (p_1,\dots,p_M)$4

is built by scanning a large corpus of word-to-phoneme pairs. For a training word with character sequence $p = (p_1,\dots,p_M)$5 and phoneme sequence $p = (p_1,\dots,p_M)$6, the update rule is

$p = (p_1,\dots,p_M)$7

where the Gaussian-decay kernel is

$p = (p_1,\dots,p_M)$8

After row-wise normalization, the final cost matrix is taken as

$p = (p_1,\dots,p_M)$9

In the second phase, for a new word of character length $g = (g_1,\dots,g_N)$0 and phoneme length $g = (g_1,\dots,g_N)$1, a dynamic-programming cost matrix $g = (g_1,\dots,g_N)$2 and back-pointer $g = (g_1,\dots,g_N)$3 are defined. Initialization uses

$g = (g_1,\dots,g_N)$4

and the recurrence is

$g = (g_1,\dots,g_N)$5

A backward pass on $g = (g_1,\dots,g_N)$6 retrieves the optimal character-phoneme path $g = (g_1,\dots,g_N)$7. Because each subword is a contiguous block of characters, this path is then lifted to a subword-phoneme alignment $g = (g_1,\dots,g_N)$8.

The alignment pipeline is summarized in the paper as three routines: PREPARE_DISTANCE_MATRIX() \to D, ALIGN_CHARS(D, c[0..I−1], p[0..J−1]) \to A_char, and LIFT_TO_SUBWORDS(A_char) \to A_subword–phoneme. Conceptually, this converts a frozen subword encoder into a source of positionally compatible priors for a phoneme encoder.

4. Pre-training objectives and masking strategy

CraBERT pre-training jointly optimizes two cross-entropy losses: masked language modeling on phonemes and phoneme-to-grapheme prediction at masked positions (Yang et al., 15 Jun 2026).

The MLM objective is

$g = (g_1,\dots,g_N)$9

The phoneme-to-grapheme objective is

$$r_g = \theta_{\mathrm{DistilBERT}}(g) \in \mathbb{R}^{N \times d}.$$0

The total loss is the simple sum

$$r_g = \theta_{\mathrm{DistilBERT}}(g) \in \mathbb{R}^{N \times d}.$$1

Masking is implemented at the phoneme level with the stated rule $$r_g = \theta_{\mathrm{DistilBERT}}(g) \in \mathbb{R}^{N \times d}.$$2 random, and $$r_g = \theta_{\mathrm{DistilBERT}}(g) \in \mathbb{R}^{N \times d}.$$3 unchanged. The model also replaces the aligned subword embedding $$r_g = \theta_{\mathrm{DistilBERT}}(g) \in \mathbb{R}^{N \times d}.$$4 by a learned mask vector at masked positions. This prevents direct copying of subword features into the prediction target and forces PBERT to model context rather than exploit a trivial alignment shortcut.

Within the paper’s framing, the two-loss setup has distinct roles. MLM supervises contextual phoneme prediction, whereas P2G links phoneme representations back to graphemic information. This suggests that CraBERT treats orthographic correspondences not as external metadata but as a supervised signal for phoneme contextualization.

5. Training configuration and computational profile

CraBERT is pre-trained on BookCorpus and English Wikipedia with maximum phoneme-sequence length $$r_g = \theta_{\mathrm{DistilBERT}}(g) \in \mathbb{R}^{N \times d}.$$5. The frozen subword encoder is DistilBERT-base-uncased. PBERT is specified as vanilla BERT$$r_g = \theta_{\mathrm{DistilBERT}}(g) \in \mathbb{R}^{N \times d}.$$6 with relative positional encodings, 6 layers, 768-dimensional hidden states, and 12 attention heads (Yang et al., 15 Jun 2026).

The optimizer is AdamW with $$r_g = \theta_{\mathrm{DistilBERT}}(g) \in \mathbb{R}^{N \times d}.$$7, $$r_g = \theta_{\mathrm{DistilBERT}}(g) \in \mathbb{R}^{N \times d}.$$8, $$r_g = \theta_{\mathrm{DistilBERT}}(g) \in \mathbb{R}^{N \times d}.$$9, and weight decay $A$0. The learning-rate schedule is linear warmup from $A$1 to $A$2 over the first $A$3 of steps, followed by linear decay. The batch size is 2,000 sequences via gradient accumulation. Training uses $A$4A100 40 GB with mixed precision (FP16/FP32). Two principal pre-training budgets are reported: CraBERT-1e with 9,000 steps, approximately one epoch, and CraBERT-10e with 90,000 steps, approximately ten epochs. The masking rate is fixed at $A$5 after a sweep.

A central computational claim is that freezing DistilBERT reduces one epoch to approximately 19 hours, compared with 27–29 hours for phoneme-only baselines. The paper further reports a pre-training speed-up of approximately $A$6 relative to MP BERT and PL BERT on the same $A$7A100 setup, while inference speed remains comparable. In practical terms, CraBERT is presented not merely as a higher-performing phoneme encoder but as a method for reducing pre-training cost by shifting representational burden to a frozen subword model.

6. Empirical findings, interpretation, and limits

The paper reports a masking-rate sweep, downstream mean-opinion-score (MOS) comparisons, and several interpretive observations derived from those experiments (Yang et al., 15 Jun 2026).

For the masking-rate sweep, all variants are pre-trained for 9,000 steps and evaluated in a VITS downstream system. The reported scores are as follows:

Mask rate	MOS $A$8 CI
15%	3.10 $A$9
30%	3.23 $A:\{1\dots N\}\to\{1\dots M\}$0
45%	3.25 $A:\{1\dots N\}\to\{1\dots M\}$1
60%	3.25 $A:\{1\dots N\}\to\{1\dots M\}$2
75%	3.29 $A:\{1\dots N\}\to\{1\dots M\}$3
90%	3.15 $A:\{1\dots N\}\to\{1\dots M\}$4

On this basis, the study fixes the masking rate at $A:\{1\dots N\}\to\{1\dots M\}$5. This is notable because it is substantially higher than conventional MLM masking rates, and it is consistent with the presence of auxiliary subword priors.

For the main MOS comparison, the paper uses 50 native listeners and 50 utterances each:

Encoder	Pre-train steps	MOS $A:\{1\dots N\}\to\{1\dots M\}$6 CI
Ground-Truth speech	—	3.60 $A:\{1\dots N\}\to\{1\dots M\}$7
VITS w/ no PPEnc	0	2.83 $A:\{1\dots N\}\to\{1\dots M\}$8
MP BERT	90 k (10 e)	3.14 $A:\{1\dots N\}\to\{1\dots M\}$9
PL BERT	90 k	3.13 $\tilde r_{g,j} = r_{g,k} \quad \text{if } A(k)=j$0
CraBERT$\tilde r_{g,j} = r_{g,k} \quad \text{if } A(k)=j$1-0e	0	2.90 $\tilde r_{g,j} = r_{g,k} \quad \text{if } A(k)=j$2
CraBERT-0e	0	3.09 $\tilde r_{g,j} = r_{g,k} \quad \text{if } A(k)=j$3
CraBERT-1e	9 k (1 e)	3.21 $\tilde r_{g,j} = r_{g,k} \quad \text{if } A(k)=j$4
CraBERT-10e	90 k	3.15 $\tilde r_{g,j} = r_{g,k} \quad \text{if } A(k)=j$5

The paper highlights three observations. First, CraBERT-1e matches or slightly outperforms MP BERT and PL BERT despite using one-tenth as many steps. Second, there is no further gain from one epoch to ten epochs, which the paper interprets as evidence that most word- and sentence-level knowledge has already been pre-fused. Third, cascade fusion is better than parallel fusion even with no phoneme pre-training.

The interpretation offered is that DistilBERT supplies word- and sentence-level context, allowing PBERT to focus its limited updates on TTS-specific phoneme nuance, including prosody and coarticulation. A common misconception would be that more phoneme pre-training should monotonically improve subjective naturalness; within this study, the comparison between CraBERT-1e and CraBERT-10e does not support that view.

The paper also states several limitations and open questions. The residual gain from PBERT beyond one epoch remains limited under the current losses, suggesting a possible mismatch between language-modeling features and the acoustic needs of TTS. Other identified directions are extension to multilingual or multi-notation phoneme sets, joint fine-tuning of DistilBERT on TTS data, and exploration of adaptive masking rates or additional prosody-aware objectives. These points delimit the current contribution: CraBERT demonstrates that cascade-fused subword priors can reduce phoneme-encoder pre-training time by an order of magnitude while preserving or exceeding the subjective naturalness achieved by the compared baselines.