CC-G2PnP: Streaming Grapheme-to-Phoneme Prosody Model

• The paper introduces CC-G2PnP, a streaming grapheme-to-phoneme-and-prosody model that leverages a Conformer-CTC architecture with minimum look-ahead to stabilize predictions.
• It employs self-conditioned CTC with intermediate losses over a unified label set to implicitly learn alignments without relying on explicit word segmentation or dictionaries.
• Integration into LLM-to-TTS pipelines reduces latency and data dependency while matching non-streaming performance in key metrics such as CER and MOS.

Searching arXiv for CC-G2PnP and related streaming Conformer-CTC work. CC-G2PnP is a streaming grapheme-to-phoneme-and-prosody model for unsegmented languages that is designed to connect a LLM and a text-to-speech system in a streaming manner. It is built on a Conformer-CTC architecture, processes input grapheme tokens chunk by chunk, and emits a single mixed sequence of phonemic and prosodic labels. Its defining characteristics are that it can run online while an LLM is still generating text, does not depend on word boundaries or dictionaries, and uses a minimum look-ahead mechanism to stabilize phoneme and prosody prediction at chunk boundaries, particularly for languages such as Japanese that lack explicit word segmentation (Shirahata et al., 19 Feb 2026).

1. Scope and problem formulation

Grapheme-to-phoneme and prosody conversion maps a grapheme sequence $\mathbf{x}$ to phonemes together with prosodic annotations. In the formulation used here, the target can be written conceptually as $(\mathbf{p}, \mathbf{r})$, or equivalently as a single interleaved label sequence $\mathbf{y} = (y_1, \dots, y_{T'})$ whose elements are either phoneme symbols or prosodic symbols. For Japanese, the prosodic labels are intonation phrase boundaries “#”, accent phrase boundaries “/”, and accent nuclei “*”. These labels encode phrase structure and accent placement, both of which strongly affect naturalness and intelligibility in Japanese TTS (Shirahata et al., 19 Feb 2026).

The model is motivated by a low-latency LLM$\rightarrow$TTS pipeline in which the LLM emits text token by token. If the TTS backend expects phonemes and prosodic labels rather than raw graphemes, then G2PnP must itself operate incrementally. A sentence-level G2PnP model applied independently to fixed chunks is inadequate in this setting because Japanese prosody depends heavily on context, including future characters and subsequent words, so chunk boundaries induce unstable predictions.

A central difficulty is that Japanese is unsegmented. Previous streaming methods that rely on explicit word boundaries are not directly applicable, because raw orthography contains no spaces and segmentation depends on morphological analysis and dictionaries. The paper therefore frames the problem as learning alignments between grapheme tokens and phoneme-plus-prosody sequences without explicit word boundaries while retaining limited look-ahead for stable streaming inference (Shirahata et al., 19 Feb 2026).

2. Architectural formulation

CC-G2PnP takes BPE tokens produced by the CALM2-7B-Chat tokenizer as input. If the token sequence is $\mathbf{x} = (x_1, x_2, \dots, x_N)$, each token is repeated with up-sampling factor $8$ before entering the encoder. This repetition is introduced to better match the finer granularity of the phoneme/prosody sequence.

The output vocabulary is a unified CTC label set containing all Japanese phoneme symbols, the prosodic symbols “#”, “/”, and “*”, and the CTC blank. The model therefore estimates $P(\mathbf{y}\mid\mathbf{x})$ for a mixed PnP sequence rather than separate phoneme and prosody streams. Prosody is not assigned to a separate prediction head; instead, the single CTC decoder learns a joint sequence in which phonemes and prosodic labels are interleaved.

The encoder is an 8-layer Conformer with hidden size 512. Each Conformer block comprises a feed-forward module, multi-head self-attention, a convolution module, and positional encodings, with the usual residual structure. If $h^{(0)}$ denotes the embedded up-sampled input, the blockwise update is written as

$h^{(l)} = \text{ConformerBlock}^{(l)}(h^{(l-1)}).$

After the final layer, a linear projection produces per-time-step logits over the unified label space (Shirahata et al., 19 Feb 2026).

The CTC objective is used to learn latent alignment between encoder time steps and the target PnP sequence. Given encoder outputs $\mathbf{H} = (h_1, \dots, h_T)$ and target sequence $(\mathbf{p}, \mathbf{r})$0,

$(\mathbf{p}, \mathbf{r})$1

with

$(\mathbf{p}, \mathbf{r})$2

and loss

$(\mathbf{p}, \mathbf{r})$3

The paper emphasizes that this implicit alignment learning eliminates any requirement for explicit grapheme-to-phoneme mapping, word segmentation, or dictionary-based alignment.

CC-G2PnP also uses self-conditioned CTC. Intermediate CTC classifiers are attached at layers 2, 4, and 6, and their predicted distributions are fed into subsequent layers as additional conditioning. The total loss is specified as

$(\mathbf{p}, \mathbf{r})$4

In the terminology of the paper, this acts as an online LLM over intermediate predictions and is intended to relax the conditional independence assumptions of standard CTC (Shirahata et al., 19 Feb 2026).

3. Streaming operation and minimum look-ahead

The streaming mechanism is based on chunk-aware processing. The up-sampled token sequence is divided into chunks of size $(\mathbf{p}, \mathbf{r})$5, and each self-attention layer can access all tokens in the current chunk together with a fixed past context of size $(\mathbf{p}, \mathbf{r})$6. This bounds latency and memory while preserving local future context inside the chunk. However, in the standard chunk-aware formulation, tokens at the end of a chunk have no future context beyond that chunk boundary, which is especially problematic for prosody.

To address this, CC-G2PnP introduces minimum look-ahead (MLA) in the first self-attention layer only. With MLA size $(\mathbf{p}, \mathbf{r})$7, every query token is allowed to attend to at most $(\mathbf{p}, \mathbf{r})$8 future tokens beyond its own chunk, so even the last token in the chunk sees at least $(\mathbf{p}, \mathbf{r})$9 future tokens. The notation used in the paper is $\mathbf{y} = (y_1, \dots, y_{T'})$0 for chunk size, $\mathbf{y} = (y_1, \dots, y_{T'})$1 for past context size, and $\mathbf{y} = (y_1, \dots, y_{T'})$2 for MLA size, with experiments using $\mathbf{y} = (y_1, \dots, y_{T'})$3 and $\mathbf{y} = (y_1, \dots, y_{T'})$4 (Shirahata et al., 19 Feb 2026).

For standard chunk-aware streaming, the allowed attention keys for a token at time index $\mathbf{y} = (y_1, \dots, y_{T'})$5 are described as

$\mathbf{y} = (y_1, \dots, y_{T'})$6

With MLA in the first layer, this becomes

$\mathbf{y} = (y_1, \dots, y_{T'})$7

Subsequent layers revert to standard chunk-aware streaming without extra future extension. The stated rationale is that once limited future context has been incorporated at the first layer, deeper layers can propagate and combine it without further increasing latency.

Inference proceeds incrementally. The LLM produces BPE tokens one by one; CC-G2PnP up-samples them, buffers until it has $\mathbf{y} = (y_1, \dots, y_{T'})$8 tokens or $\mathbf{y} = (y_1, \dots, y_{T'})$9 when MLA is used, runs the chunk through the streaming Conformer, and applies standard CTC decoding to obtain a partial PnP sequence. The start latency when combined with an LLM is given as $\rightarrow$0 for CC-G2PnP and $\rightarrow$1 for the baseline chunked Dict-DNN system, where $\rightarrow$2 is LLM time per token and the per-chunk G2PnP compute is approximately $\rightarrow$3–$\rightarrow$4 s (Shirahata et al., 19 Feb 2026).

A common misconception in this problem setting is that streaming G2PnP for Japanese can be obtained by simply chunking a sentence-level pipeline. The reported formulation suggests the opposite: naive chunking does not provide stable prosodic predictions because chunk ends lose the future context that Japanese accent and phrasing often require. The addition of even one token of guaranteed future context is presented as sufficient to materially improve stability.

4. Prosodic representation and training setup

The prosodic representation is compact and discrete. The three prosodic symbols used in training and inference are summarized below.

Symbol	Meaning
#	Intonation phrase boundary (IP)
/	Accent phrase boundary (AP)
*	Accent nucleus (AN)

These symbols are embedded directly into the same linear output sequence as phonemes. The paper gives an example of a mixed sequence of the form

$\rightarrow$5

to illustrate how accent phrase boundaries, accent nuclei, and intonation phrase boundaries are interleaved with phoneme symbols (Shirahata et al., 19 Feb 2026).

Training relies on large-scale text-only supervision. The input side consists of 14,960,911 Japanese transcriptions from ReazonSpeech and 4,802 validation examples. For each transcription, a target PnP sequence is generated by a non-streaming Dict-DNN pipeline: a MeCab-based morphological analyzer with the Unidic dictionary performs text normalization and yields word-level surface form, reading, pronunciation, and POS, and a DNN-based prosodic label predictor trained on 80,061 manually annotated prosodic label sequences produces the phoneme-plus-prosody output. CC-G2PnP is therefore trained to emulate a high-quality non-streaming system using automatically generated labels over a massive corpus (Shirahata et al., 19 Feb 2026).

Evaluation of G2PnP uses 6D-Eval, comprising 2,722 sentences from six domains—chat, interview, news, novel, practical book, and SNS—with expert-annotated phonemes and prosodic labels. For TTS subjective evaluation, a separate internal Japanese speech dataset of 207.96 hours, containing 173,987 utterances from 17 speakers and manually annotated PnP labels, is used to train a PnP-conditioned NANSY-TTS model; evaluation is conducted with a single female speaker for consistency.

The principal hyperparameters reported are 8 Conformer layers, hidden dimension 512, up-sampling factor 8, intermediate CTC losses at layers 2, 4, and 6, exponential learning-rate decay from $\rightarrow$6 to $\rightarrow$7 over 1.2M steps, dynamic batching up to 8,192 tokens per mini-batch, chunk sizes $\rightarrow$8, past context size $\rightarrow$9, and MLA sizes $\mathbf{x} = (x_1, x_2, \dots, x_N)$0. No external alignment tool is used; all alignment is learned through the CTC objective (Shirahata et al., 19 Feb 2026).

5. Baselines, metrics, and empirical findings

The baseline streaming system is a chunked version of the non-streaming Dict-DNN pipeline. Fixed-size grapheme chunks of 5, 10, or 20 tokens are processed independently by the non-streaming pipeline, yielding Dict-DNN-5, Dict-DNN-10, and Dict-DNN-20. This baseline has access to full morphological analysis and dictionary knowledge within each chunk, but no future context across chunk boundaries. Non-streaming Dict-DNN-NS and CC-G2PnP-NS are also reported for reference (Shirahata et al., 19 Feb 2026).

The evaluation metrics are Character Error Rate and Sentence Error Rate on full PnP sequences, corresponding phoneme-only CER/SER with prosodic labels removed, and normalized PnP CER/SER in which IP and AP symbols are treated as identical. The study also reports CPU processing time on Apple M4 Pro and start latency in the integrated LLM setting. Speech naturalness is measured with MOS on a 5-point scale using 15 native Japanese raters and 50 sentences from 6D-Eval.

Among streaming baselines, Dict-DNN-20 is the strongest, with PnP CER 2.28, Norm. PnP CER 1.72, Phoneme CER 0.56, and start latency $\mathbf{x} = (x_1, x_2, \dots, x_N)$1 s. The best streaming CC-G2PnP configuration is CC-G2PnP-5-1, with $\mathbf{x} = (x_1, x_2, \dots, x_N)$2, $\mathbf{x} = (x_1, x_2, \dots, x_N)$3, and $\mathbf{x} = (x_1, x_2, \dots, x_N)$4. Its reported scores are PnP CER 1.79, Norm. PnP CER 1.28, Phoneme CER 0.52, PnP SER 41.4, and start latency $\mathbf{x} = (x_1, x_2, \dots, x_N)$5 s. The non-streaming CC-G2PnP-NS model attains PnP CER 1.80 and Phoneme CER 0.48, which is described as comparable to Dict-DNN-NS. These figures support the claim that Conformer-CTC with implicit alignment can approach dictionary-based non-streaming performance without explicit segmentation while remaining streamable (Shirahata et al., 19 Feb 2026).

The effect of MLA is reported as consistent and substantial. For $\mathbf{x} = (x_1, x_2, \dots, x_N)$6, PnP CER improves from 2.44 with $\mathbf{x} = (x_1, x_2, \dots, x_N)$7 to 1.90 with $\mathbf{x} = (x_1, x_2, \dots, x_N)$8 and 1.85 with $\mathbf{x} = (x_1, x_2, \dots, x_N)$9. For $8$0, it improves from 2.01 with $8$1 to 1.79 with $8$2 and 1.80 with $8$3. This suggests that one-token minimal look-ahead captures most of the available benefit, while larger MLA yields diminishing returns.

The subjective TTS results show the same pattern. MOS values for streaming baselines are 2.73 ± 0.11 for Dict-DNN-5 and 3.35 ± 0.11 for Dict-DNN-10. CC-G2PnP-5-0 reaches 3.81 ± 0.10, and CC-G2PnP-5-1 reaches 4.02 ± 0.09. The non-streaming systems report 4.07 ± 0.09 for Dict-DNN-NS, 4.02 ± 0.09 for CC-G2PnP-NS, and 4.16 ± 0.07 for ground-truth labels. Within the reported setup, the best streaming CC-G2PnP condition is therefore close to the non-streaming systems and the ground-truth-label condition in perceived reading and prosodic naturalness (Shirahata et al., 19 Feb 2026).

The data-scaling experiment further shows that performance improves markedly with training set size. For CC-G2PnP-5-1, 1% of training data yields PnP CER 4.55 and Phoneme CER 1.97; 10% yields 2.34 and 0.71; and 100% yields 1.79 and 0.52. A plausible implication is that the architecture trades explicit lexical resources at inference time for large-scale auto-labeled text supervision during training.

6. Role in LLM-TTS pipelines, limitations, and open directions

Within the paper’s system view, CC-G2PnP functions as the bridge between incremental text generation and speech synthesis. An ASR or audio front-end provides user speech; an LLM such as CALM2-7B-Chat generates text incrementally; CC-G2PnP converts the streamed BPE tokens into phoneme and prosodic labels; and a PnP-conditioned TTS model, instantiated here with NANSY-TTS, synthesizes audio as the labels arrive (Shirahata et al., 19 Feb 2026).

Two properties are particularly consequential for this role. First, CC-G2PnP operates on the same tokenizer as the LLM, which removes lexical mismatch between text generation and downstream pronunciation/prosody prediction. Second, it externalizes explicit prosodic structure, reducing the burden on the TTS model to infer accent and phrasing directly from graphemes. The paper states that this makes training feasible with much less paired data and improves stability and consistency, especially in Japanese.

The limitations are explicit. The system depends heavily on large amounts of labeled G2PnP data, although this is partially mitigated by auto-labeling large text-only corpora with Dict-DNN. Empirical validation is restricted to Japanese; extension to other unsegmented languages such as Chinese or Thai is proposed but not demonstrated. The prosodic inventory is limited to IP, AP, and AN, so more nuanced prosodic dimensions such as speaking style, emphasis, and emotion are outside the present scope. The subjective experiments use a non-streaming TTS model, so end-to-end streaming behavior of CC-G2PnP plus TTS is inferred rather than directly evaluated (Shirahata et al., 19 Feb 2026).

The future directions named in the paper are correspondingly targeted: integrating external knowledge such as dictionaries or LLM priors to improve rare-word pronunciation and reduce data requirements, extending the framework to more languages and richer prosodic representations, and developing tighter LLM integration so that the LLM can influence prosody more directly. Taken together, these directions frame CC-G2PnP as a modular streaming front end for pronunciation and prosody in low-latency spoken dialogue systems rather than as a complete end-to-end speech generation model.