---
title: Taiwanese Mandarin Spoken Language Model
url: https://www.emergentmind.com/topics/taiwanese-mandarin-spoken-language-model
type: topic
---

# Taiwanese Mandarin Spoken Language Model

A Taiwanese Mandarin Spoken Language Model (SLM) is a neural architecture designed to process, generate, and understand Taiwanese Mandarin speech, typically incorporating explicit mechanisms for handling unique phonotactic, lexical, and conversational properties of the language. Recent advancements demonstrate sophisticated end-to-end models enabling real-time, multi-turn, speech-to-speech interactions, robust phonetic control, and effective disambiguation of Mandarin polyphones. Such systems combine large language model (LLM) backbones, specialized tokenization, and advanced acoustic modeling to achieve conversational fluency and high-fidelity synthesis in Taiwanese Mandarin contexts [2411.07111][2501.17790].

## 1. System Architectures

Contemporary Taiwanese Mandarin spoken language models (SLMs) exhibit highly modular, streaming-friendly system designs. The "spml-omni" spoken-LLM pipeline [2411.07111] utilizes:

- **Streaming ASR**: Segments audio every 0.3 s, producing confirmed text tokens via distilled Whisper with VAD and hallucination filtering.
- **Streaming Speech Units**: Discrete acoustic features (HuBERT units, 0.1 s chunks), providing rich subword information.
- **Interleaver**: Aligns word-level ASR outputs and speech units by timestamps, generating a unified, interleaved token sequence.
- **Decoder-only Transformer**: Initialized from LLaMA-3.1 8B, using prompt engineering for conversational and duplex control. No architectural modifications are applied; turn-taking utilizes a special $\langle$EOT$\rangle$ token.
- **Denoising Diffusion Decoder**: Converts predicted discrete units or text to mel-spectrograms.
- **HiFi-GAN Vocoder**: Final waveform synthesis for speech output.

A high-level block diagram is:

```
┌───────────────┐      ┌───────────────┐
│ Streaming ASR │      │  Streaming    │
│   (0.3 s)     │      │ Speech Units  │
└───────────────┘      └───────────────┘
         │                      │
         └──────────────┬───────┘
                        │
                ┌──────────────┐
                │ Interleaver  │
                └──────────────┘
                        │
                ┌──────────────┐
                │ Transformer  │
                └──────────────┘
                /                 \
      ┌─────────────┐      ┌─────────────┐
      │ Extract     │      │ Extract     │
      │ speech unit │      │ text token  │
      └─────────────┘      └─────────────┘
            │                    │
  ┌─────────────────┐    ┌─────────────┐
  │ Mel-spectrogram │    │ Text/Screen │
  │  Diffusion      │    │ Display     │
  └─────────────────┘    └─────────────┘
            │
     ┌──────────────┐
     │  HiFi-GAN    │
     └──────────────┘
            │
        waveform
```

BreezyVoice [2501.17790], a neural TTS system, advances tokenization and synthesis:

- **Supervised Semantic Speech (S³) Tokenizer**: Converts raw speech to discrete semantic units by combining convolutional/transformer encoders with VQ, supervised by ASR loss.
- **LLM for Text-to-Unit Generation**: Predicts unit sequences from phonetic-augmented text and speaker embeddings.
- **Optimal-Transport Flow-Matching Model (OT-CFM)**: Maps discrete unit sequences to mel-spectrograms, conditioned on speaker reference and context.
- **Grapheme-to-Phoneme Predictor (g2pW)**: State-of-the-art contextual disambiguation for Mandarin polyphones.

## 2. Data Preparation and Augmentation

End-to-end Taiwanese Mandarin SLMs leverage hybrid datasets:

- **Real Dialogues**: ~40,000 h of segmented and transcribed multi-turn conversation, processed via diarization, source separation, and ASR correction.
- **Synthetic Dialogues**: ~100,000 h generated using GPT-4o (scenario scripting, role-playing, interruptions), then synthesized via fine-tuned CosyVoice-300M TTS [2411.07111].
- **Token Sequence Construction**: Hybrid input mixing text (ASR), discrete units (HuBERT/S³), and explicit phonetic information (g2pW). Alignment is achieved through timestamp-based interleaving.

In BreezyVoice [2501.17790], training corpora comprise 300+ h of speech (including manually and LLM-derived labels), balanced for age, gender, region, and pitch. Text datasets include both general monologue (TCMD) and code-switching contexts (TCCSD).

## 3. Training Objectives and Modalities

SLMs employ multitask, multimodal training strategies:

- **Pre-training**: Next-token cross-entropy with ASR (unit→text, unit→hybrid), TTS (text→unit, text→hybrid), and text-only objectives. For "spml-omni," hybrid modality provides sufficient cross-modal alignment; no explicit duration losses are needed [2411.07111].
- **Supervised Fine-tuning (SFT)**: Expands hybrid input/output mapping, adding spoken dialogue, unit-hybrid, and hybrid-hybrid tasks (~100 kh).
- **Phonetic Augmentation**: BreezyVoice up-weights polyphone loss terms during LLM and g2pW training, injecting disambiguating phonemes at the character level with masking to maintain generalization [2501.17790].

Table: Example Pre-training Task Mix (spml-omni, 44kh)
| Task | Input | Output | Hours |
|------|-------|--------|-------|
| ASR  | unit  | text   | 6k    |
| TTS  | text  | unit   | 6k    |
| ...  | ...   | ...    | ...   |

Hybrid training enables unified handling of both spoken and textual modalities and supports prompt-based turn-taking, full-duplex control, and robust alignment of speech units and text tokens.

## 4. Phonetic Disambiguation and Polyphone Handling

Polyphone disambiguation is critical for Taiwanese Mandarin due to high ambiguity in character-to-phoneme mapping. BreezyVoice implements:

- **BERT-based g2pW Predictor**: Produces context-sensitive phoneme predictions for polyphonic characters using conditional weighted softmax and POS/context features.
- **Phonetic Symbol Injection**: Augments input sequences as $[y_1, p_1, y_2, p_2, \dots]$, allowing the LLM strong control over unit prediction.
- **Loss Margins for Polyphones**: Training up-weights polyphonic tokens by factor $\lambda_{\mathrm{poly}}=2.0$ to emphasize correct disambiguation.
- **Empirical Results**: On 23 hard polyphone instances, g2pW augmentation reduced error from 8/23 to 1/23 [2501.17790].

A plausible implication is that explicit contextual-phonetic injection substantially increases model fidelity for rare and ambiguous character forms.

## 5. Real-time Interaction and Latency

Streaming and full-duplex capability are central:

- **ASR + Unit Segmentation**: Streaming with windowed segmentation (ASR: 0.3 s; Units: 0.1 s).
- **Dynamic Chunking in Decoder**: Approximates streaming by scheduling chunked synthesis to precede audio completion by a safety $\epsilon$ (50 ms typical).
- **Latency Measurements**: "spml-omni" reports worst-case end-to-end delay of ≲2.5 s (optimized ≲1.9 s) for speech-to-speech interaction [2411.07111]. Human conversational turn-taking averages ≲500 ms, indicating substantial room for further improvement via neural streaming decoders.

## 6. Evaluation Protocols and Performance

Automatic and human-centric metrics quantify intelligibility, naturalness, and coherence:

- **Automatic Metrics**: 
    - Intelligibility: Character Error Rate (CER), e.g., 28.99% for s2s, 32.80% for u2s at final SFT checkpoint in "spml-omni" [2411.07111].
    - Speech Quality: Mean Opinion Score (MOS) predicted via NISQA or SSL-MOS (1–5 scale).
    - Contextual Coherence: LLM-Score (Relevance, Adherence, Grammar; 1–5).
    - Phoneme Error Rate (PER): BreezyVoice achieves PER 0.80% (TCMD), outperforming Z (1.17%), Y (1.45%), U (0.69%), M (0.43%) [2501.17790].
- **Human Preference**: In pairwise tests, BreezyVoice wins 100% vs Z, 73.3% vs Y, etc.
- **Code-switching Accuracy**: BreezyVoice shows robust handling except in toponyms (e.g., 3/10 score).
- **Live and Agent Evaluations**: "Forum" agent↔agent roleplay, live user up/down-vote ratings, and subjective response analysis are ongoing [2411.07111].

Table: Selected Dialogue Test Results (spml-omni, [2411.07111])
| Model           | Modality | CER (%) | MOS   | LLM-Score |
|-----------------|----------|---------|-------|-----------|
| spml-omni-last  | s2s      | 28.99   | 3.42  | 3.4       |
| spml-omni-last  | u2s      | 32.80   | 3.51  | 1.8       |

## 7. Practical Challenges and Future Directions

Key challenges include:

- **Robustness to Long-Tail Speakers**: Error analysis shows most synthesis errors originate from LLM unit prediction; "Iconic Unit Augmented Cloning" reduces PER on noisy speakers by 61.2% with minor speaker similarity loss [2501.17790].
- **Polyphone and Rare-word Disambiguation**: Explicit g2pW injection corrects the majority of ambiguous instances and rare-characters.
- **Streaming Limitations**: Decoders are not fully streaming; future architectures must address this gap relative to conversational human latency.
- **Ethical Concerns**: Few-shot cloning incurs spoofing risks; anti-spoofing and consent protocols are essential in deployment [2501.17790].
- **Limitations**: Toponyms in code-switching contexts, optimal phonetic control integration in neural decoders, and meta-learning for long-tail speaker adaptation are identified as ongoing research targets.

Taken collectively, these efforts demonstrate the viability of end-to-end, duplex-capable, highly controllable spoken language models for Taiwanese Mandarin, setting benchmarks for fluency, intelligibility, and phonetic precision in both research and applied domains [2411.07111][2501.17790].

Source: https://www.emergentmind.com/topics/taiwanese-mandarin-spoken-language-model