Continuous Speech Tokenization

• Continuous speech tokenization is the process of converting high-rate audio signals into information-dense token sequences that capture linguistic, acoustic, and emotional features.
• Tokenization frameworks balance information preservation, compactness, rate control, and robustness by integrating techniques such as vector quantization and self-supervised learning based embeddings.
• Advanced methods, including hierarchical, duration-aligned, and flow-based approaches, enable robust multi-modal integration and low-latency real-time processing.

Continuous speech tokenization is the process of converting raw audio waveforms into sequences of symbolic or continuous tokens that capture the multifaceted content of speech—semantics, acoustics, prosody, speaker, and emotion—in a form suitable for downstream computational models. Tokenization frameworks must balance information preservation, compactness, rate control, and robustness to signal variations. Recent advances span both discrete tokenization (vector quantization, clustering, alignment to linguistic units) and continuous embedding-based approaches, each with distinct tradeoffs in modeling power, compressibility, and application scope.

1. Foundations of Continuous Speech Tokenization

Continuous speech tokenization targets the conversion of high-rate continuous signals (audio waveforms, e.g., sampled at 16–24 kHz) to information-rich, low-rate symbolic or continuous representations, enabling speech generation, understanding, compression, and integration with LLMs. Historically, speech representations for downstream tasks relied on Mel-spectrograms, phoneme sequences, or frame-based feature vectors. Modern tokenizers seek higher semantic fidelity, cross-modal alignment, and robustness to domain shifts and distortions.

Practical tokenizers must address:

• Information preservation: Retain both linguistic (phonetic, lexical) and acoustic (prosody, speaker, emotion) cues.
• Compactness and control: Minimize token sequence length and support tunable rate (Hz, bps) while controlling quality loss.
• Robustness: Maintain invariance or disentanglement to speaker, noise, and environment.
• Alignment: Support alignment to linguistic units (characters, syllables, words) and arbitrary durations.
• Applicability: Enable seamless integration into speech LMs, text-to-speech (TTS), speech-to-text (ASR), translation, and multimodal systems.

Symbolic frameworks (discrete tokens) dominate language modeling integrations; continuous tokens are gaining traction in high-fidelity generation, robust multimodality, and low-latency applications.

2. Discrete Tokenization Architectures and Algorithms

2.1. Frame-wise Clustering and Vector Quantization

Early neural codecs (EnCodec, SoundStream) and self-supervised learning (SSL) models (HuBERT, WavLM) produce latent representations. Discrete quantization is performed via methods including residual vector quantization (RVQ) (Jung et al., 9 Jul 2025), k-means clustering (Zhu et al., 2023), finite scalar quantization (FSQ) (Huang et al., 31 Jan 2026), and codebooks learned with VQ-VAE objectives (Turetzky et al., 2024).

A canonical pipeline:

• Feature extraction: Encoder (CNN, Transformer, SSL model) yields latent vectors at a target frame rate (e.g., 50 Hz or 25 Hz).
• Quantization: Each frame is mapped to a discrete codeword/categorical index:
  • RVQ assigns a codebook index per layer, with residuals quantized at each step (see Section 2 in (Jung et al., 9 Jul 2025)):

  $$Q_i(r^{i-1}) = e^i_{k_i}, \quad k_i = \underset{j=1\ldots K}{\mathrm{argmin}\;\|r^{i-1} - e^i_j\|_2^2}$$

  $r^0 = x, \quad r^i = r^{i-1} - Q_i(r^{i-1})$ - K-means assigns each latent to its nearest cluster centroid (Zhu et al., 2023). - FSQ splits each feature into scalars and quantizes each independently (Huang et al., 31 Jan 2026).

• Token sequence formation: Token streams are optionally deduplicated (collapsing runs), byte-pair encoded, or compressed.

2.2. Hierarchical and Disentangled Token Streams

State-of-the-art tokenizers structure the quantization process hierarchically, with lower layers focusing on semantic/phonetic content (guided by distillation from frozen HuBERT, BERT, or large text LMs) and higher layers capturing residual acoustic, prosodic, and speaker information (Jung et al., 9 Jul 2025, Ahasan et al., 2024, Zhang et al., 2023).

Some tokenizers implement explicit branch separation for content (phonetics, prosody) and global (speaker, channel) factors (Huang et al., 31 Jan 2026), or attach multiple codebooks to force finer phonetic alignment and speaker invariance (Chang et al., 2024).

2.3. Duration and Alignment Modeling

Token sequence length is typically fixed by frame rate, but variable‐rate and dynamically aligned tokenization is emerging. DyCAST deploys a hazard-based boundary predictor to chunk audio into variable-length, character-aligned segments and models durations explicitly (negative-binomial predictors) (Libera et al., 30 Jan 2026), substantially reducing sequence length with controlled information loss.

TASTE-S performs streaming, text-aligned tokenization by integrating ASR-based CTC decoding and cross-attention aggregation, achieving 1:1 alignment between speech tokens and text tokens (Tseng et al., 12 Mar 2026).

SyllableLM extracts syllable-like boundaries by analyzing sharp changes in MLM loss and iteratively distills representations, achieving ultra-low token rates (5–8 Hz) while maintaining high semantic preservation (Baade et al., 2024).

3. Continuous Token Approaches and Flow-Based Generation

Continuous tokenization eschews quantization, instead representing each audio segment directly as a learned continuous embedding:

• Cont-SPT preserves input information rates across frequencies better than discrete tokenizers, as measured by frequency retention $R(f)$ and MOS scores, and notably avoids the steep high-frequency loss of discrete RVQ (Li et al., 2024).
• CLEAR utilizes a variational autoencoder (VAE) with aggressive downsampling (up to 2048:1), modeling the resulting sequence of real-valued latents autoregressively. The latent sequence ($\sim$7.8 Hz) enables low-latency, high-fidelity TTS, surpassing discrete-token models in synthesis speed and word error rate (Wu et al., 26 Aug 2025).
• Flow-based models (Flow-Omni) jointly train LLMs and continuous token predictors using conditional flow matching losses, avoiding quantizer artifacts and enabling robust multi-modality (text + speech) learning and generation (Yuan et al., 2024).

Continuous tokens maintain gradient flow, offering benefits in naturalness, prosody, resilience to out-of-domain audio, and computational efficiency for generation.

4. Training Objectives, Losses, and Optimization

Training tokenizers involves multiple coordinated losses:

• Reconstruction Losses: L1 waveform loss, multi-resolution STFT, GAN-based adversarial and feature-matching terms; these ensure the quantized (or continuous) tokens reconstruct the original audio with perceptual fidelity (Jung et al., 9 Jul 2025, Wu et al., 26 Aug 2025, Ahasan et al., 2024).
• Commitment and Diversity Losses: Encourage encoder outputs to utilize the codebooks fully (VQ commitment), avoid collapse, and approximate a uniform marginal token distribution (Jung et al., 9 Jul 2025, Messica et al., 2024).
• Semantic and Acoustic Distillation: Align token streams to frozen semantic SSL features (HuBERT, wav2vec 2.0) or text LM embeddings (BERT, OPT), explicitly optimizing for joint content and contextual fidelity; crucial for multi-purpose LMs and cross-modal alignment (Jung et al., 9 Jul 2025, Ahasan et al., 2024, Turetzky et al., 2024).
• Robustness/Augmentation Losses: Apply augmentations (noise, pitch, reverb, stretch) in training and enforce invariance between clean and perturbed outputs (e.g., via cross-entropy alignment, edit-distance matching) (Messica et al., 2024, Chang et al., 2024).
• Flow/ODE-based Losses: Train continuous token predictors to match data distributions via conditional flow matching and denoising ODE trajectories (Yuan et al., 2024, Wu et al., 26 Aug 2025).
• Self-Alignment and Sequence Likelihood: PairAlign trains tokenizers via sequence-level self-alignment: an encoder produces continuous features, and an autoregressive decoder generates symbolic tokens whose alignment is enforced via cross-view likelihood and in-batch contrast (Banerjee et al., 7 May 2026).

Loss combinations and weightings are tuned to balance reconstruction with semantic, speaker, and prosodic preservation. Adapters may be inserted to facilitate LM-aware training and unification of text and speech token domains (Turetzky et al., 2024).

5. Evaluation Metrics, Benchmarks, and Practical Guidelines

Performance and suitability of speech tokenizers are assessed by a spectrum of metrics and benchmarks:

• Reconstruction and Naturalness: PESQ, STOI, SDR, DNSMOS, UTMOS, ViSQOL, MUSHRA, and MOS scores for reconstructed or synthesized audio (Jung et al., 9 Jul 2025, Ahasan et al., 2024, Wu et al., 26 Aug 2025).
• Linguistic/ASR Content: WER (Word Error Rate), CER; evaluated both on token streams resynthesized through vocoders and on token-prediction by downstream ASR or LLMs (Jung et al., 9 Jul 2025, Chang et al., 2024).
• Speaker, Prosody, Emotion: Speaker similarity metrics (SIM, EER), emotion classification accuracy, prosody correlation (F0Corr) (Huang et al., 31 Jan 2026, Jung et al., 9 Jul 2025).
• Robustness: Augmentation invariance (chrF, UED) under pitch, noise, and speed perturbations (Vashishth et al., 2024, Messica et al., 2024).
• Compressibility/Rate: Token deduplication, Huffman and BPE compression efficiencies, rate vs. performance trade-offs (Vashishth et al., 2024, Baade et al., 2024).
• Alignment Probes: Phonetic/character mutual information (PNMI, CNMI), ABX error rates for phoneme boundaries (Chang et al., 2024).
• Downstream Integration: Task-level metrics (BLEU for speech-to-text translation, sWUGGY/sBLIMP for spoken language modeling, topic/story cloze accuracy) (Jung et al., 9 Jul 2025, Chang et al., 2024, Baade et al., 2024).
• Benchmarks: SLMTokBench for speech LMs (Zhang et al., 2023), STAB (Speech Tokenizer Assessment Benchmark) for task correlations and invariance/compressibility/vocabulary axes (Vashishth et al., 2024).

Key guidelines include:

• Prefer semantic-first VQ or hybrid RVQ tokenizers for ASR/TTS/multimodal LMs, and acoustic-only tokenizers for waveform fidelity/fine voice conversion (Jung et al., 9 Jul 2025, Ahasan et al., 2024, Zhang et al., 2023).
• Tune rate, codebook size, and quantization granularity for the desired balance of quality, robustness, and model speed (Baade et al., 2024, Libera et al., 30 Jan 2026).
• Employ robustness-enforcing augmentations and measure invariance before downstream deployment (Messica et al., 2024, Vashishth et al., 2024).
• For dynamic or long-form applications, use alignment-aware (DyCAST, TASTE-S, SyllableLM) or streaming-capable (DC-Spin) approaches for efficiency and real-time requirements (Libera et al., 30 Jan 2026, Tseng et al., 12 Mar 2026, Chang et al., 2024).

6. Advances in Dynamic and Efficient Tokenization

Recent methods address limitations of uniformly sampled, high-rate tokenizers by introducing event-aligned, variable-length, or autoregressively generated token streams:

• Variable-Rate and Alignment-Aware Tokenization: DyCAST (Libera et al., 30 Jan 2026) demonstrates that soft character-alignment and explicit duration control can drastically reduce token rates (from 50–80 Hz to 6–18 Hz), enabling efficient real-time speech processing while maintaining or improving downstream performance.
• Sequence-Predictive Tokenization: PairAlign replaces local quantization with encoder-decoder sequence prediction, training tokens for consistency under content-preserving augmentations and explicit sequence alignment objectives (edit distance, likelihood contrast, entropy regularization), yielding compact, variable-length, high-consistency token streams (Banerjee et al., 7 May 2026).
• Syllable-Level Abstraction: SyllableLM (Baade et al., 2024) extracts syllable-like boundaries via self-supervised masked-loss correlation and iterative distillation, achieving rates down to 5 Hz (~60 bps) while maintaining high spoken-language modeling performance and drastically reducing compute and inference costs.
• Streaming and Real-Time Tokenizers: TASTE-S (Tseng et al., 12 Mar 2026) provides a fully streamable, text-aligned speech-unit tokenizer with sub-second latency and minimal loss relative to offline methods. DC-Spin (Chang et al., 2024) implements chunked streaming via overlapping windows and boundary trimming, showing low latency and negligible sequence mismatch.

These advances pave the way for speech tokenizers adaptable to varying application requirements—compression, real-time interaction, robust multimodal LLMs, and efficient retrieval/search.

7. Future Directions and Research Challenges

Continuous speech tokenization remains an area of active research due to its foundational role in speech-based computation. Outstanding challenges and emergent trends include:

• Unified Modeling of Text and Speech: LM-aware and multi-modal tokenizers (e.g., LAST (Turetzky et al., 2024), Flow-Omni (Yuan et al., 2024)) demonstrate the feasibility of using a single LLM backbone for both speech and text, bridging modality gaps and enabling fully integrated conversational agents.
• Fine-Grained Control over Disentanglement: Architectures such as Kanade (Huang et al., 31 Jan 2026) and DC-Spin (Chang et al., 2024) show that bottleneck design and loss configuration can induce token streams that are highly selective for content, prosody, or speaker characteristics, enabling targeted applications in anonymization, speaker transfer, or emotion recognition.
• Benchmarks and Automated Assessment: Frameworks like STAB (Vashishth et al., 2024) provide systematic tools for evaluating tokenizers across core properties—information preservation, invariance, robustness—which can guide tokenizer selection or bespoke design for specific tasks.
• Hybrid Discrete-Continuous Approaches: Approaches combining continuous and discrete tokens, or allowing flexible post-tokenization adaptation (e.g., retrieval-augmented decoding (Libera et al., 30 Jan 2026)), are likely to improve both generation quality and modeling/explainability.

A plausible implication is that future continuous speech tokenization systems will more deeply integrate text supervision, semantic and prosodic disentanglement, adaptive alignment, low-latency streaming, and explicit control over compression, aligning tokenization not merely with reconstruction but with the ultimate structure and goals of downstream multimodal LLMs.