HoliTok:A Coutinuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understanding

Abstract: Unified speech foundation models require a holistic tokenization space that is both learnable by LLMs and decodable into high-quality waveforms. Existing speech tokenizers, however, often fail to satisfy these requirements simultaneously, leading to increased architectural complexity and more involved training designs. We propose HoliTok, a continuous Holistic speech Tokenization model designed for unified generation-understanding modeling. HoliTok encodes 48~kHz speech into a compact 25~Hz sequence of 128-dimensional latents. It is trained with a progressive strategy that jointly preserves signal-level fidelity, incorporates semantic information, and maintains strong latent learnability. Based on this tokenization, we build a unified AR+DiT model for speech synthesis and recognition, where the same latent sequence supports both generation-specific and unified generation-understanding tasks. Experiments show that HoliTok achieves competitive reconstruction fidelity, improves generative learnability for high-quality and controllable synthesis, and, among the evaluated representations, is the only one that operates robustly in our unified generation-understanding architecture without additional optimization tricks. These results suggest that HoliTok serves as an effective speech tokenizer and a foundational representation interface for unified spoken language modeling. The code is available at: https://github.com/bovod-sjtu/HoliTok.

• The paper introduces a learnable, decodable continuous tokenization framework that integrates a three-stage training paradigm for high-fidelity reconstruction and semantic enrichment.
• It demonstrates robust performance with a 7.5× compression ratio while maintaining strong metrics like PESQ, STOI, and low WER across speech synthesis and recognition tasks.
• The approach unifies generative and recognition tasks in a single AR+DiT model, paving the way for efficient multimodal foundation model development.

HoliTok: Continuous Holistic Tokenization for Unified Speech Generation and Understanding

Motivation and Background

The transition toward unified spoken language modeling necessitates a foundational representation space that enables both speech generation and understanding within a shared architecture. Existing tokenizers for speech largely fall short of this standard—discrete codecs introduce quantization artifacts and modular complexities, while most continuous tokenizers focus solely on reconstruction or synthesis fidelity. These limitations have prompted researchers to adopt increasingly intricate downstream models with task-specific components, which erode the abstraction and efficiency intended by unified modeling. HoliTok addresses this gap by proposing a learnable, decodable, and semantically synergistic continuous tokenization, optimized for both generative and recognition tasks within a unified AR+DiT structure.

Methodological Framework

HoliTok employs a progressive three-stage training paradigm to sequentially establish high-fidelity reconstruction, variational regularity, and downstream semantic enrichment:

1. Stage I: Deterministic autoencoder pretraining grounds the latent space in waveform fidelity via a multi-objective loss encompassing mel-spectral, adversarial, and discriminator feature matching terms.
2. Stage II: Temporal variational bottleneck training (with weak KL constraint) transfers the deterministic latent encoding into a stochastic, regularized space while keeping encoder and decoder weights frozen. This implicit fidelity transfer ensures that variational samples remain close to the autoencoder manifold, as formalized by a Lipschitz-bounded distortion guarantee.
3. Stage III: Joint optimization introduces multi-granularity distillation against SSL teacher models (WavLM, x-vector) and task-conditioned supervision (ASR, emotion, audio captioning, sound event detection), further shaping the latent space for downstream utility without sacrificing reconstruction.

The final HoliTok tokenization encodes $48\,\mathrm{kHz}$ speech into a compact $25\,\mathrm{Hz}$ sequence of 128-dim continuous latents, supporting both generation and understanding within a unified AR+DiT model. The backbone comprises a causal convolutional encoder with residual dilation stacks, LSTM-based variational bottleneck with flow regularization, and a BigVGAN-style decoder.

Figure 1: HoliTok's progressive three-stage training and downstream AR+DiT architecture for unified generation-understanding.

Experimental Evaluation

Reconstruction and Compression

HoliTok maintains tight signal reconstruction metrics (PESQ, STOI, UTMOS) and achieves a strong compression ratio (7.5$\times$ at 25 TPS). Notably, HoliTok preserves linguistic (WER), paralinguistic (SPKSIM, EMOSIM), and perceptual qualities even under substantial bitrate reduction. Compared with discrete codecs and vanilla VAEs, HoliTok is distinguished by robust fidelity and semantic preservation in the latent space.

Speech Synthesis and Learnability

HoliTok's latent representation serves as a highly learnable generative target for AR+DiT-based TTS models. Zero-shot evaluations demonstrate competitive WER and speaker similarity, while controllable synthesis assessments yield best-in-class WER and CLSP scores with strong emotion similarity, underscoring the model's controllability and expressive diversity without compromising intelligibility.

Figure 2: HoliTok outperforms baselines on controllable TTS evaluation, notably in both WER and style control metrics.

Unified Spoken Language Modeling

When instantiated within unified ASR-TTS tasks, HoliTok-Base achieves balanced performance for both recognition and synthesis, outperforming competitors (Semantic-VAE, MingTok-Audio) that exhibit modality biases or diminished generative capacity. HoliTok-Unite, leveraging a causal semantic encoder, further advances performance across both tasks (average TTS WER reduced by over 12 points, ASR WER improved by 4 points), without reliance on architectural trickery or incremental component specialization.

Ablation studies confirm that pure reconstruction or distillation does not suffice for holistic usability; downstream-aware supervision is critical. DiT initialization from TTS-specialized checkpoints enhances generative quality, and maintaining semantic encoder adaptivity is essential for comprehensive modeling.

Implications and Prospects

HoliTok demonstrates that properly regularized and semantically enriched continuous latent spaces can serve as a single interface for robust speech modeling—invalidating the necessity for task-specific modularity. This finding supports broader trends in multimodal foundation models, where learnable, decodable representations become central for unified architectures.

Practically, HoliTok enables efficient speech compression, versatile controllable generation, and joint recognition capabilities—all on high-resolution audio. The approach generalizes to AR+DiT frameworks, suggesting compatibility with other modality-unification efforts (visual, audio, multimodal). The theoretical underpinnings (implicit fidelity transfer, multi-task semantic regularization) offer directions for generalized tokenization paradigms and deeper analysis of downstream-aware latency bottlenecks.

Key limitations remain: current experiments prioritize spoken content and do not fully explore non-speech audio or alternative backbone architectures (e.g., fully non-autoregressive models). Extending HoliTok to environmental sound, music, and broader audio domains, as well as analyzing robustness to domain transfer and architectural variation, forms the basis for successive research.

Conclusion

HoliTok establishes a principled method for continuous, holistic speech tokenization that meets the requirements of unified generation-understanding modeling. Through progressive regularization and downstream-aware semantic enrichment, HoliTok achieves competitive reconstruction, strong generative learnability, and robust joint modeling without incremental architectural complexity. The research substantiates the feasibility of unified continuous latent interfaces and opens pathways for multimodal foundation model development and efficient speech processing.