HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Models

Abstract: Discrete audio representations have become increasingly popular for building multimodal text-audio systems and integrating audio capabilities into LLMs. However, numerous studies report performance degradation on various downstream tasks due to information loss during discretization. To address this, we propose a novel approach combining temporally compressed discrete tokens with dimensionality-reduced continuous residuals. Our framework consists of a hybridized discrete-continuous focal modulation codec and a hybrid Transformer. This architecture performs autoregressive inference in the discrete domain, coupled with non-autoregressive prediction and continuous residual upsampling. Experimental results show that our approach significantly improves the retention of speaker characteristics compared to discrete-only methods, while simultaneously reducing the number of required autoregressive steps.

• The paper introduces HybridCodec, which fuses discrete tokens and continuous residuals to restore fine-grained speech details lost in aggressive compression.
• It employs a dual-path architecture with autoregressive and non-autoregressive modeling to significantly improve speech intelligibility and speaker fidelity under low bitrates.
• This unified approach accelerates inference and enhances accuracy for both TTS and ASR applications, setting a new standard in efficient speech modeling.

HybridCodec: Joint Discrete-Continuous Representation for Efficient Speech LLMs

Motivation and Problem Statement

The discretization of audio signals has enabled scalable integration of speech capabilities within LLMs, facilitating zero-shot TTS and unified speech-text modeling. Contemporary neural audio codecs (NACs) compress audio into low-bitrate discrete tokens, benefiting from stable autoregressive (AR) modeling and seamless LLM compatibility. However, evidence from speech benchmarks indicates that purely discrete representations incur an irreducible information bottleneck due to quantization, severely degrading fine-grained acoustic details and speaker identity, especially at aggressive compression rates. Attempts at task-specific continuous modeling—such as masked or diffusion-based approaches—are typically siloed, unable to deliver a unified, efficient generative-discriminative speech modeling framework.

Architecture: HybridCodec and HybridLM

The HybridCodec architecture, illustrated in (Figure 1), addresses these limitations through dual-path temporal compression and joint representation modeling. The codec comprises two intertwined streams: one produces temporally downsampled discrete tokens via a quantization pathway, while the other encodes continuous residuals, capturing information lost during quantization through an auxiliary encoder with configurable temporal resolution. The representations from both streams are fused to reconstruct the speech signal at synthesis time.

Figure 1: The HybridCodec architecture (left) applies dual-path discrete-continuous compression; HybridLM (right) unifies sequential AR and NAR modeling of token streams for generative and discriminative tasks.

HybridLM, a decoder-only Transformer, operationalizes these hybrid representations. It incorporates both AR decoding for discrete tokens and non-autoregressive (NAR) regression for continuous residuals within a single network backbone. Mode-specific layer normalization (AdaLN) is deployed so that the model dynamically conditions itself for AR (classification) or NAR (regression) processing at each layer, precluding destructive interference between objectives. Speaker conditioning is achieved via ECAPA-TDNN embeddings injected into token input sequences, maintaining speaker identity in both ASR and TTS tasks.

Quantitative Evaluation

The empirical analysis uses the LibriTTS dataset, benchmarking both reconstruction (codec resynthesis) and end-to-end downstream speech tasks (TTS, ASR) with standard metrics: UTMOS (neural MOS estimator), dWER (differential WER relative to ground-truth), SpkSim (speaker similarity), WER/CER (recognition accuracy), and codebook utilization measures.

Key findings include:

• At low frame rates (12.5 Hz), HybridCodec achieves a dWER of 1.47 and UTMOS of 4.09, meaningfully improving intelligibility over discrete-only codecs where dWER typically exceeds 7.9.
• Robustness under ultra-compressed conditions (6.25 Hz) is substantial: UTMOS remains high (3.98) and speaker similarity (97.1) is preserved, where discrete baselines collapse (UTMOS 1.44, SpkSim 0.707).
• For generative TTS, HybridCodec more than doubles UTMOS (4.10 vs. 1.99) and achieves less than half the dWER at 12.5 Hz, confirming that the continuous track recovers essential information lost in quantization.
• HybridLM unifies both modalities, reducing AR generation steps by a factor proportional to the downsampling rate, with negligible impact to quality at moderate compression, thus dramatically accelerating inference for long-form synthesis.
• On ASR tasks, hybrid modeling consistently reduces WER and CER—demonstrating that continuous residuals improve semantic recognizability without impairing the benefits of discrete AR modeling.

Broader Implications and Theoretical Impact

The results establish a strong case for hybrid compressive representations in speech modeling. Crucially, HybridCodec restores much of the fine temporal detail and speaker-dependent nuance lost in discrete-token-only approaches, achieving both high semantic robustness and signal fidelity at extreme compression. This is accomplished with negligible increases in computational and storage cost relative to the fully discrete baseline. The architectural unification delivered by HybridLM suggests that future large-scale speech LMs can bypass multi-stage RVQ stacks, eliminate task-specific post-processing, and integrate continuous refinement pathways with minimal engineering overhead.

Practically, this architecture enables high-fidelity, low-latency speech synthesis and recognition on resource-constrained or streaming platforms. The method supports generalizable, multi-task, and multi-speaker capabilities within a single Transformer backbone, aligning unified modeling objectives across the generative and discriminative speech domains.

Theoretically, the work offers new insights into rate-distortion trade-offs: by treating the quantization error as an explicit modeling target, the hybrid paradigm sidesteps some constraints of classical rate-distortion optimization, effectively using learned continuous channels to close the performance gap at ultralow bitrates. The use of adaptive normalization for AR/NAR co-training in a Transformer context may generalize to other vision, audio, and multimodal synchronisations involving mixed data types.

Future Directions

Potential research avenues include scaling HybridCodec to multilingual, code-switching, or noisy environments, and examining extensions to multi-modal generative LMs beyond speech (for example, video or tactile signals). Direct comparison with deep stacked RVQ architectures and integration with promptable diffusion modeling remain open, as does the exploration of task-adaptive hybridization for reinforcement learning or event-driven representation learning.

Conclusion

HybridCodec introduces a principled unification of discrete and continuous audio representations within a single neural codec and LLM architecture (2606.27627). By interleaving autoregressive and non-autoregressive modeling regimes within the HybridLM, the system delivers high-fidelity speech synthesis and recognition at unprecedented compression rates, setting a new standard for efficiency and fidelity in self-supervised speech LLMs. The framework’s success at reducing inference cost while simultaneously enhancing quality and semantics suggests a paradigm shift in efficient speech AI system design.