SecoustiCodec: Cross-Modal Low-Bitrate Codec

• SecoustiCodec is a streaming cross-modal speech codec that separates semantic and paralinguistic information to enable low-bitrate, causal audio reconstruction.
• It employs VAE-based semantic quantization with FSQ to improve codebook utilization and mitigate long-tail token distributions.
• The architecture achieves ultra-low latency (12.08 ms) while delivering high reconstruction quality at operating points of 0.27 to 1 kbps.

Searching arXiv for SecoustiCodec and closely related codec papers to ground the article. SecoustiCodec is a cross-modal aligned low-bitrate streaming speech codec that disentangles semantic and paralinguistic information in a single-codebook space. It was proposed to address residual paralinguistic leakage in “semantic” tokens, insufficient semantic completeness, limited reconstruction quality at ultra-low bitrates, lack of streaming support in single-codebook methods, long-tail token distribution, and poor codebook utilization. Its central design claim is that semantic completeness and reconstruction fidelity can be improved by introducing an explicit paralinguistic encoding pathway, while semantic-only efficient quantization based on VAE and FSQ improves codebook utilization and alleviates the long-tail distribution problem. The reported operating points are 0.27 kbps at 20 Hz and 1 kbps at 80 Hz, with PESQ values of 1.77 and 2.58 respectively, under a fully causal architecture with initial latency 12.08 ms (Qiang et al., 4 Aug 2025).

1. Problem setting and design objective

SecoustiCodec is situated in the recent effort to make speech codecs serve as a bridge between speech and text LLMs. In that setting, discrete speech representations are expected to be simultaneously compressive, semantically informative, reconstructible, and streamable. The problem formulation identifies several persistent failure modes in prior codec families: “semantic” tokens often retain speaker timbre, broad emotion, and prosody; first-layer semantic tokens may not be semantically complete without auxiliary acoustic tokens; ultra-low-bitrate reconstruction quality degrades under causal constraints; and single-codebook approaches either lack streaming support or rely on non-causal architectures that require lookahead. A separate issue is token statistics: long-tail token distributions and poor codebook utilization hinder LLM training, and VQ-based systems may exhibit dead codes and unbalanced occupancy (Qiang et al., 4 Aug 2025).

The proposed remedy is a codec that disentangles semantic and paralinguistic information in one single-codebook space. The semantic channel is intended to carry “pure” semantics, the paralinguistic channel is intended to carry dominant residuals such as speaker timbre, broad emotional characteristics, and macro-prosodic style, and the acoustic representation is treated as the composition of those factors. This factorization is operationalized through the relationship $S + G \approx A$, where $S$ denotes semantic embeddings, $G$ a global paralinguistic vector, and $A$ full acoustic content. The design is therefore not only a compression strategy but also a representational claim about how speech should be decomposed for downstream modeling (Qiang et al., 4 Aug 2025).

A common misconception is that a single-codebook codec necessarily sacrifices semantic purity or reconstruction quality. SecoustiCodec was explicitly introduced to contest that assumption: it uses a single-codebook semantic path, but supplements it with a dedicated paralinguistic encoder and an acoustic-constrained optimization strategy so that reconstruction does not depend on retaining paralinguistics inside the semantic tokens themselves (Qiang et al., 4 Aug 2025).

2. Architecture and factorized representation

The system is organized as three-factor modeling with causal modules. The acoustic path is continuous and frame-level, implemented as SpeechEncoder → AcousticProjection → SpeechDecoder. The semantic path is discrete and frame-level, implemented as SpeechEncoder → SemanticProjection (VAE) → FSQ quantizer → Semantic tokens S. The paralinguistic path is continuous and global-level, implemented as ParalinguisticEncoder (VAE) → global vector G. A Semantic Connector then predicts acoustic embeddings from $S$ and $G$, enforcing the factorization through $\hat{A} = \mathrm{Connector}(S, G)$ and the relation $S + G \approx A$ (Qiang et al., 4 Aug 2025).

During training, speech mel-spectrogram $S_{in}$ and text phonemes $\tilde{P}_{in}$ are length-aligned to $S$0 frames via a duration-based regulator, giving $S$1 and $S$2. The phoneme encoder produces $S$3, compressed by factor $S$4. The speech encoder produces continuous features that are mapped by AcousticProjection to $S$5, compressed by factor 4. The SemanticProjection, implemented as a causal transformer with a VAE, outputs $S$6, followed by reparameterization and FSQ to produce $S$7. The ParalinguisticEncoder operates on a 3 s window and outputs a global vector $S$8 (Qiang et al., 4 Aug 2025).

All inference-critical modules are causal. The speech encoder and decoder use causal SeaNet blocks with 80-channel convolutions, kernel size 7, ELU, dilation base 2, 1 residual layer with true skip, and compression factor 2. The acoustic projection, semantic projection, and semantic connector are causal masked transformers with 8 layers, 8 heads, RoPE, 512-D model, and 2048-D FFN. AcousticProjection processes 250-frame context windows in a causal layout. The codec runs at 20 Hz in the 0.27 kbps configuration and 80 Hz in the 1 kbps configuration (Qiang et al., 4 Aug 2025).

This architectural split has two immediate implications. First, semantic tokens are not expected to encode all reconstructive detail. Second, the codec’s streaming behavior is built into the architectural primitives rather than added by post hoc chunking. That is significant because many single-codebook semantic codecs have historically traded away causality to recover quality; SecoustiCodec instead defines streaming as a first-class constraint (Qiang et al., 4 Aug 2025).

3. Semantic-only quantization and cross-modal alignment

The semantic path uses a VAE followed by FSQ rather than a conventional VQ-VAE codebook lookup. Reparameterization is defined as

$S$9

The standard VAE objective is stated as

$G$0

In SecoustiCodec, the reconstruction pressure associated with the ELBO is realized through explicit regression losses rather than through a separate decoder-likelihood term: $G$1 in Stage 1, and $G$2 in Stage 2 (Qiang et al., 4 Aug 2025).

FSQ maps each scalar channel to one of $G$3 bounded levels:

$G$4

with implied factorized codebook size $G$5. Proj_down reduces to a low-rank space, $G$6 bounds values to $G$7, round performs scalar quantization, and Proj_up restores the model dimension. Gradients are propagated through the quantization step by straight-through estimation. In contrast to VQ-VAE, FSQ does not use learned centroid lookup with commitment loss for the semantic channel; the stated motivation is better utilization and less long-tail skew (Qiang et al., 4 Aug 2025).

The codec reports a usage ratio of 98.06%, where

$G$8

It also reports that most token frequencies stay below 0.2% and nearly all bins activate. The paper further states that perplexity $G$9 increases with balanced code usage. These statistics are used to support the claim that VAE+FSQ alleviates long-tail token distributions while maintaining high codebook utilization (Qiang et al., 4 Aug 2025).

Semantic disentanglement is enforced by frame-level contrastive alignment between speech frames and phoneme embeddings. With reshaped speech and phoneme matrices $A$0, the similarity matrix is

$A$1

and the symmetric contrastive loss is

$A$2

where

$A$3

Because text and phonemes carry only semantic content, the stated effect is to drive $A$4 toward semantics while discouraging timbre, emotion, and other paralinguistic variation from remaining in the semantic channel. No adversarial GRL was needed; contrastive learning replaced ASR classification or HuBERT/WavLM distillation, which are described as tending to retain paralinguistics (Qiang et al., 4 Aug 2025).

4. Optimization, causality, and runtime behavior

SecoustiCodec adopts an acoustic-constrained multi-stage optimization strategy. The rationale is to learn robust acoustic embeddings first, then freeze them so that semantic and paralinguistic factorization does not destabilize acoustic learning. The reconstruction losses are

$A$5

The KL term for the paralinguistic encoder is written with a margin:

$A$6

Stage 1 covers steps $A$7 and trains SpeechEncoder, AcousticProjection, and SpeechDecoder using only $A$8. Stage 2 covers steps $A$9, freezes the Stage 1 modules, and trains PhonemeEncoder, ParalinguisticEncoder, SemanticProjection, and Semantic Connector with

$S$0

with $S$1, $S$2, and KL warm-up defined by $S$3, $S$4, and $S$5 (Qiang et al., 4 Aug 2025).

The streaming design is fully causal. Transformers use causal masks, SeaNet convolutions have no lookahead, and AcousticProjection processes causal windows in a masked fashion. The initial latency is 12.08 ms. The system runs at 20 Hz and 80 Hz depending on bitrate. The paper characterizes this as low-latency streaming suitable for interactive operation (Qiang et al., 4 Aug 2025).

The implementation details reinforce that the streaming design is not merely conceptual. Training uses 8× NVIDIA TESLA A800 80GB GPUs with batch size 64 per GPU and Adam with initial learning rate $S$6. Due to frame-level reshaping in contrastive learning, the number of sample pairs per step is 4k–32k. The parameter size is 379 MB. The Real-Time Factor is reported as total 0.04, encoding 0.002, and decoding 0.038 including the vocoder; excluding the vocoder, decoding RTF is approximately 0.001 and is described as faster than Encodec’s decoding (Qiang et al., 4 Aug 2025).

A second misconception is that single-codebook semantic codecs are inherently non-streaming. SecoustiCodec explicitly rejects that premise by making all inference-critical modules causal and by reporting concrete latency and RTF numbers under streaming settings (Qiang et al., 4 Aug 2025).

5. Experimental profile, baselines, and ablations

The training corpus consists of 1,000 hours of labeled text–speech pairs combining an internal dataset with AISHELL-3 and LibriTTS, covering approximately 3,000 speakers. Audio is sampled at 22 kHz and represented as 80-band mel spectrograms with window size 1024 and hop size 256. Baselines include streaming multi-codebook codecs such as Encodec, SoundStream, and MimiCodec; semantic disentanglement codecs such as FACodec and SpeechTokenizer; and single-codebook codecs such as VQ-CTAP, WavTokenizer, BigCodec, and TAAE (Qiang et al., 4 Aug 2025).

At the main operating points, the paper reports the following reconstruction results.

Bitrate	Token rate	PESQ
0.27 kbps	20 Hz	1.77
1 kbps	80 Hz	2.58

At 0.27 kbps and 20 Hz with $S$7, SecoustiCodec attains PESQ 1.77 and is reported to outperform other single-codebook codecs at similar or higher bitrates; it also surpasses many multi-codebook codecs at much higher token rates. At 1 kbps and 80 Hz with $S$8, it achieves best performance on PESQ 2.58, EmoSim, LSD, MR, and WER compared to many multi-codebook and single-codebook baselines under causal constraints (Qiang et al., 4 Aug 2025).

The ablation studies identify several causal factors behind these results. Quantization matters: VAE+FSQ beats VQ-VAE, SimVQ, and pure FSQ variants in most metrics, and the final system uses a 256-D acoustic embedding with a single quantizer. Codebook utilization also matters: VAE+FSQ achieves 98.06% usage and mitigates long-tail behavior. Causality matters: non-causal models trade off some metrics but lose streaming, and the causal design is described as essential for deployment. Pitch features in the semantic path are harmful: adding $S$9 into the semantic path significantly harms semantic modeling, with MSEP and WER degrading. The multi-stage strategy is also essential: removing it degrades convergence and overall metrics due to interference with acoustic learning. Finally, replacing frame-level contrastive loss by ASR classification loss improves WER slightly but reduces speaker and emotion similarity, which is presented as evidence that contrastive alignment is better for disentanglement (Qiang et al., 4 Aug 2025).

These results should be interpreted with some care. The codec is optimized for the extreme low-bitrate streaming regime, not for high-bitrate non-causal fidelity. The reported advantages therefore concern a specific operating region: very low token rates, causal inference, and single-codebook semantic representation with explicit paralinguistic side information (Qiang et al., 4 Aug 2025).

6. Position within the codec literature, nomenclature, and limitations

SecoustiCodec belongs to a broader movement toward semantically structured codecs, but its specific configuration is distinctive: a single-codebook semantic stream, a continuous global paralinguistic vector, frame-level speech–phoneme contrastive alignment, FSQ-based semantic quantization, and explicit causal streaming (Qiang et al., 4 Aug 2025). A plausible implication of later comparative summaries is that the name “SecoustiCodec” has also come to denote a broader codec design motif rather than only the specific model of (Qiang et al., 4 Aug 2025). In that broader sense, related systems reserve early codebooks for semantic structure and later mechanisms for acoustic detail or contextual information.

STACodec, for example, integrates external semantic tokens directly into the first residual vector quantization layer through semantic token assignment and uses semantic pre-distillation to remove dependence on external SSL tokenizers at inference (Zhang et al., 5 Feb 2026). OmniCodec similarly implements semantic–acoustic decoupling by assigning the first codebook to a semantic vector quantizer derived from a pretrained audio understanding encoder and using residual acoustic codebooks for fine detail at 6.25–12.5 Hz (Hu et al., 21 Mar 2026). SECodec approaches the same broad objective from an information-theoretic direction by constructing adaptive codebooks through 2D structural entropy minimization and quantizing by minimizing the increase in structural entropy rather than by Euclidean nearest-neighbor assignment (Wang et al., 2024). DM-Codec addresses a different but related target: it distills acoustic, semantic, and contextual information into RVQ tokens through LM-guided and LM+SM-guided distillation while keeping inference identical to a standard RVQ codec (Ahasan et al., 2024). Separately, work on audio codec-based speech separation performs separation in the embedding space of a pretrained neural audio codec rather than defining a speech codec itself; that line uses codec latents as the domain of separation, not as a single-codebook streaming speech representation (Yip et al., 2024).

The principal limitations of SecoustiCodec are explicit. Paralinguistics are modeled as a global 3-second vector and therefore do not capture fine-grained rapid emotional or prosodic variations; some residual leakage may remain in the semantic channel. Training data are English and Chinese only, so generalization to more languages remains to be validated. The codec is causal and streaming, but the paper notes that non-causal models may still achieve higher absolute fidelity at high bitrates; SecoustiCodec instead targets the extreme low-bitrate streaming regime. Future work identified by the authors includes unsupervised disentanglement, extension to more languages and domains, and refinement of paralinguistic modeling to capture finer temporal variation (Qiang et al., 4 Aug 2025).

In that sense, SecoustiCodec is best understood not as a generic synonym for neural speech tokenization, but as a specific answer to a narrow technical question: how to make a single-codebook speech codec semantically aligned, reconstructible, and fully causal at ultra-low bitrate. Its contribution lies in showing that these objectives can be pursued simultaneously when semantic, paralinguistic, and acoustic factors are explicitly separated rather than compressed into a single undifferentiated token stream (Qiang et al., 4 Aug 2025).