S3Codec: Split RVQ Codec for Semantic Speech

• S3Codec is a split residual vector quantization speech codec that injects explicit linguistic features into its primary codebook while decoupling semantic and acoustic quantization.
• It achieves high audio fidelity at a low 1.2kbps bitrate by leveraging ASR teacher supervision for robust semantic distillation and fine acoustic detail reconstruction.
• Integrated within the CaT-TTS framework, S3Codec reduces text-to-speech modeling challenges, yielding improved WER, speaker similarity, and overall TTS performance.

S3Codec is a Split Residual Vector Quantization (RVQ) speech codec with Semantic Distillation designed to produce discrete audio tokens that retain both fine-grained acoustic information and high-level linguistic semantics (Cao et al., 26 Sep 2025). It is presented within the CaT-TTS framework as a response to a specific trade-off in neural audio codecs: single-codebook codecs are described as rich in semantics but limited by poor reconstruction and missing acoustic detail, whereas multi-codebook (RVQ) codecs provide good audio fidelity but poor semantic alignment, increasing the difficulty of downstream text-to-speech learning. S3Codec addresses this by injecting explicit linguistic features into the primary codebook and decoupling the semantic and acoustic quantization paths via a split RVQ structure (Cao et al., 26 Sep 2025).

1. Design objective and conceptual position

The stated objective of S3Codec is to combine high acoustic fidelity, a linguistically meaningful discrete bottleneck, and low bitrate within a single codec formulation (Cao et al., 26 Sep 2025). In the source formulation, this objective is tied directly to the needs of autoregressive text-to-speech systems, where the discrete representation generated by a neural audio codec becomes the modeling substrate for subsequent language-model-based generation.

The codec is positioned between two established design tendencies. On one side are single-codebook codecs, which are characterized as well suited to text LLMs but prone to significant information loss. On the other side are hierarchical acoustic tokens, typically generated via RVQ, which are described as lacking explicit semantic structure and therefore imposing a heavy learning burden on the model. S3Codec is defined as a resolution of this dichotomy through a split RVQ codec in which the first codebook is made explicitly linguistic and the remaining codebooks focus on acoustic reconstruction (Cao et al., 26 Sep 2025).

A central implication of this design is that the codec is not treated merely as a compression mechanism. It is also a representation-learning component whose output is expected to simplify text-audio alignment for downstream generation. The paper states this goal explicitly by framing S3Codec as a codec that preserves both semantic and acoustic content while reducing the modeling burden in CaT-TTS (Cao et al., 26 Sep 2025).

2. Architecture and split RVQ organization

S3Codec is built on the DAC (DeepAudioCodec) architecture (Cao et al., 26 Sep 2025). Its encoder processes a single-channel waveform $\mathbf{x} \in \mathbb{R}^T$ into a latent representation $$\mathbf{A} = \mathrm{enc}(\mathbf{x}) \in \mathbb{R}^{L \times D}$$. The encoder is described as using stacked residual convolutional blocks with a mix of dilated and strided convolutions, Snake activations, and weight normalization, with downsampling via [2, 4, 5, 6, 8]. The decoder mirrors the encoder, reconstructing the waveform with upsampling [8, 6, 5, 4, 2], and the reported internal channel dimension is 2048 (Cao et al., 26 Sep 2025).

The defining architectural component is the Split RVQ Quantizer. The quantization path is explicitly divided to separate semantic (linguistic) and acoustic codebooks. The first codebook is a standard VQ and receives direct semantic distillation from a pretrained ASR model (Whisper). The subsequent $K-1$ codebooks form an RVQ, act in parallel, reconstruct the residual, and focus on fine acoustic detail. The codebook outputs are summed, with the representation described as $\mathbf{C} \in \mathbb{R}^{K \times L \times D}$ (Cao et al., 26 Sep 2025).

This organization is contrasted with classic RVQ or hierarchical quantizers, which force the entire stack to reconstruct audio and can thereby dilute semantic information. In S3Codec, the split arrangement is used so that direct semantic injection occurs in the first codebook without requiring the later codebooks to serve the same representational role. The design therefore separates semantic structuring from residual acoustic refinement rather than requiring one codebook stack to satisfy both objectives simultaneously (Cao et al., 26 Sep 2025).

3. Semantic distillation and optimization

The semantic component of S3Codec is implemented through semantic distillation via ASR teacher supervision, with Whisper used as the teacher rather than a self-supervised SSL model such as HuBERT (Cao et al., 26 Sep 2025). The stated reason is that Whisper is state-of-the-art for ASR and imbues clear linguistic features. For each audio frame $t$, the first-quantizer embedding $\mathbf{C}_t^0$ is trained to align with a projected Whisper embedding through a cosine-distance objective. This makes the first codebook explicitly linguistic, while the residual codebooks remain dedicated to audio fidelity (Cao et al., 26 Sep 2025).

Training uses a multi-task loss that combines several terms. The codec includes a time-domain reconstruction loss $$\mathcal{L}_t = \| \mathbf{x} - \hat{\mathbf{x}} \|_1$$, a frequency-domain reconstruction loss $\mathcal{L}_f$ based on multi-scale Mel-spectrograms, a GAN loss, a feature matching loss, an RVQ commitment loss, and the semantic distillation loss. The adversarial component uses multi-period and complex multi-scale STFT discriminators, with hinge loss for discriminator and generator and feature matching loss (Cao et al., 26 Sep 2025).

The reported semantic-distillation weight is $\lambda_{\text{distill}} = 0.1$. Optimization uses AdamW with $\beta_1 = 0.8$ and $$\mathbf{A} = \mathrm{enc}(\mathbf{x}) \in \mathbb{R}^{L \times D}$$0, and the training run is described as converging in ≈900K steps with batch size 128 (Cao et al., 26 Sep 2025).

These choices locate S3Codec within a familiar GAN-based neural codec training regime, but with the semantic distillation term making the first quantizer structurally different from the remaining RVQ stages. A plausible implication is that the codec is trained not only to reconstruct audio accurately but also to expose a more useful first-layer token stream for downstream modeling; this interpretation follows directly from the stated role of the first codebook and the downstream CaT-TTS results (Cao et al., 26 Sep 2025).

4. Quantization regime and empirical characteristics

The implementation details reported for S3Codec are specific: it operates on 24kHz audio, uses 8 codebooks, and each codebook has 4096 entries (Cao et al., 26 Sep 2025). The codec quantizes audio at 12.5Hz (frame rate), which is described as lower than typically used and therefore beneficial for compression (Cao et al., 26 Sep 2025).

The codec is compared against Encodec, DAC, SpeechTokenizer, BigCodec, Xcodec, MBCodec, Mimi, and S3Codec under the metrics PESQ, STOI, STFT & Mel, and SIM (Cao et al., 26 Sep 2025). The reported objective result for S3Codec is:

Measure	S3Codec
Bitrate (bps)	1.2k
PESQ	2.85
STOI	0.94
SIM	0.89
STFT	0.12
Mel	4.01

Within the reported summary, S3Codec at 1.2kbps is said to achieve comparable or better perceptual quality and intelligibility to high-bitrate codecs (e.g. Encodec, DAC) and much higher speaker similarity and lower WER than semantic-only codecs like SpeechTokenizer (Cao et al., 26 Sep 2025). The data block also reports representative comparison points: MBCodec at 2.2k with PESQ 2.98, STOI 0.94, SIM 0.87, STFT 0.17, Mel 3.62; Mimi at 1.1k with PESQ 2.24, STOI 0.90, SIM 0.73; SpeechTokenizer at 1k with PESQ 1.25, STOI 0.77, SIM 0.36, STFT 0.68, Mel 8.02; and DAC-8 at 6k with PESQ 3.46, STOI 0.95, SIM 0.96, STFT 0.06, Mel 2.02 (Cao et al., 26 Sep 2025).

These results are used in the source to support two specific claims: first, that S3Codec maintains strong speaker similarity at a very low bitrate; second, that its discrete representation preserves more linguistic information than semantic-only codecs while remaining substantially more compressive than higher-bitrate reconstruction-oriented codecs (Cao et al., 26 Sep 2025).

5. Function within CaT-TTS

Within CaT-TTS, S3Codec provides the discrete audio representation that feeds the system’s “Understand-then-Generate” dual-Transformer architecture (Cao et al., 26 Sep 2025). The framework is described as using an initial “Understanding” Transformer to model the cross-modal relationship between text and the audio’s semantic tokens and form a high-level utterance plan, followed by a “Generation” Transformer that autoregressively synthesizes hierarchical acoustic tokens. In this system, S3Codec is the source of the token sequence that mediates between text conditioning and waveform reconstruction (Cao et al., 26 Sep 2025).

The source attributes several downstream advantages to this codec design. Because linguistic information is injected into the main codebook, the upstream semantic transformer can easily correlate text with audio tokens, thereby reducing the LLM’s modeling burden and improving sample efficiency and stability (Cao et al., 26 Sep 2025). The codec is therefore not only a front-end tokenizer but also a structural prior for dual language modeling.

An ablation comparison between a DAC-Based system and an S3Codec-Based system reports lower WER for the S3Codec variant on all three listed test sets:

Model	SeedTTS-test WER	PGC-Hard WER	PGC-Poly WER
DAC-Based	4.21	12.83	19.27
S3Codec-Based	3.30	9.75	16.53

The source summarizes this result by stating that S3Codec enables significantly better linguistic information preservation (lower WER) (Cao et al., 26 Sep 2025). It further states that CaT-TTS with S3Codec achieves SOTA or competitive WER, SIM, and UTMOS on several test sets, especially in speech intelligibility and generalization (Cao et al., 26 Sep 2025).

6. Relation to benchmarks and similarly named codecs

S3Codec is not part of the published Codec-SUPERB benchmark coverage in the cited benchmark papers. The earlier Codec-SUPERB study lists SpeechTokenizer, AudioDec, AcademiCodec, Descript-Audio-Codec (DAC), Encodec, and FunCodec as the codec models covered and explicitly states that S3Codec is not included, benchmarked, mentioned, or analyzed in any way (Wu et al., 2024). The later Codec-SUPERB @ SLT 2024 challenge likewise reports 5 participant systems (with 13 configurations)—FunCodec, SemantiCodec, APCodec, AFACodec, and SpeechTokenizer—with Encodec as baseline, and states that S3Codec is not among the submitted or evaluated models in this round (Wu et al., 2024).

This absence matters because Codec-SUPERB was designed as a lightweight, training-free and computationally efficient benchmark emphasizing both application-level metrics and objective signal-level metrics under a unified evaluation pipeline (Wu et al., 2024). A plausible implication is that S3Codec had not yet been integrated into that evaluation ecosystem at the time of those releases, even though the benchmark infrastructure could later serve as a standardized environment for assessment.

S3Codec should also be distinguished from TS3-Codec, whose similarity in name can obscure major architectural differences. TS3-Codec is a Transformer-Based Simple Streaming Single Codec with a purely transformer-based, convolution-free architecture, single-codebook VQ, and streaming operation via causal sliding-window attention (Wu et al., 2024). By contrast, S3Codec is built on DAC, uses stacked residual convolutional blocks, and organizes quantization through a split RVQ design in which the first codebook is explicitly semantic and the remaining codebooks reconstruct residual acoustic detail (Cao et al., 26 Sep 2025). The two systems therefore address related codec problems through markedly different architectural commitments: S3Codec emphasizes semantic distillation within RVQ, whereas TS3-Codec emphasizes a transformer-only streaming single-codebook codec (Wu et al., 2024).

7. Significance and interpretation

The significance attributed to S3Codec in its source paper lies in its attempt to unify representational semantics and acoustic fidelity within a single discrete bottleneck (Cao et al., 26 Sep 2025). The codec is described as providing a balanced mix for both comprehension and high-fidelity synthesis, in contrast to systems using only semantic tokens or only acoustic tokens. This balance is central to its role in CaT-TTS, where the codec output is expected to support both cross-modal understanding and robust autoregressive generation (Cao et al., 26 Sep 2025).

Its reported strengths are correspondingly specific: high audio quality at low bitrate, linguistic-structural alignment via ASR-based distillation, improved text-audio mapping, and empirical gains in WER, SIM, and UTMOS relative to baseline codec choices inside the same TTS framework (Cao et al., 26 Sep 2025). The low 12.5Hz frame rate and 1.2kbps operating point further indicate a design oriented toward compact token sequences rather than only waveform reconstruction (Cao et al., 26 Sep 2025).

A careful reading also limits broader claims. S3Codec’s published evidence in the provided sources is concentrated in the CaT-TTS setting rather than in the general-purpose benchmark literature, and the benchmark papers explicitly do not evaluate it (Wu et al., 2024). Accordingly, the most defensible characterization is that S3Codec is a codec architecture and tokenization strategy specialized to semantically grounded speech generation, with reported benefits in reconstruction, linguistic preservation, and downstream TTS learning, rather than a benchmark-established universal winner across all codec tasks (Cao et al., 26 Sep 2025).