SAQ-Decoder in SAC Speech Codec

• SAQ-Decoder is the decoding component of SAC that fuses discrete semantic and acoustic streams to produce a 16 kHz waveform.
• It employs a ConvNeXt-based fusion prenet and mirrored decoder architecture to reconstruct both waveform and auxiliary features.
• It balances reconstruction quality using multi-scale L1, VQ, and adversarial losses while ensuring semantic and speaker fidelity.

Searching arXiv for papers mentioning “SAQ-Decoder” and the SAC codec paper to ground the article in the relevant literature. SAQ-Decoder denotes the decoder component of SAC, a neural speech codec with semantic-acoustic dual-stream quantization, in which semantic and acoustic discrete token streams are fused and mapped back to a 16 kHz waveform. In the SAC formulation, the decoder is responsible not only for waveform reconstruction but also for auxiliary recovery of fine-grained continuous semantic features and estimation of speaker embeddings from the fused latent sequence. The design is motivated by the observation that existing speech codecs often struggle to balance high-quality reconstruction with semantically rich representations; SAC addresses this by disentangling semantic and acoustic modeling into two dedicated streams (Chen et al., 19 Oct 2025).

1. Definition and placement within SAC

Within SAC, the decoder operates downstream of two quantized streams. The semantic stream consists of quantized semantic embeddings $S_q'$ at 50 Hz. The original semantic features $S_c$ at 50 Hz are extracted by a frozen pre-trained speech tokenizer, then pooled to 12.5 Hz, quantized to tokens, embedded to $S_q$, and up-sampled back to 50 Hz via a small ConvNeXt adapter to produce $S_q'$. The acoustic stream consists of quantized acoustic embeddings $A_q$ at either 25 Hz in the low-bitrate setting or 50 Hz in the high-bitrate setting, obtained by an Encodec-style encoder with temporal downsampling strides $\tau=(2,2,4,5,8)$ or $\tau=(2,4,5,8)$ and single-codebook quantization (Chen et al., 19 Oct 2025).

The decoder therefore sits at the convergence point of SAC’s dual-stream representation. Its immediate input is not raw discrete indices, but continuous vectors recovered from those indices: $S_q'$ for semantics and $A_q$ for acoustics. This arrangement is central to the codec’s claim that semantic and acoustic factors can be optimized for distinct roles while remaining jointly decodable.

2. Decoder architecture

The decoder begins with a fusion prenet. The semantic and acoustic streams are concatenated along the channel dimension to form $U$, which is then passed through a ConvNeXt-based prenet. This prenet temporally upsamples to 50 Hz if needed and produces the fused feature sequence $S_c$0 (Chen et al., 19 Oct 2025).

The main waveform decoder is a mirrored decoder relative to the Encodec-style encoder. The fused sequence $S_c$1 is processed by a stack of convolutional layers interleaved with transposed-convolution upsampling layers. The deconvolution strides are $S_c$2, exactly inverting the encoder’s downsampling so that the final output is a 16 kHz waveform $S_c$3 (Chen et al., 19 Oct 2025).

Two auxiliary decoder-side heads are attached to the same fused representation. First, an auxiliary semantic decoder applies a small CNN to $S_c$4 in order to reconstruct fine-grained continuous semantic features $S_c$5 at 50 Hz. Second, an auxiliary speaker predictor summarizes $S_c$6 using temporal mean and standard deviation,

$S_c$7

and passes this statistic through a two-layer MLP, denoted “Proj,” to produce $S_c$8, an estimate of the speaker embedding (Chen et al., 19 Oct 2025).

This architecture makes the fused representation multi-purpose: it drives waveform synthesis, semantic feature reconstruction, and speaker inference. A plausible implication is that SAC does not treat speech reconstruction as a purely signal-level inversion problem, but as a jointly constrained reconstruction problem in waveform, semantic, and speaker spaces.

3. Mathematical objectives and training signals

The waveform reconstruction loss is defined as a multi-scale $S_c$9 objective on both linear- and log-scale spectrograms of $S_q$0 versus $S_q$1:

$S_q$2

The paper notes that no closed-form equation is given in the text and that the formulation follows DAC (Chen et al., 19 Oct 2025).

For the acoustic stream, vector quantization is trained with

$S_q$3

where $S_q$4 is the commitment weight, fixed at $S_q$5, $S_q$6 is the stop-gradient operator, and an additional codebook loss weight $S_q$7 scales the first term (Chen et al., 19 Oct 2025).

Adversarial supervision is provided by two discriminators, Multi-Period and Multi-Scale STFT, trained with least-squares GAN. The generator therefore also receives adversarial loss $S_q$8 and feature-matching loss $S_q$9, where the latter is an $S_q'$0 distance between discriminator feature maps (Chen et al., 19 Oct 2025).

The semantic auxiliary loss is

$S_q'$1

with $S_q'$2 predicted by the CNN decoder and $S_q'$3 given by the frozen tokenizer’s continuous semantic features. The speaker auxiliary loss is

$S_q'$4

where $S_q'$5 is the frozen ERes2Net speaker embedding (Chen et al., 19 Oct 2025).

These terms are combined into the generator objective

$S_q'$6

The reported weights are $S_q'$7, $S_q'$8, $S_q'$9, $A_q$0, $A_q$1, and $A_q$2 (Chen et al., 19 Oct 2025).

4. De-quantization, continuous reconstruction, and implementation parameters

The decoder’s de-quantization procedure is direct. In the semantic stream, each discrete token index $A_q$3 selects the $A_q$4-th row of the frozen semantic tokenizer’s codebook to produce embedding $A_q$5, after which a ConvNeXt upsampler yields $A_q$6. In the acoustic stream, each frame’s continuous embedding $A_q$7 is replaced by its nearest codebook entry $A_q$8 through $A_q$9 search in the learned single codebook of size $\tau=(2,2,4,5,8)$0. At decoding time, no further inverse quantization is needed, because $\tau=(2,2,4,5,8)$1 and $\tau=(2,2,4,5,8)$2 are already continuous vectors; these are concatenated and passed into the prenet (Chen et al., 19 Oct 2025).

Several decoder-side hyperparameters characterize the implementation. The semantic codebook size is $\tau=(2,2,4,5,8)$3, and the acoustic codebook size is also $\tau=(2,2,4,5,8)$4. Semantic features are pooled from 50 Hz to 12.5 Hz before quantization and restored to 50 Hz after upsampling. Acoustic features operate at 25 Hz for low bitrates and 50 Hz for high bitrates. The generator has approximately $\tau=(2,2,4,5,8)$5 million parameters, of which approximately $\tau=(2,2,4,5,8)$6 million are trainable, while the semantic tokenizer and speaker encoder remain frozen (Chen et al., 19 Oct 2025).

Training uses AdamW with $\tau=(2,2,4,5,8)$7, $\tau=(2,2,4,5,8)$8, initial learning rate $\tau=(2,2,4,5,8)$9, and exponential decay. The generator is pretrained for $\tau=(2,4,5,8)$0 steps before the discriminator is activated, and EMA is used (Chen et al., 19 Oct 2025).

Component	Reported specification
Prenet	ConvNeXt-based; exact number of layers not specified
Main decoder strides	$\tau=(2,4,5,8)$1
Semantic codebook	$\tau=(2,4,5,8)$2
Acoustic codebook	$\tau=(2,4,5,8)$3
Acoustic frame rate	25 Hz or 50 Hz
Output	16 kHz waveform $\tau=(2,4,5,8)$4

A plausible implication is that the decoder is intentionally structured so that semantic and acoustic streams meet only after each has already been converted into a continuous latent trajectory, thereby simplifying the fusion problem to one of feature integration rather than symbolic sequence transduction.

5. Reconstruction quality and bitrate regimes

SAC evaluates reconstruction with STOI, PESQ-NB, PESQ-WB, UTMOS, Speaker SIM, and WER. At the high-bitrate setting, corresponding to a 62.5 Hz token rate and 875 bps, the reported scores are UTMOS $\tau=(2,4,5,8)$5, WER $\tau=(2,4,5,8)$6, SIM $\tau=(2,4,5,8)$7, PESQ-NB $\tau=(2,4,5,8)$8, PESQ-WB $\tau=(2,4,5,8)$9, and STOI $S_q'$0. The paper states that this UTMOS is the best among approximately 1 kbps codecs and even above ground-truth $S_q'$1, while the WER is close to ground truth $S_q'$2 (Chen et al., 19 Oct 2025).

At the low-bitrate setting, corresponding to a 37.5 Hz token rate and 525 bps, the reported scores are UTMOS $S_q'$3, WER $S_q'$4, SIM $S_q'$5, PESQ-NB $S_q'$6, PESQ-WB $S_q'$7, and STOI $S_q'$8 (Chen et al., 19 Oct 2025).

Under noisy conditions on LibriSpeech test-other, SAC is reported to still top UTMOS, with values $S_q'$9, maintain low WER at approximately $A_q$0, and retain SIM of approximately $A_q$1–$A_q$2 (Chen et al., 19 Oct 2025).

Setting	Token rate / bitrate	Key reported outcomes
High bitrate	62.5 Hz, 875 bps	UTMOS 4.25, WER 2.35%, SIM 0.86
Low bitrate	37.5 Hz, 525 bps	UTMOS 4.27, WER 2.53%, SIM 0.78
Noisy test-other	not separately restated	UTMOS 3.84/3.90, WER $A_q$35.8/6.4%, SIM $A_q$40.77–0.85

These results are presented as evidence that the decoder preserves perceptual quality and intelligibility across bitrate regimes and in noisy conditions. Because the decoder is the component that fuses the semantic and acoustic streams back into waveform space, the reported quality metrics also function as an empirical validation of the decoder design itself.

6. Ablations, disentanglement, controllability, and nomenclature

The paper’s ablation studies directly probe the decoder’s role in disentanglement. Removing the speaker loss by setting $A_q$5 causes SIM to drop from $A_q$6 to $A_q$7, while PESQ increases slightly. Removing the semantic loss by setting $A_q$8 has negligible effect on STOI, PESQ, UTMOS, and WER, which the paper interprets as confirming that the frozen semantic stream plus the dual-stream design already preserves semantics (Chen et al., 19 Oct 2025).

Semantic-only and acoustic-only reconstructions further characterize the decoder. When the acoustic stream $A_q$9 is masked, the semantic-only output yields WER $U$0 versus SemantiCodec’s $U$1, SIM $U$2, and MSIM $U$3, which the paper describes as indicating strong lexical preservation and clean timbre removal. When the semantic stream $U$4 is masked, the acoustic-only output is noise-like and has zero intelligibility. Full reconstruction retains both high-fidelity timbre and semantics. The paper concludes that this confirms the decoder’s ability to fuse or ignore each stream cleanly (Chen et al., 19 Oct 2025).

These ablations suggest that the decoder is not merely combining two redundant latent codes. Rather, the semantic stream appears to carry lexical content in a form that remains intelligible even when speaker identity is largely removed, whereas the acoustic stream alone is insufficient for intelligible reconstruction. This suggests a decoder-level factorization of speech content and timbre that may be useful for controllable speech applications, a possibility explicitly noted in the SAC analysis (Chen et al., 19 Oct 2025).

The term “SAQ-Decoder” requires care because it is not unique across the 2025 arXiv literature. In the SAC paper, it refers to the decoder just described. However, the same label also appears in unrelated domains: as a simulated-annealing decoder for the XZZX code (Sakashita, 22 Sep 2025), and as a stabilizer-aware quantum error-correction decoder built from a dual-stream transformer and constraint-aware post-processing (Zenati et al., 9 Dec 2025). In speech codec literature, the relevant referent is the SAC decoder of semantic-acoustic dual-stream quantization (Chen et al., 19 Oct 2025).