SimWhisper-Codec: Neural Speech Codec

• SimWhisper-Codec is a neural speech codec that employs a semantic-first design by simplifying the Whisper ASR encoder to preserve spectral detail and semantic alignment.
• It streamlines quantization and decoding by removing GELU activations and absolute positional encodings, thereby enhancing acoustic reconstruction.
• Empirical results demonstrate superior word error rates and perceptual quality metrics compared to semantically-supervised codecs at similar low bitrates.

SimWhisper-Codec is a neural speech codec designed to achieve an optimal balance between semantic preservation and acoustic fidelity at very low bitrates through targeted simplification of the Whisper automatic speech recognition (ASR) model. In contrast to prevailing approaches that introduce additional semantic supervision to acoustic codecs, SimWhisper-Codec adopts a "semantic-first" strategy: it leverages the inherent semantic capacity of the Whisper encoder, removing architectural impediments to acoustic detail, and integrates an efficient quantization and decoding path. The methodology demonstrates superior performance in both word error rate (WER) and perceptual audio quality compared to semantically-supervised codecs at comparable bitrates.

1. Architectural Simplification of Whisper

SimWhisper-Codec departs from the Whisper ASR model's original configuration by applying two critical modifications to its encoder:

• Removal of Convolutional Front-End Nonlinearity:

The original Whisper front-end applies GELU activations after two convolutional layers to enhance ASR-relevant feature abstraction. However, these nonlinearities suppress fine-grained spectral details necessary for high-fidelity acoustic reconstruction. In SimWhisper-Codec, these activations are removed, rendering the convolutional layers strictly linear and preserving the spectral structure essential for perceptual quality metrics such as PESQ and STOI.

• Elimination of Absolute Positional Encodings:

Whisper utilizes absolute positional encodings in its Transformer blocks to encode sequence order. SimWhisper-Codec omits these encodings, which, while beneficial for ASR, enforce a location-specific content representation that impedes generalization over repetitive acoustic patterns. By removing these encodings, the self-attention mechanism can prioritize the signal content itself, leading to improved acoustic reconstruction.

A schematic depiction of these modifications:

[Original Whisper Encoder] [SimWhisper-Codec Encoder] ├→ Conv + GELU ├→ Conv (linear) ├→ Conv + GELU ├→ Conv (linear) ├→ Transformer + Abs PE ├→ Transformer (no Abs PE) └→ ... └→ ...

2. Semantic-First Codec Design Paradigm

Unlike codecs that superimpose extrinsic semantic objectives onto acoustic modeling, SimWhisper-Codec relies on the semantic alignment already present in Whisper. The encoder is frozen during training, ensuring that its intrinsic semantic structure is preserved; only the downstream quantizer and decoder are learned to recover the acoustic waveform. This design avoids the need for additional semantic distillation, multitask losses, or supervision from external models such as HuBERT or wav2vec 2.0, as seen in semantically-supervised alternatives (e.g., XCodec2.0, Mimi-RVQ8).

This semantic-first approach directly leverages Whisper’s text-aligned embeddings, with empirical results confirming that architectural simplification suffices to maintain low WER while boosting acoustic naturalness.

3. Quantization and Reconstruction Pathway

SimWhisper-Codec features a streamlined quantization and decoding process:

• Downsampler: Aggregates frames and compresses feature dimensionality using a sequence of residual blocks with dilated convolutions and Snake activation functions.
• Finite Scalar Quantization (FSQ): Replaces traditional vector quantization with FSQ to avoid codebook collapse and simplify the training process (removing the need for exponential moving averages or codebook commitment penalties). The FSQ operates on the linearly projected encoder features, discretizing them into low-bitrate representations.
• Upsampler and Decoder: Employs symmetric upsampling—nearest-neighbor and transposed convolution layers—to restore the original temporal resolution, followed by a decoder mirroring the encoder structure (with transposed convolutions replacing convolutional layers).

This design maintains architectural symmetry and allows efficient, real-time waveform reconstruction.

4. Training Objectives and Losses

SimWhisper-Codec is trained end-to-end (excluding the frozen encoder) with a generator objective comprising three components:

• Multi-Scale Reconstruction Loss: Ensures fidelity across various time-frequency scales between the reconstructed and original waveforms.
• Adversarial Loss: Uses a Least Squares GAN objective to enhance perceptual quality and reduce artifacts.
• Feature Matching Loss: Encourages the generator to match intermediate discriminator feature maps, improving convergence and perceptual realism.

The total generator loss is expressed as:

$$\mathcal{L}_G = \lambda_\mathrm{recon}\mathcal{L}_\mathrm{recon} + \lambda_\mathrm{adv}\mathcal{L}_\mathrm{adv} + \lambda_\mathrm{feat}\mathcal{L}_\mathrm{feat}$$

where the weights $\lambda_i$ are empirically tuned.

5. Comparative Performance Metrics

SimWhisper-Codec’s empirical validation centers on bitrate (1.1 kbps), semantic fidelity, and perceptual audio quality. Key reported outcomes include:

Model	WER	SIM	STOI	PESQ-NB	PESQ-WB
SpeechTokenizer (sem. supervised)	5.92	0.37	0.70	1.42	1.15
Mimi-RVQ8 (sem. supervised)	4.36	0.73	0.90	2.62	2.13
XCodec2.0 (sem. supervised)	3.61	0.82	0.91	2.95	2.32
SimWhisper-Codec (sem.-first)	3.10	0.83	0.91	2.98	2.36

WER is assessed using transcriptions obtained from an external ASR model. Speaker similarity (SIM), intelligibility (STOI), and perceptual quality (PESQ-NB, PESQ-WB) consistently demonstrate the effectiveness of the architectural simplification and the semantic-first paradigm. Ablation studies confirm cumulative improvements from both removing the GELU nonlinearity and eliminating absolute positional encodings.

6. Technical Implementation and Open Source Availability

SimWhisper-Codec is implemented around the Whisper small configuration, with architectural changes localized to the front-end and attention mechanisms. Model training involves only those modules responsible for quantization and decoding, with the encoder frozen. The design avoids reliance on external semantic targets or multitask objectives.

The codebase, complete with setup instructions, training scripts, and reproducibility documentation, is publicly accessible at:

• https://github.com/ZhangXinWhut/SimWhisper-Codec

This enables rigorous benchmarking and rapid adoption in research focused on low-bitrate semantic-preserving speech coding.

7. Relationship to Prior Approaches and Broader Implications

SimWhisper-Codec positions itself with respect to two dominant lines of codec research:

• Semantically-Supervised Codecs: Approaches such as X-Codec integrate semantic encoders (e.g., HuBERT, wav2vec 2.0) with explicit semantic reconstruction losses to augment the semantic capacity of acoustic tokens (Ye et al., 30 Aug 2024).
• Acoustic-Driven Codecs with Voice Conversion: Early work focuses on deterministic DSP modifications and voice conversion to achieve specialized audio transformations, such as phonated-to-whispered speech conversion via DNN- or GMM-based mapping (Cotescu et al., 2019).

In contrast, SimWhisper-Codec demonstrates that targeted architectural simplification of a semantically-trained encoder allows for superior trade-offs—achieving low WER and high perceptual scores—without additional supervisory signals. A plausible implication is that large-scale ASR models, when stripped of certain inductive biases, may serve as optimal neural codec backbones for a diverse array of speech transmission and synthesis applications, particularly where semantic integrity and acoustic detail must simultaneously be preserved.

SimWhisper-Codec thus reframes codec design through the principle of minimalist adaptation, capitalizing on the latent semantic capacity of existing ASR encoders while enabling efficient, high-quality waveform reconstruction at extreme compression ratios (Zhang et al., 23 Oct 2025).