CodecBench: Audio Codec Evaluation Benchmark

• CodecBench is a comprehensive benchmark that evaluates audio codecs based on both acoustic quality (e.g., PESQ, SDR) and semantic preservation (e.g., ASR probing, classification).
• The benchmark uses dual evaluation methods by resynthesizing audio for perceptual metrics while extracting codec embeddings for downstream semantic tasks.
• CodecBench covers diverse data domains including speech, music, sound, and general audio to highlight trade-offs between compression efficiency, reconstruction quality, and contextual retention.

CodecBench is a comprehensive benchmark for evaluating audio codecs from two complementary perspectives: acoustic quality and semantic preservation. It was introduced in response to the growing use of audio codecs as discrete-token front ends for multimodal LLMs and speech LLMs, where a codec is no longer only a compression mechanism but also a representation layer that must retain fidelity, intelligibility, speaker information, emotion, context, and other semantically relevant cues across diverse scenarios (Deng et al., 28 Aug 2025).

1. Conceptual basis and problem setting

CodecBench is organized around the distinction between acoustic information and semantic information. In the benchmark’s formulation, acoustic quality concerns how well a codec reconstructs audio signals perceptually and at the waveform or spectral level, whereas semantic preservation concerns whether codec representations retain information useful for language, context, emotion, speaker identity, and related properties needed by speech-LLMs (Deng et al., 28 Aug 2025).

The benchmark addresses a specific mismatch in contemporary evaluation practice. Traditional codec evaluation emphasizes reconstruction in controlled settings, often on clean speech and with relatively narrow metric sets. CodecBench instead targets the conditions under which audio codecs are increasingly deployed: scenarios with multiple speakers, background noise, richer paralinguistic information, music, non-linguistic sounds, and mixed real-world audio. The benchmark therefore evaluates codecs not only as signal compressors but also as representation systems whose outputs may need to support ASR, TTS, emotion recognition, speaker identity retention, music and sound understanding, and general audio reasoning (Deng et al., 28 Aug 2025).

Its workflow reflects that dual objective. The datasets are first resynthesized by codec models and then scored with acoustic metrics. In parallel, the codec’s embeddings or token representations are used for semantic evaluation. This division is central to the benchmark’s design: a codec may reconstruct audio well while still discarding information needed for downstream semantic modeling, and CodecBench is constructed to expose exactly that discrepancy (Deng et al., 28 Aug 2025).

2. Coverage across speech, music, sound, and general audio

CodecBench covers four data domains—Speech, Music, Sound, and General Audio—using 18 open-source datasets and 1 self-collected dataset, distributed as 6 speech datasets, 3 music datasets, 5 sound datasets, and 5 general audio datasets. The paper presents this breadth as one of the benchmark’s defining features because codec behavior changes materially across vocal, non-vocal, structured, and mixed-content conditions (Deng et al., 28 Aug 2025).

In the benchmark’s terminology, speech is human-generated audio whose linguistic content is carried by phonemes and prosody; music is structured audio with melody, rhythm, and harmony; sound covers non-linguistic, non-musical audio from humans, animals, or the environment; and general audio refers to mixtures or complex real-world audio that can combine speech, music, and sound. The self-collected Bilibili dataset is notable because it targets difficult cases such as quarrels, background music with speech, vocal overlays, and highly expressive dialogue (Deng et al., 28 Aug 2025).

Domain	Characterization	Example datasets
Speech	Human-generated audio with linguistic content carried by phonemes and prosody	KeSpeech; LibriSpeech; Libri2Mix; MELD; RAVDESS; CREMA-D
Music	Structured audio with melody, rhythm, and harmony	NSynth; GTZAN; Musical Instrument Chord Classification
Sound	Non-linguistic, non-musical audio from humans, animals, or the environment	Laughterscape; VocalSound; ESC-50; CatDog; Gunshot Triangulation
General Audio	Audio mixtures or complex real-world audio combining speech, music, and sound	AudioSet; Air-Bench Chat; WavCaps Soundbible; Clotho-AQA; self-collected Bilibili dataset

The domain choices are tightly coupled to downstream use. The speech datasets support evaluation for tasks such as ASR and TTS. The sound and general-audio subsets are intended to probe robustness in noisy, expressive, multi-source settings. The music subset extends evaluation beyond speech-centric codec assessment. A plausible implication is that CodecBench is designed less as a narrow quality leaderboard than as a stress test for codec representations under heterogeneous real-world operating conditions (Deng et al., 28 Aug 2025).

3. Acoustic evaluation protocol

The acoustic component evaluates the fidelity and perceptual quality of reconstructed audio. For fair comparison, all audio was resampled to 16 kHz. The metric set is broad and explicitly heterogeneous: Mel Loss measures Mel spectrogram difference for perceptual quality; PESQ measures speech quality for perceptual accuracy; Spectral Convergence (SC) measures spectral feature difference for frequency accuracy; SDR measures signal-to-distortion ratio for audio quality; SI-SDR measures scale-invariant signal quality for robustness; Speaker Similarity (SIM) measures speaker identity retention for speech synthesis; STOI measures speech intelligibility for noisy environments; and ViSQOL measures speech or audio perceptual quality. The benchmark marks directionality explicitly: lower is better for Mel Loss and SC, and higher is better for PESQ, SDR, SI-SDR, SIM, STOI, and ViSQOL (Deng et al., 28 Aug 2025).

The paper reports 14 audio codec models for the acoustic evaluation. The models include DAC, MaskGCT Codec, BigCodec, Mimi, Stable-Codec, X-Codec-2.0, FlowDec, and Baichuan-Audio tokenizer. For each codec, the benchmark also records bitrate, sample rate, and the number of quantizers (nq). That design makes the benchmark explicitly comparative across codec families rather than confined to a single architectural lineage (Deng et al., 28 Aug 2025).

Acoustic evaluation in CodecBench is not limited to clean speech reconstruction. Because the datasets span speech, music, sound, and general audio, the metric suite is applied over materially different signal classes. This is important for interpreting results: the same codec may behave differently on intelligibility-oriented measures such as STOI, identity-oriented measures such as SIM, and broader fidelity measures such as SDR or ViSQOL. The benchmark therefore treats acoustic quality as multi-dimensional rather than reducible to a single scalar score (Deng et al., 28 Aug 2025).

4. Semantic evaluation methodology

CodecBench uses two semantic evaluation methods: an ASR probing task and a classification task. Both are embedding-oriented rather than reconstruction-oriented. The semantic question is not merely whether the waveform sounds plausible after resynthesis, but whether the codec’s internal representation preserves task-relevant information (Deng et al., 28 Aug 2025).

The ASR probing task is adapted from the SUPERB framework. The pretrained codec is frozen, its quantized embeddings are extracted, and the embeddings are upsampled to a minimum frame rate of 50 Hz via replication before being fed into the downstream ASR model. The downstream model is a two-layer bidirectional LSTM trained with CTC loss for character-level prediction. Training uses LibriSpeech train-clean-100; evaluation uses LibriSpeech dev-clean; and the primary metric is Word Error Rate (WER). The optimization settings are batch size 4, max learning rate $1\times10^{-4}$, and 400,000 training steps. The appendix explains the upsampling requirement in terms of CTC alignment constraints, stating that $T \geq U$ and, in the worst case, $T \geq 2U + 1$, where $T$ is the input sequence length and $U$ is the target label length (Deng et al., 28 Aug 2025).

The classification task is intended to capture broader semantic cues that ASR probing does not measure adequately, especially contextual, emotional, and paralinguistic information. The procedure is to extract codec embeddings, train classifiers on labeled datasets, and evaluate classification accuracy. The speech datasets are RAVDESS with 8 labels, CREMA-D with 6 labels, and MELD with 7 labels. The music datasets are GTZAN with 10 labels, Musical Instrument Chord Classification with 2 labels, and NSynth with 10 labels. The sound datasets are ESC-50 with 50 labels and VocalSound with 6 labels. Training uses batch size 16, max learning rate $1\times10^{-3}$, and 20 epochs per dataset (Deng et al., 28 Aug 2025).

The paper also reports a token-based semantic experiment. For models with a single large codebook, such as Stable-Codec and X-Codec-2.0, learning a mapping from the very high-dimensional token space to a unified embedding space was described as nearly infeasible using only LibriSpeech. For Mimi, by contrast, token-based results broadly matched the embedding-based trends. This result is methodologically important because it indicates that embedding-based evaluation is practical across codec families, whereas token-based evaluation may depend strongly on codebook design (Deng et al., 28 Aug 2025).

5. Empirical findings and codec trade-offs

CodecBench reports a consistent separation between acoustic and semantic behavior. On the acoustic side, DAC tends to perform best overall at higher bitrates, especially on PESQ, STOI, SIM, and ViSQOL. The paper also reports that DAC-44k-rvq9 can outperform DAC variants on MSE, SDR, and SI-SDR, which it attributes to higher frame rates improving temporal resolution. Stable-Codec generally underperforms, especially at low bitrates. BigCodec is strong in single-codebook, low-bitrate settings. FlowDec and Baichuan-Audio tokenizer are reported as competitive among flow-matching approaches. Across domains, music and sound are harder than speech at low bitrate, and degradation is sharper on non-vocal data (Deng et al., 28 Aug 2025).

The semantic findings are more diagnostic. The benchmark shows that strong acoustic reconstruction does not necessarily imply strong semantic preservation. In particular, DAC and BigCodec, despite good acoustic performance, show relatively poor WER in ASR probing. Conversely, X-Codec-2.0 achieves better WER in some cases and can show stronger SIM than Mimi at similar WER, yet it performs poorly on the classification tasks. The paper hypothesizes that single-codebook, low-bitrate designs may prioritize text-related semantics while sacrificing broader paralinguistic information, whereas models with larger codebooks can preserve more contextual and emotional information even without explicit semantic supervision (Deng et al., 28 Aug 2025).

These results lead to a general trade-off structure. The benchmark highlights tensions between compression efficiency and reconstruction quality, and between acoustic fidelity and semantic richness. It also identifies several current limitations of audio codecs: weak performance on complex real-world audio, especially in noisy, expressive, multi-speaker, or mixed-audio conditions; poor handling of non-vocal audio at low bitrate; and persistent under-modeling of semantic information. The paper further notes that the present semantic evaluation remains incomplete and states that future work will expand semantic evaluation and refine CodecBench (Deng et al., 28 Aug 2025).

6. Relation to prior benchmark ecosystems

CodecBench belongs to a broader family of codec benchmarks, but its emphasis is distinct. "Codec-SUPERB: An In-Depth Analysis of Sound Codec Models" evaluates codec models through representative sound applications and signal-level metrics, spanning speech, general audio, and music, and includes an online leaderboard for community comparison (Wu et al., 2024). "Codec-SUPERB @ SLT 2024: A lightweight benchmark for neural audio codec models" narrows that idea into a lightweight, training-free, license-free protocol built around sampled open and hidden sets, fixed downstream task models, and objective metrics such as PESQ, STOI, SDR, and MelLoss (Wu et al., 2024). "OpenACE: An Open Benchmark for Evaluating Audio Coding Performance" focuses on open and reproducible audio and speech coding quality assessment, with full-band content, seven public datasets, ViSQOL Audio, POLQA, and MUSHRA-based subjective evaluation (Coldenhoff et al., 2024).

Against that background, CodecBench is characterized by its explicit separation of acoustic and semantic evaluation and by its coverage of speech, music, sound, and general audio within a single benchmark (Deng et al., 28 Aug 2025). Relative to OpenACE, it moves beyond speech and audio quality assessment into embedding-based semantic probing. Relative to Codec-SUPERB and its SLT 2024 variant, it places stronger emphasis on codec representations as inputs to downstream semantic models rather than only on resynthesized audio scored by frozen application models. This suggests a broader change in benchmark design: codec evaluation is increasingly being framed not only around perceptual reconstruction but also around the preservation of information required for speech-language-model integration.

In that sense, CodecBench marks a shift from judging codecs solely as compression systems to judging them as multi-purpose representational interfaces. Its central contribution is not a single metric or leaderboard statistic, but a benchmark structure that exposes when acoustic quality and semantic usefulness diverge—and therefore where current codec design remains insufficient for modern multimodal applications.