---
title: 'CodecBench: Audio Codec Evaluation Benchmark'
url: https://www.emergentmind.com/topics/codecbench
type: topic
---

# CodecBench: Audio Codec Evaluation Benchmark

CodecBench is a comprehensive benchmark for evaluating audio codecs from two complementary perspectives: acoustic quality and semantic preservation. It was introduced in response to the growing use of audio codecs as discrete-token front ends for multimodal large language models and speech language models, where a codec is no longer only a compression mechanism but also a representation layer that must retain fidelity, intelligibility, speaker information, emotion, context, and other semantically relevant cues across diverse scenarios [2508.20660].

## 1. Conceptual basis and problem setting

CodecBench is organized around the distinction between **acoustic** information and **semantic** information. In the benchmark’s formulation, acoustic quality concerns how well a codec reconstructs audio signals perceptually and at the waveform or spectral level, whereas semantic preservation concerns whether codec representations retain information useful for language, context, emotion, speaker identity, and related properties needed by speech-language models [2508.20660].

The benchmark addresses a specific mismatch in contemporary evaluation practice. Traditional codec evaluation emphasizes reconstruction in controlled settings, often on clean speech and with relatively narrow metric sets. CodecBench instead targets the conditions under which audio codecs are increasingly deployed: scenarios with multiple speakers, background noise, richer paralinguistic information, music, non-linguistic sounds, and mixed real-world audio. The benchmark therefore evaluates codecs not only as signal compressors but also as representation systems whose outputs may need to support ASR, TTS, emotion recognition, speaker identity retention, music and sound understanding, and general audio reasoning [2508.20660].

Its workflow reflects that dual objective. The datasets are first **resynthesized** by codec models and then scored with acoustic metrics. In parallel, the codec’s **embeddings** or **token representations** are used for semantic evaluation. This division is central to the benchmark’s design: a codec may reconstruct audio well while still discarding information needed for downstream semantic modeling, and CodecBench is constructed to expose exactly that discrepancy [2508.20660].

## 2. Coverage across speech, music, sound, and general audio

CodecBench covers **four data domains**—**Speech**, **Music**, **Sound**, and **General Audio**—using **18 open-source datasets and 1 self-collected dataset**, distributed as **6 speech datasets**, **3 music datasets**, **5 sound datasets**, and **5 general audio datasets**. The paper presents this breadth as one of the benchmark’s defining features because codec behavior changes materially across vocal, non-vocal, structured, and mixed-content conditions [2508.20660].

In the benchmark’s terminology, **speech** is human-generated audio whose linguistic content is carried by phonemes and prosody; **music** is structured audio with melody, rhythm, and harmony; **sound** covers non-linguistic, non-musical audio from humans, animals, or the environment; and **general audio** refers to mixtures or complex real-world audio that can combine speech, music, and sound. The self-collected Bilibili dataset is notable because it targets difficult cases such as quarrels, background music with speech, vocal overlays, and highly expressive dialogue [2508.20660].

| Domain | Characterization | Example datasets |
|---|---|---|
| Speech | Human-generated audio with linguistic content carried by phonemes and prosody | KeSpeech; LibriSpeech; Libri2Mix; MELD; RAVDESS; CREMA-D |
| Music | Structured audio with melody, rhythm, and harmony | NSynth; GTZAN; Musical Instrument Chord Classification |
| Sound | Non-linguistic, non-musical audio from humans, animals, or the environment | Laughterscape; VocalSound; ESC-50; CatDog; Gunshot Triangulation |
| General Audio | Audio mixtures or complex real-world audio combining speech, music, and sound | AudioSet; Air-Bench Chat; WavCaps Soundbible; Clotho-AQA; self-collected Bilibili dataset |

The domain choices are tightly coupled to downstream use. The speech datasets support evaluation for tasks such as ASR and TTS. The sound and general-audio subsets are intended to probe robustness in noisy, expressive, multi-source settings. The music subset extends evaluation beyond speech-centric codec assessment. A plausible implication is that CodecBench is designed less as a narrow quality leaderboard than as a stress test for codec representations under heterogeneous real-world operating conditions [2508.20660].

## 3. Acoustic evaluation protocol

The acoustic component evaluates the **fidelity and perceptual quality** of reconstructed audio. For fair comparison, **all audio was resampled to 16 kHz**. The metric set is broad and explicitly heterogeneous: **Mel Loss** measures Mel spectrogram difference for perceptual quality; **PESQ** measures speech quality for perceptual accuracy; **Spectral Convergence (SC)** measures spectral feature difference for frequency accuracy; **SDR** measures signal-to-distortion ratio for audio quality; **SI-SDR** measures scale-invariant signal quality for robustness; **Speaker Similarity (SIM)** measures speaker identity retention for speech synthesis; **STOI** measures speech intelligibility for noisy environments; and **ViSQOL** measures speech or audio perceptual quality. The benchmark marks directionality explicitly: lower is better for Mel Loss and SC, and higher is better for PESQ, SDR, SI-SDR, SIM, STOI, and ViSQOL [2508.20660].

The paper reports **14 audio codec models** for the acoustic evaluation. The models include **DAC**, **MaskGCT Codec**, **BigCodec**, **Mimi**, **Stable-Codec**, **X-Codec-2.0**, **FlowDec**, and **Baichuan-Audio tokenizer**. For each codec, the benchmark also records **bitrate**, **sample rate**, and the number of quantizers (**nq**). That design makes the benchmark explicitly comparative across codec families rather than confined to a single architectural lineage [2508.20660].

Acoustic evaluation in CodecBench is not limited to clean speech reconstruction. Because the datasets span speech, music, sound, and general audio, the metric suite is applied over materially different signal classes. This is important for interpreting results: the same codec may behave differently on intelligibility-oriented measures such as STOI, identity-oriented measures such as SIM, and broader fidelity measures such as SDR or ViSQOL. The benchmark therefore treats acoustic quality as multi-dimensional rather than reducible to a single scalar score [2508.20660].

## 4. Semantic evaluation methodology

CodecBench uses **two semantic evaluation methods**: an **ASR probing task** and a **classification task**. Both are embedding-oriented rather than reconstruction-oriented. The semantic question is not merely whether the waveform sounds plausible after resynthesis, but whether the codec’s internal representation preserves task-relevant information [2508.20660].

The **ASR probing task** is adapted from the **SUPERB** framework. The pretrained codec is frozen, its **quantized embeddings** are extracted, and the embeddings are **upsampled to a minimum frame rate of 50 Hz via replication** before being fed into the downstream ASR model. The downstream model is a **two-layer bidirectional LSTM** trained with **CTC loss** for **character-level prediction**. Training uses **LibriSpeech train-clean-100**; evaluation uses **LibriSpeech dev-clean**; and the primary metric is **Word Error Rate (WER)**. The optimization settings are **batch size 4**, **max learning rate \(1\times10^{-4}\)**, and **400,000 training steps**. The appendix explains the upsampling requirement in terms of CTC alignment constraints, stating that \(T \geq U\) and, in the worst case, \(T \geq 2U + 1\), where \(T\) is the input sequence length and \(U\) is the target label length [2508.20660].

The **classification task** is intended to capture broader semantic cues that ASR probing does not measure adequately, especially contextual, emotional, and paralinguistic information. The procedure is to extract codec embeddings, train classifiers on labeled datasets, and evaluate **classification accuracy**. The speech datasets are **RAVDESS** with **8 labels**, **CREMA-D** with **6 labels**, and **MELD** with **7 labels**. The music datasets are **GTZAN** with **10 labels**, **Musical Instrument Chord Classification** with **2 labels**, and **NSynth** with **10 labels**. The sound datasets are **ESC-50** with **50 labels** and **VocalSound** with **6 labels**. Training uses **batch size 16**, **max learning rate \(1\times10^{-3}\)**, and **20 epochs** per dataset [2508.20660].

The paper also reports a **token-based semantic experiment**. For models with a **single large codebook**, such as **Stable-Codec** and **X-Codec-2.0**, learning a mapping from the very high-dimensional token space to a unified embedding space was described as nearly infeasible using only LibriSpeech. For **Mimi**, by contrast, token-based results broadly matched the embedding-based trends. This result is methodologically important because it indicates that embedding-based evaluation is practical across codec families, whereas token-based evaluation may depend strongly on codebook design [2508.20660].

## 5. Empirical findings and codec trade-offs

CodecBench reports a consistent separation between acoustic and semantic behavior. On the acoustic side, **DAC** tends to perform best overall at higher bitrates, especially on **PESQ**, **STOI**, **SIM**, and **ViSQOL**. The paper also reports that **DAC-44k-rvq9** can outperform DAC variants on **MSE**, **SDR**, and **SI-SDR**, which it attributes to higher frame rates improving temporal resolution. **Stable-Codec** generally underperforms, especially at low bitrates. **BigCodec** is strong in **single-codebook, low-bitrate** settings. **FlowDec** and **Baichuan-Audio tokenizer** are reported as competitive among **flow-matching approaches**. Across domains, **music and sound are harder than speech at low bitrate**, and degradation is sharper on non-vocal data [2508.20660].

The semantic findings are more diagnostic. The benchmark shows that **strong acoustic reconstruction does not necessarily imply strong semantic preservation**. In particular, **DAC** and **BigCodec**, despite good acoustic performance, show relatively poor **WER** in ASR probing. Conversely, **X-Codec-2.0** achieves better WER in some cases and can show stronger **SIM** than **Mimi** at similar WER, yet it performs poorly on the classification tasks. The paper hypothesizes that **single-codebook, low-bitrate designs** may prioritize **text-related semantics** while sacrificing **broader paralinguistic information**, whereas models with **larger codebooks** can preserve more **contextual** and **emotional** information even without explicit semantic supervision [2508.20660].

These results lead to a general trade-off structure. The benchmark highlights tensions between **compression efficiency** and **reconstruction quality**, and between **acoustic fidelity** and **semantic richness**. It also identifies several current limitations of audio codecs: **weak performance on complex real-world audio**, especially in **noisy, expressive, multi-speaker, or mixed-audio conditions**; **poor handling of non-vocal audio at low bitrate**; and persistent under-modeling of semantic information. The paper further notes that the present semantic evaluation remains incomplete and states that future work will expand semantic evaluation and refine CodecBench [2508.20660].

## 6. Relation to prior benchmark ecosystems

CodecBench belongs to a broader family of codec benchmarks, but its emphasis is distinct. **"Codec-SUPERB: An In-Depth Analysis of Sound Codec Models"** evaluates codec models through representative sound applications and signal-level metrics, spanning speech, general audio, and music, and includes an online leaderboard for community comparison [2402.13071]. **"Codec-SUPERB @ SLT 2024: A lightweight benchmark for neural audio codec models"** narrows that idea into a **lightweight**, **training-free**, **license-free** protocol built around sampled open and hidden sets, fixed downstream task models, and objective metrics such as **PESQ**, **STOI**, **SDR**, and **MelLoss** [2409.14085]. **"OpenACE: An Open Benchmark for Evaluating Audio Coding Performance"** focuses on open and reproducible **audio and speech coding quality assessment**, with **full-band** content, seven public datasets, **ViSQOL Audio**, **POLQA**, and **MUSHRA**-based subjective evaluation [2409.08374].

Against that background, CodecBench is characterized by its explicit separation of **acoustic** and **semantic** evaluation and by its coverage of **speech**, **music**, **sound**, and **general audio** within a single benchmark [2508.20660]. Relative to OpenACE, it moves beyond speech and audio quality assessment into embedding-based semantic probing. Relative to Codec-SUPERB and its SLT 2024 variant, it places stronger emphasis on codec representations as inputs to downstream semantic models rather than only on resynthesized audio scored by frozen application models. This suggests a broader change in benchmark design: codec evaluation is increasingly being framed not only around perceptual reconstruction but also around the preservation of information required for speech-language-model integration.

In that sense, CodecBench marks a shift from judging codecs solely as compression systems to judging them as multi-purpose representational interfaces. Its central contribution is not a single metric or leaderboard statistic, but a benchmark structure that exposes when acoustic quality and semantic usefulness diverge—and therefore where current codec design remains insufficient for modern multimodal applications.

Source: https://www.emergentmind.com/topics/codecbench