CodecSep: Codec-Native Source Separation

• CodecSep is a codec-native approach to audio separation that processes signals in the latent space of neural codecs, enabling text-guided universal separation.
• It employs a Transformer masker with FiLM modulation based on CLAP text embeddings to isolate audio sources efficiently while preserving codec integrity.
• Empirical results show CodecSep outperforms spectrogram-based models in SI-SDR while dramatically reducing computational load and supporting low-bitrate deployment.

CodecSep denotes a codec-native approach to source separation in which separation is performed inside the representation space of a neural audio codec rather than on raw waveforms or spectrograms. In the narrow sense, the term refers to the model introduced in "Neural Audio Codecs for Prompt-Driven Universal Source Separation" (Banerjee et al., 15 Sep 2025), described as the first NAC-based model for prompt-driven universal source separation. In a broader literature usage, closely related work uses codec latents for speech separation through Codecformer (Yip et al., 2024), and a later codec-driven formulation, CodeSep, performs joint speech separation and low-bitrate compression in a discrete RVQ token space (Du et al., 19 Jan 2026). Across these usages, the unifying idea is that a pretrained codec defines the operational domain for separation, with efficiency, bitstream compatibility, and deployment constraints treated as first-class design objectives.

1. Terminological scope and historical placement

Within the 2025 literature, CodecSep is a text-guided, universal source separation system that operates directly in the latent space of a neural audio codec and is designed to provide the flexibility of prompt-driven separation with the efficiency and deployment friendliness of codec-based models while avoiding fixed-class limitations (Banerjee et al., 15 Sep 2025). The motivating comparison is between spectrogram/text-guided separators such as AudioSep, which are compute-heavy and typically deployed as cloud services, and NAC-based separators such as CodecFormer and SDCodec, which are more efficient but are limited to fixed-class separation.

A broader codec-native separation lineage predates this specific model. "Towards Audio Codec-based Speech Separation" (Yip et al., 2024) formulates speech separation inside the embedding space of Descript Audio Codec (DAC) and proposes Codecformer, a model derived from Sepformer but simplified by the codec’s temporal compression. That work characterizes codec-based separation as a setting in which a neural audio codec encodes the mixture waveform into a continuous embedding sequence, a separator operates on that sequence, and the codec decoder reconstructs separated waveforms. The later "CodeSep: Low-Bitrate Codec-Driven Speech Separation with Base-Token Disentanglement and Auxiliary-Token Serial Prediction" (Du et al., 19 Jan 2026) moves from continuous codec embeddings to discrete RVQ tokens and integrates separation with low-bitrate transmission or storage.

These works can be organized as follows.

Work	Operating domain	Core task
CodecSep (Banerjee et al., 15 Sep 2025)	DAC continuous latents / code streams	Prompt-driven universal source separation
Codecformer (Yip et al., 2024)	DAC continuous embeddings	Speech separation
CodeSep (Du et al., 19 Jan 2026)	RVQ discrete tokens	Joint speech separation and low-bitrate compression

This progression suggests a widening of codec-native separation from two-speaker speech separation, to open-vocabulary text-guided source separation, to token-level separation under explicit bitrate budgets.

2. Problem formulation and architectural principles

In the universal setting used by CodecSep, a mono mixture is modeled as

$x(t) \;=\; \sum_{s \in \mathcal{S}} y_s(t),$

where the number of sources is unknown and the set of sound classes $\mathcal{S}$ is potentially unbounded (Banerjee et al., 15 Sep 2025). The task is: given a natural-language query $\tau_s$ describing some target source or sources, recover the corresponding waveform $y_s(t)$ from $x(t)$. The paper explicitly frames this as text-guided universal source separation, not limited to a fixed ontology, with prompts such as “speech”, “music”, “dog barking, Animal, engine rumbling, motor vehicle”, and “crowd cheering and fireworks”.

CodecSep instantiates this by using DAC as a frozen encoder-decoder backbone. For an audio fragment $x(t)$ sampled at $F_s$, DAC’s encoder downsamples by a factor $M=320$ to produce latent frames

$$Z = [z_t \in \mathbb{R}^d \mid t \in [T]], \quad T = \frac{F_s}{M}.$$

At 24 kHz, $T = 75$ frames per second. Given bitrate $\mathcal{S}$0, each frame carries

$\mathcal{S}$1

allocated across $\mathcal{S}$2 residual vector quantizers, each with 10 bits and a 1024-sized codebook. The RVQ quantizer maps $\mathcal{S}$3 to discrete codes

$\mathcal{S}$4

and codebook lookup yields continuous embeddings

$\mathcal{S}$5

which DAC decodes back to waveform (Banerjee et al., 15 Sep 2025).

Separation itself is performed above RVQ in continuous latent space. CodecSep uses a Transformer masker with $\mathcal{S}$6 layers and embedding size $\mathcal{S}$7. After the last layer, per-source 1D convolutional heads output masks. For a given source $\mathcal{S}$8,

$\mathcal{S}$9

Snake activations $\tau_s$0 are used throughout to preserve a periodic inductive bias consistent with the DAC backbone. For text-guided separation, CLAP produces a text embedding $\tau_s$1, a lightweight query network maps that embedding to per-layer FiLM parameters $\tau_s$2, and FiLM is applied channel-wise inside Transformer layers: $\tau_s$3 For single-source text-guided separation, the mask is

$\tau_s$4

The architectural contrast with spectrogram-domain systems is explicit. AudioSep operates in the complex STFT domain with a large spectrogram separator, whereas CodecSep follows

$\tau_s$5

Because $\tau_s$6 is low-dimensional, the masking network is much cheaper than a spectrogram-domain separator operating on high-resolution $\tau_s$7 (Banerjee et al., 15 Sep 2025).

3. Training regimes and objectives

CodecSep is trained in two main regimes. The primary regime uses dnr-v2, described as a Cocktail Fork problem dataset with 60-s mixtures of speech, music, and SFX drawn from LibriSpeech, FMA, and FSD50K respectively (Banerjee et al., 15 Sep 2025). LUFS normalization is applied, validation and test segments are 5–10 s, and segments where a stem is present for less than 50% of the time are removed. For SFX, stems often contain multiple overlapping events, and prompts are compositional, generated from FSD50K’s hierarchical labels. Prompt types are generic for speech and music and fine-grained for SFX.

The alternate regime uses AudioCaps, derived from AudioSet with more than 46k 10-s clips with captions; the authors synthesize mixtures by summing three clips. Additional open-domain evaluation uses ESC-50, Clotho-v2, AudioSet-eval, and VGGSound, again with three-clip mixtures and text prompts derived from class names or captions (Banerjee et al., 15 Sep 2025).

The primary loss is negative SI-SDR on each separated source and on the reconstructed mixture. Given ground truth $\tau_s$8 and estimate $\tau_s$9,

$y_s(t)$0

Letting $y_s(t)$1 denote the source prediction and

$y_s(t)$2

the full loss is

$y_s(t)$3

The paper states that this encourages accurate per-source separation and mixture consistency (Banerjee et al., 15 Sep 2025). It also mentions codec SI-SDR (cSI-SDR), where ground truth is passed through the codec to account explicitly for compression artifacts.

Training details are tightly constrained. DAC and CLAP backbones are frozen; only the Transformer masker and query network are trained. The optimizer is ADAM with learning rate $y_s(t)$4, a ReduceLROnPlateau scheduler halves the learning rate when validation stops improving, batch size is 4, training uses 2-s segments, and runs for approximately 400k iterations on a single 24 GB A30 GPU (Banerjee et al., 15 Sep 2025). Two variants are reported: CodecSep + dnr-v2 and CodecSep + AudioCaps. For matched comparisons, TDANet and CodecFormer are re-trained from scratch on dnr-v2 under the same setup, AudioSep is evaluated both as a public diverse pre-trained model and in retrained versions, and SDCodec uses official pre-trained weights. To simulate realistic deployments, all inputs to TDANet and AudioSep are first processed by a codec during training and inference.

A closely related but narrower training formulation appears in Codecformer. That work introduces cSI-SDR because evaluating against pristine clean speech can be unrealistic when the deployment path is codec-distorted (Yip et al., 2024). There, the codec-distorted target is

$y_s(t)$5

and cSI-SDR is computed between $y_s(t)$6 and $y_s(t)$7. This formulation makes training and evaluation consistent with cloud-side codec-based deployment.

4. Empirical performance and ablations

CodecSep is evaluated on six benchmarks: dnr-v2, AudioCaps-test, ESC-50, Clotho-v2, AudioSet-eval, and VGGSound, using SI-SDR for separation fidelity, ViSQOL MOS-LQO for perceptual similarity, and GMACs for computational efficiency (Banerjee et al., 15 Sep 2025). The principal baselines are AudioSep, TDANet, CodecFormer, and SDCodec.

On dnr-v2 in the universal setting with generic speech and music prompts and compositional SFX prompts, CodecSep + dnr-v2 exceeds AudioSep + dnr-v2 in SI-SDR for all three stems. The reported numbers are: music 1.15 dB versus $y_s(t)$8 dB, speech 9.97 dB versus 7.68 dB, and SFX 0.89 dB versus $y_s(t)$9 dB (Banerjee et al., 15 Sep 2025). For ViSQOL, CodecSep matches or slightly exceeds AudioSep for speech and music; for SFX, AudioSep pre-trained zero-shot has slightly higher ViSQOL than CodecSep, but SI-SDR favors CodecSep.

A masker ablation in which FiLM is applied directly on encoder latents, without a dedicated Transformer masker, yields good perceptual quality but much worse separation SI-SDR (Banerjee et al., 15 Sep 2025). The authors interpret this as evidence that disturbing the mixture representation itself is harmful, whereas applying FiLM inside a dedicated masker preserves the integrity of the frozen DAC representation.

Table 2 in that work compares fixed-stem baselines, generic prompt setups, and the full universal setup. SDCodec is reported as a strong fixed-stem baseline with music 1.85 dB, speech 11.32 dB, and SFX 1.77 dB, with ViSQOL up to approximately 3.5 (Banerjee et al., 15 Sep 2025). Under matched generic prompts, CodecSep performs better than AudioSep on SFX and similarly on speech and music. In the universal setting, CodecSep + dnr-v2 outperforms both AudioSep variants across all stems in SI-SDR. The paper further states that fine-grained SFX prompts improve not only SFX but speech and music stems as well, attributing this to richer supervision and better scene disentanglement.

Architectural ablations compare decoder-style generation with latent masking. Results show that an unguided 3-stem CodecSep masker improves SI-SDR significantly over CodecFormer’s decoder-style generation across all stems, and adding text guidance yields further small gains (Banerjee et al., 15 Sep 2025). This supports the specific design choice of masking DAC latents rather than generating them from scratch.

For open-domain generalization, the pre-trained broad-data AudioSep is strongest overall on AudioCaps-test, but under matched training conditions CodecSep exceeds AudioSep in SI-SDR, both when trained on dnr-v2 and when both are trained on AudioCaps (Banerjee et al., 15 Sep 2025). On ESC-50, Clotho-v2, AudioSet-eval, and VGGSound, CodecSep + dnr-v2 consistently beats AudioSep + dnr-v2 in SI-SDR under matched training, with modest ViSQOL differences that mostly favor CodecSep. Prompt paraphrasing experiments show that both systems degrade when “speech” and “music” are replaced with synonyms, but CodecSep degrades more gracefully: for speech, 4.19 dB versus 4.14 dB under ambiguous prompts; for music, $x(t)$0 dB versus $x(t)$1 dB (Banerjee et al., 15 Sep 2025).

The speech-only codec-native antecedent also reports competitive separation performance. Codecformer achieves a 52x reduction in MAC while producing separation performance comparable to a cloud deployment of Sepformer (Yip et al., 2024). Against codec-distorted references, Sepformer (Cloud) attains cSI-SDRi 9.6 dB and cSDRi 10.1 dB, whereas Codecformer reports cSI-SDRi 9.9 dB and cSDRi 10.4 dB, with slightly lower but comparable PESQ.

5. Compute efficiency, bitstream compatibility, and deployment

Compute efficiency is central to the definition of CodecSep. In full inference complexity, AudioSep’s separator alone requires 33.5 GMACs, whereas CodecSep’s masker architecture requires 1.35 GMACs (Banerjee et al., 15 Sep 2025). For audio-stream input, CodecSep must perform encoder, masker, and decoder computation: $x(t)$2 For code-stream input, the end-to-end path reduces to the masker alone, 1.35 GMACs, because continuous embeddings can be reconstructed from the code stream through codebook lookup. The paper therefore reports two compute ratios: approximately $x(t)$3 cheaper architecture-only and approximately $x(t)$4 cheaper end-to-end in code-stream deployment relative to AudioSep (Banerjee et al., 15 Sep 2025).

The corresponding deployment path is explicitly bitstream-compatible. If the server receives discrete DAC codes $x(t)$5, it reconstructs continuous embeddings by lookup,

$x(t)$6

applies the same masker on $x(t)$7, and can optionally re-quantize separated sources: $x(t)$8 The server never needs to decode to waveform, and the client can decode separated stems locally from $x(t)$9 (Banerjee et al., 15 Sep 2025). This directly addresses the deployment constraint that many practical systems already transmit neural codec bitstreams.

Codecformer is motivated by a closely related deployment argument. The 2024 work assumes that compression and decompression MACs are common to both classic cloud separation and codec-native separation, and therefore measures the separator alone; under that assumption, Sepformer uses 77.3 GMACs while Codecformer uses 1.5 GMACs for 2-s utterances (Yip et al., 2024). That paper presents codec-native separation as attractive for mobile devices, communication apps, hearing aids, wearables, IoT, and unified communication stacks in which the codec front-end is already present.

A separate but relevant codec-integrated design principle is illustrated by LACE, a strictly causal neural postfilter integrated into Opus/SILK with fully compatible bitstreams, approximately 300k parameters, and approximately 100 MFLOPS (Büthe et al., 2023). LACE is not a separator, but it shows that codec-side neural processing can be engineered around zero added algorithmic delay, low CPU complexity, and decoder-side integration. This suggests a broader deployment pattern for codec-aware enhancement and separation under tight latency and power constraints.

6. Variants, related paradigms, and limitations

The principal conceptual rationale for CodecSep is that NAC latent domains are much smaller than STFT domains and carry structure shaped by perceptual and adversarial training. The 2025 paper states three main consequences: dimensionality reduction, a semantic prior in the DAC latent manifold, and improved stability from masking rather than generating (Banerjee et al., 15 Sep 2025). It also notes an RVQ hierarchy in which early quantizers capture coarse structure and later ones refine residuals. This suggests why codec latents can support both efficient masking and fine-detail control.

The discrete-token extension of this idea appears in CodeSep. That model addresses a joint task: separate a mixed speech signal into individual speakers and represent each speaker using discrete, low-bitrate tokens suitable for transmission or storage (Du et al., 19 Jan 2026). It combines an RVQ-based plain neural speech codec, a Base-Token Disentanglement module, and Parallel Auxiliary-Token Serial Prediction modules. BTD predicts first-stage VQ tokens for each speaker from a mixture using permutation-invariant cross-entropy; ATSP predicts later-stage tokens using teacher forcing. Only first-stage tokens are transmitted or stored, yielding approximately 1 kbps total bitrate for two speakers in the reported setup. On Libri2Mix-clean, CodeSep at 1 kbps reports UTMOS 3.14, DNSMOS 3.67, NMOS $x(t)$0, and SMOS $x(t)$1, exceeding both FCTS and FSTC at the same bitrate and outperforming FSTC even at 2, 4, and 8 kbps on objective metrics (Du et al., 19 Jan 2026). The paper explicitly avoids SI-SDR, arguing that codec-based tokenized separation is better characterized by perceptual measures under quantization.

The limitations of CodecSep as a universal text-guided separator are explicitly acknowledged. Experiments use 16 kHz mono for reproducibility and compute; higher-rate and stereo extensions are described as straightforward through a different codec backbone (Banerjee et al., 15 Sep 2025). Training data and prompt diversity are modest relative to web-scale audio; temporal prompts such as “applause after the song” are not tackled; SFX perceptual quality can lag the best baseline even when SI-SDR is strong; and no formal MOS tests are reported. The paper also notes that negative prompts or joint multi-prompt inference are left as future directions.

The earlier Codecformer work has its own narrower limitations: DAC was not trained on speech mixtures, RVQ can reduce performance by 2–3 dB in SI-SDR-like metrics and about 0.5 PESQ, experiments are restricted to two-speaker single-channel WSJ0-2mix, and evaluation against clean-waveform SI-SDR is problematic once lossy coding is introduced (Yip et al., 2024). CodeSep likewise remains limited to two-speaker, 16 kHz, clean-speech mixtures and does not discuss streaming or latency (Du et al., 19 Jan 2026).

Taken together, these strands define CodecSep less as a single fixed architecture than as a codec-native separation paradigm. In one branch, separation is performed over continuous DAC latents with CLAP-conditioned FiLM modulation for open-vocabulary queries (Banerjee et al., 15 Sep 2025). In another, separation and compression are unified in a discrete token space with explicit bitrate control (Du et al., 19 Jan 2026). A plausible implication is that future systems will increasingly treat the neural codec not as a peripheral pre-/post-processing module, but as the native representational substrate for source separation, enhancement, and downstream audio manipulation.