---
title: Base-Token Disentanglement (BTD) Module
url: https://www.emergentmind.com/topics/base-token-disentanglement-btd-module
type: topic
---

# Base-Token Disentanglement (BTD) Module

The Base-Token Disentanglement (BTD) module is a neural network component designed to factorize mixed-speech representations into discrete token streams, each corresponding to an underlying speaker, while jointly supporting ultra-low bitrate speech compression within the CodeSep framework. BTD operates directly on mel-spectrograms of overlapped speech, synthesizing source-differentiated “base tokens” suitable for subsequent expansion and waveform reconstruction via a codec-driven pathway. This capability enables the simultaneous achievement of speech separation and transmission efficiency, with only the base token streams requiring communication or storage, supporting transmission bitrates as low as 1 kbps [2601.12757].

## 1. Module Placement and Function within CodeSep

BTD occupies a central position in the inference chain of CodeSep, which is structured as four sequential modules:

1. **Codec Encoder (MDCTCodec encoder):** Converts input audio to latent, framewise representations.
2. **Base-Token Disentanglement (BTD):** Receives mixed-speech mel-spectrograms and emits two base-token sequences, one per embedded speaker.
3. **Auxiliary-Token Serial Prediction (ATSP):** Independently expands each base-token stream into a full multi-stage sequence of codec tokens via serial prediction.
4. **Codec Decoder (MDCTCodec decoder):** Reconstructs separated speaker waveforms from combined base and auxiliary tokens.

During inference, a mixture signal $x_1 + x_2$ is first transformed into an 80-dimensional mel-spectrogram. BTD processes this spectrogram to produce, for each frame, two sequences of discrete base tokens, each intended to represent a different speaker. These base tokens are then expanded by ATSP modules to full N-stage code token sequences, which the decoder ultimately maps back to time-domain waveforms. In deployment, the system transmits or stores only the base tokens—thereby leveraging the ensuing stages for local or downstream expansion—thus directly contributing to extremely low bitrate operation [2601.12757].

## 2. Input and Output Structures

The BTD module works with strictly defined framing and encoding conventions:

- **Input:** 80-dimensional mel-spectrogram frames extracted using an 80-sample frame shift at a 16 kHz sampling rate, delivering high temporal resolution.
- **Mel Downsampling:** The Mel Downsampling Block ($\phi_{\text{meld}}$) consists of $N_{\text{meld}} = 3$ strided 1D convolutions (stride = 2), reducing the temporal frame rate and projecting features to 256 dimensions per frame ($K_{\text{meld}} = 256$).
- **Output:** For each time frame $t$ and each speaker $i \in \{1, 2\}$, BTD yields a categorical probability vector $p_{\text{base}}^{(i)}(t) \in \mathbb{R}^M$ ($M = 1024$ codebook entries). The argmax index defines the discrete base token: $d_{\text{base}}^{(i)}(t) = \arg\max_m p_{\text{base}}^{(i)}(t)[m]$.

This design ensures emissive compactness: only one token per frame per speaker is output, substantially limiting required transmission bandwidth.

## 3. Internal Neural Architecture

The BTD is structurally modular, integrating downsampling, “intra-source” contextualization, and cross-source disentanglement:

- **Mel Downsampling ($\phi_{\text{meld}}$):** Three sequential 1D convolutional layers with stride $>1$ compress the input spectrogram, producing $z_{\text{meld}}(t) \in \mathbb{R}^{256}$.
- **Source-Intra Transformers ($\phi_{\text{intra}}$):** Four Transformer layers with multi-head self-attention and feed-forward sublayers further encode $z_{\text{meld}}(t)$ to $z_{\text{intra}}(t) \in \mathbb{R}^{256}$.
- **Anti-Consistency Source-Inter Transformer ($\phi_{\text{ac-inter}}$):**
  - The Anti-Consistency Bias Generator (ACBG) yields two trainable bias vectors $\delta^{(1)}, \delta^{(2)} \in \mathbb{R}^{256}$, used to perturb the single intra-source representation into two disequilibrium states: $z^{(1)} = z_{\text{intra}} + \delta^{(1)}$, $z^{(2)} = z_{\text{intra}} + \delta^{(2)}$.
  - These vectors are processed in parallel via four cross-attention Transformer blocks, producing $z_{\text{inter}}^{(1)}, z_{\text{inter}}^{(2)}$.
- **Token Heads:** Independent linear and softmax layers map $z_{\text{inter}}^{(i)}$ to probability vectors $p_{\text{base}}^{(i)}$; discrete tokens $d_{\text{base}}^{(i)}$ are sampled via $\arg\max$.

The anti-consistency design is empirically critical for effective disentanglement: ablation of the ACBG component (“w/o ACBG”) leads to significant degradation in objective speaker similarity metrics [2601.12757].

## 4. Mathematical Formulation and Training Loss

The BTD’s learning objective explicitly addresses the arbitrary ordering of speakers in mixture data by adopting a permutation-invariant cross-entropy loss (PI-CE):

Let $y$ denote the mixed spectrogram, and $d_1^{(1)}, d_1^{(2)}$ the ground-truth first-stage codebook indices from the Residual Vector Quantizer (RVQ), corresponding to unmixed sources $x_1, x_2$. The network computes:

\[
z_{\text{meld}} = \phi_{\text{meld}}(y)
\]
\[
z_{\text{intra}} = \phi_{\text{intra}}(z_{\text{meld}})
\]
\[
z_{\text{inter}}^{(1)}, z_{\text{inter}}^{(2)} = \phi_{\text{inter}}(z_{\text{intra}} + \delta^{(1)}, z_{\text{intra}} + \delta^{(2)})
\]
\[
p_{\text{base}}^{(i)} = \mathrm{softmax}(W_{\text{base}}^{(i)} z_{\text{inter}}^{(i)} + b_{\text{base}}^{(i)})
\]
\[
d_{\text{base}}^{(i)} = \arg\max_m p_{\text{base}}^{(i)}[m]
\]

The loss is then:

\[
\mathcal{L}_{\mathrm{PI-CE}} = -\mathbb{E}_{(y, x_1, x_2)\sim D_m} \min_{\pi\in\{(1,2),(2,1)\}} \sum_{i=1}^{2} \ln\bigl[p_{\text{base}}^{(i)}[d_1^{(\pi(i))}]\bigr]
\]

This enforces that each predicted base-token stream matches one of the ground-truth speaker token streams, regardless of their ordering in the mixture [2601.12757].

## 5. Integration with Residual Vector Quantizer and Auxiliary-Token Serial Prediction

Training leverages the RVQ module: the ground-truth for BTD consists of first-stage RVQ indices assigned by encoding each single-speaker utterance. During validation or deployment:

- Only the base-token sequence $d_{\text{base}}^{(i)}$ is transmitted or stored for each speaker ($1$ token per frame per speaker, at $1$ kbps total).
- At the receiver, for each speaker, an ATSP branch takes $d_{\text{base}}^{(i)}$ as input and autoregressively predicts the remaining $N-1$ token indices per frame via a sequence modeling stack (LSTM+Conformer).
- The complete $N$ token streams, per source, are mapped via codebook lookups and summed, then passed to the MDCTCodec decoder for waveform reconstruction.

This design achieves the simultaneous goals of multi-speaker disentanglement (through parallel BTD branches and cross-stream regularization) and lossy coded transmission at ultra-low bitrates [2601.12757].

## 6. Empirical Validation and Ablation

Extensive objective and subjective evaluations substantiate the crucial role of BTD and its anti-consistency mechanisms. At $1$ kbps, CodeSep demonstrates:

| Metric             | CodeSep (BTD) | FCTS (1 kbps) | FSTC (1 kbps) |
|--------------------|:-------------:|:-------------:|:-------------:|
| UTMOS              |    3.14       |    1.34       |    1.99       |
| DNSMOS             |    3.67       |    3.03       |    3.33       |
| NMOS               |    3.65       |    2.96       |    3.24       |
| SMOS               |    3.43       |    2.86       |    3.15       |

A/B/X speaker similarity tests confirm the anti-consistency bias’s necessity: ablation (“w/o ACBG”) reduces preference from 54.23% to 38.33% (statistically significant at $p < 0.01$). These results indicate that BTD’s combination of downsampling, intra- and inter-source Transformer processing, and PI-CE loss robustly extracts speaker-partitioned base tokens, yielding separation and compression performance unattainable by baseline methods without such disentanglement [2601.12757].

## 7. Context and Significance

The BTD module exemplifies an integrated approach to discrete-continuous representation disentanglement under coupled constraints of separation and rate efficiency. Its architecture leverages source-invariant and cross-source architecture features, and its training incorporates permutation-invariance to accommodate the inherent ambiguities of multi-source mixtures. This approach is particularly relevant for scenarios such as real-time online meetings and dialogue archiving, where efficient, privacy-preserving, and accurate utterance separation and storage are paramount. A plausible implication is the generalization of BTD-style modules to multi-modal or generalized source separation tasks where compact, interpretable, and disentangled representations are operationally beneficial [2601.12757].

Source: https://www.emergentmind.com/topics/base-token-disentanglement-btd-module