---
title: 'AD-FlowTSE: Adaptive Flow Matching for TSE'
url: https://www.emergentmind.com/topics/ad-flowtse
type: topic
---

# AD-FlowTSE: Adaptive Flow Matching for TSE

Adaptive Deterministic Flow Matching for Target Speaker Extraction (AD-FlowTSE) is a one-step generative framework for target speaker extraction (TSE) that leverages optimal-transport-based flow matching with an adaptive mixture-ratio-aware (MR-aware) initialization and step size. AD-FlowTSE frames TSE as a deterministic transport problem between the empirical background and clean-speech distributions, establishing a linear path parameterized by the mixing ratio τ. By learning both the transport field and an auxiliary MR estimator, AD-FlowTSE extracts target speech from audio mixtures given an enrollment utterance, achieving low-latency, high-fidelity separation with a single generative step. This method is foundational for subsequent paradigms such as MeanFlow-TSE and represents a significant advance in real-time generative speech enhancement [2510.16995, 2512.18572, 2603.10701].

## 1. Mathematical Formulation and Deterministic Flow

AD-FlowTSE considers observed single-channel mixtures $x = \tau s_1 + (1-\tau) b$ in the time or spectral domain, where $s_1$ is the clean target speaker, $b$ is the background (interferers or noise), and $\tau \in [0,1]$ is the mixing ratio. The central idea is to define a deterministic "noising" trajectory in spectral (STFT) space:

$$
\mathbf{z}_t = (1-t)\,\mathbf{B} + t\,\mathbf{S}, \quad t \in [0,1],
$$

where $\mathbf{S}$ and $\mathbf{B}$ are the STFTs of $s_1$ and $b$. The observed mixture $\mathbf{Y}$ sits at position $t = \lambda$ (mixture-matching MR) on this path, i.e., $\mathbf{Y} = \lambda\,\mathbf{S} + (1-\lambda)\,\mathbf{B}$. The ground-truth velocity (flow field) is constant: $u_t(\mathbf{z}_t) = \mathbf{S} - \mathbf{B}$ [2510.16995, 2512.18572, 2603.10701].

## 2. Generative Model Architecture

The backbone of AD-FlowTSE is a mean-velocity network $u_\theta$ (or $v_\theta$ in some conventions), typically instantiated as a UDiT (U-Net Diffusion Transformer) model. The architecture includes:

- **Inputs and Conditioning**: 
  - Input mixture spectrogram $\mathbf{Y}$
  - Enrollment utterance $\mathbf{E} = \mathrm{STFT}(e)$ for the target speaker
  - Mixture ratio estimate $\hat{\tau}$, predicted by an MR-predictor network $p_\phi$ implemented as an ECAPA-TDNN encoder followed by an MLP.
- **Processing**:
  - The mean-velocity network $u_\theta$ receives $\mathbf{Y}$, $\hat{\tau}$, and $\mathbf{E}$, producing a velocity prediction $\Delta \mathbf{Z}$.
  - Output spectrogram: $\widehat{\mathbf{S}} = \mathbf{Y} + (1-\hat{\tau}) \cdot \Delta \mathbf{Z}$
- **Inference**: Single-step extraction (NFE=1) is achieved by applying the predicted mean velocity along the residual background-to-target path determined by $\hat{\tau}$.
- **Feature Engineering**: All spectra are handled as concatenated real+imaginary channels (complex STFT domain) [2603.10701, 2510.16995].

## 3. Optimization Objective and Training Procedure

AD-FlowTSE is trained with a flow-matching objective on the spectral linear path:

- **Flow-Matching Anchor Loss**:
  $$
  \mathcal{L}_{\mathrm{FM}}(\theta) = \mathbb{E}_{\tau, \mathbf{B}, \mathbf{S}, \mathbf{E}} \left\| u_\theta(\mathbf{z}_\tau, \tau, 1; \mathbf{E}) - (\mathbf{S}-\mathbf{B}) \right\|_2^2
  $$
- **Auxiliary MR Regression Loss** (for $p_\phi$):
  $$
  \mathcal{L}_{\mathrm{MR}}(\phi) = \mathbb{E} [ (\hat{\tau} - \tau)^2 ]
  $$
- **Total Loss**:
  $$
  \mathcal{L}_{\text{total}} = \lambda_{\mathrm{FM}} \mathcal{L}_{\mathrm{FM}} + \lambda_{\mathrm{MR}} \mathcal{L}_{\mathrm{MR}}
  $$
Training alternates between mean-velocity flow-matching and MR regression. Unlike more advanced variants (e.g., AlphaFlowTSE), AD-FlowTSE does not employ interval-consistency teacher-student regularization or Jacobian-vector product (JVP)-free stabilization; the only targets are the flow-matching anchor and MR regression [2603.10701].

**Training and Inference Pseudocode** (algorithmic core, one-step regime):
```python
# Training loop
for each minibatch (B,S,Y,E,τ*):
    τ̂ = sigmoid(pφ(Y, E))
    L_MR = (τ̂ - τ*)**2

    τ = random_uniform(0, 1)
    x_τ = (1-τ)*B + τ*S
    v_bg = S - B
    u = uθ(x_τ, τ, 1, E)
    L_FM = norm(u - v_bg) ** 2

    L = λ_FM * L_FM + λ_MR * L_MR
    # update θ, φ by gradient descent
```
```python
# Inference
Y = STFT(y)
E = STFT(e)
τ̂ = sigmoid(pφ(Y, E))
ΔZ = uθ(Y, τ̂, 1, E)
Ŝ = Y + (1 - τ̂) * ΔZ
ŷ = iSTFT(Ŝ)
```
[2603.10701, 2510.16995]

## 4. Adaptive MR Estimation and Step Budget

A primary innovation of AD-FlowTSE is MR‐aware initialization. At test time, the MR-predictor $p_\phi$ estimates $\hat{\tau}$, and extraction proceeds from $\mathbf{z}_{\hat{\tau}} = \mathbf{Y}$ towards $\mathbf{z}_1 = \mathbf{S}$, correcting only the residual distortion corresponding to $1-\hat{\tau}$. This enables adaptive step-sizing:

- For step budget $N=1$, extraction is achieved in a single Euler step of width $1-\hat{\tau}$.
- For $N>1$, the update is recursively applied in uniform increments from $\hat{\tau}$ to $1$, but empirical results show best performance for $N=1$ or $N=5$ with degradation at large $N$.
- MR initialization ensures the model does not over- or under-correct for noise, as $τ$-mismatch can degrade SI-SDR significantly (e.g., random $τ$ reduces SI-SDR to ~9.14 dB vs. oracle $τ$ at ~12.85 dB) [2510.16995, 2603.10701].

## 5. Empirical Results and Comparative Analysis

Benchmarking on Libri2Mix (16 kHz, min) and REAL-T datasets yields the following performance profile for one-step AD-FlowTSE (NFE=1):

| Method         | SI-SDR (dB) | PESQ | ESTOI | SpkSim |
|----------------|-------------|------|-------|--------|
| AD-FlowTSE     | 17.49 (clean)/12.70 (noisy) | 2.89/2.15 | 0.90/0.81 | 0.95/0.87 |
| MeanFlowTSE    | 18.80/12.85 | 3.26/2.21 | 0.93/0.82 | 0.92/0.73 |
| AlphaFlowTSE   | 19.17/13.16 | 3.27/2.28 | 0.94/0.85 | 0.93/0.76 |

Ablation studies demonstrate that removing the MR predictor results in heavy degradation (SI-SDR −4.95 dB, PESQ −19.4 %) for AD-FlowTSE, whereas more advanced methods are less sensitive. On zero-shot real mixtures (e.g., REAL-T) without MR supervision, AD-FlowTSE yields 34% WER and 0.54 SpkSim, underperforming baselines with more robust MR-agnostic formulations [2603.10701].

Efficiency metrics (on NVIDIA L40) indicate AD-FlowTSE achieves RTF = 0.017 and GPU memory ≈1.5 GB, enabling real-time operation [2512.18572].

## 6. Limitations and Directions for Enhancement

- **MR-predictor Dependence**: AD-FlowTSE relies critically on an accurate, supervised MR regressor ($p_\phi$). In mismatched or real-world conditions where mixture ratio labels are absent, performance is not robust.
- **One-Step Constraint**: Although NFE=1 enables low latency, modeling only a single finite-interval update constrains the method to cases where the mixture closely follows the linear background-target interpolation; significant target/background interaction or nonlinear corruption is less well-addressed.
- **No Interval-Consistency Teacher**: Lack of an interval-consistency (teacher-student) target, as in AlphaFlowTSE, renders AD-FlowTSE less stable under MR uncertainty.
- **Future Directions**: Improvements could include joint MR/velocity distillation, hybrid one-/multi-step inference, trajectory parameterization beyond linear MR paths, or multimodal conditioning (multi-enrollment, audio-visual) [2603.10701].

## 7. Relationship to Subsequent Paradigms

AD-FlowTSE provides the structural substrate for one-step generative TSE, with its MR-indexed transport and deterministic velocity matching becoming central in later models. MeanFlow-TSE generalizes the loss to mean-flow (α-flow) objectives with further empirical gains, and AlphaFlowTSE introduces teacher-student consistency in an attempt to regularize and robustify extraction under uncertainty [2510.16995, 2512.18572, 2603.10701]. This situates AD-FlowTSE as both a baseline and a conceptual precursor to higher-performing, more robust generative TSE frameworks.

Source: https://www.emergentmind.com/topics/ad-flowtse