AD-FlowTSE: Adaptive Flow Matching for TSE

• The paper introduces AD-FlowTSE, a one-step generative framework for target speaker extraction using deterministic flow matching and adaptive mixture-ratio estimation.
• It formulates target extraction as a deterministic transport problem in the spectral domain and employs a UDiT-based network for velocity prediction.
• Empirical results on datasets like Libri2Mix demonstrate high SI-SDR gains with low latency, while performance remains sensitive to accurate MR estimation.

Adaptive Deterministic Flow Matching for Target Speaker Extraction (AD-FlowTSE) is a one-step generative framework for target speaker extraction (TSE) that leverages optimal-transport-based flow matching with an adaptive mixture-ratio-aware (MR-aware) initialization and step size. AD-FlowTSE frames TSE as a deterministic transport problem between the empirical background and clean-speech distributions, establishing a linear path parameterized by the mixing ratio τ. By learning both the transport field and an auxiliary MR estimator, AD-FlowTSE extracts target speech from audio mixtures given an enrollment utterance, achieving low-latency, high-fidelity separation with a single generative step. This method is foundational for subsequent paradigms such as MeanFlow-TSE and represents a significant advance in real-time generative speech enhancement (Hsieh et al., 19 Oct 2025, Shimizu et al., 21 Dec 2025, Li et al., 11 Mar 2026).

1. Mathematical Formulation and Deterministic Flow

AD-FlowTSE considers observed single-channel mixtures $x = \tau s_1 + (1-\tau) b$ in the time or spectral domain, where $s_1$ is the clean target speaker, $b$ is the background (interferers or noise), and $\tau \in [0,1]$ is the mixing ratio. The central idea is to define a deterministic "noising" trajectory in spectral (STFT) space:

$$\mathbf{z}_t = (1-t)\,\mathbf{B} + t\,\mathbf{S}, \quad t \in [0,1],$$

where $\mathbf{S}$ and $\mathbf{B}$ are the STFTs of $s_1$ and $b$. The observed mixture $\mathbf{Y}$ sits at position $s_1$0 (mixture-matching MR) on this path, i.e., $s_1$1. The ground-truth velocity (flow field) is constant: $s_1$2 (Hsieh et al., 19 Oct 2025, Shimizu et al., 21 Dec 2025, Li et al., 11 Mar 2026).

2. Generative Model Architecture

The backbone of AD-FlowTSE is a mean-velocity network $s_1$3 (or $s_1$4 in some conventions), typically instantiated as a UDiT (U-Net Diffusion Transformer) model. The architecture includes:

• Inputs and Conditioning:
  • Input mixture spectrogram $s_1$5
  • Enrollment utterance $s_1$6 for the target speaker
  • Mixture ratio estimate $s_1$7, predicted by an MR-predictor network $s_1$8 implemented as an ECAPA-TDNN encoder followed by an MLP.
• Processing:
  • The mean-velocity network $s_1$9 receives $b$0, $b$1, and $b$2, producing a velocity prediction $b$3.
  • Output spectrogram: $b$4
• Inference: Single-step extraction (NFE=1) is achieved by applying the predicted mean velocity along the residual background-to-target path determined by $b$5.
• Feature Engineering: All spectra are handled as concatenated real+imaginary channels (complex STFT domain) (Li et al., 11 Mar 2026, Hsieh et al., 19 Oct 2025).

3. Optimization Objective and Training Procedure

AD-FlowTSE is trained with a flow-matching objective on the spectral linear path:

• Flow-Matching Anchor Loss:

$b$6

• Auxiliary MR Regression Loss (for $b$7):

$b$8

• Total Loss:

$b$9

Training alternates between mean-velocity flow-matching and MR regression. Unlike more advanced variants (e.g., AlphaFlowTSE), AD-FlowTSE does not employ interval-consistency teacher-student regularization or Jacobian-vector product (JVP)-free stabilization; the only targets are the flow-matching anchor and MR regression (Li et al., 11 Mar 2026).

Training and Inference Pseudocode (algorithmic core, one-step regime): $$\mathbf{z}_t = (1-t)\,\mathbf{B} + t\,\mathbf{S}, \quad t \in [0,1],$$7 $$\mathbf{z}_t = (1-t)\,\mathbf{B} + t\,\mathbf{S}, \quad t \in [0,1],$$8 (Li et al., 11 Mar 2026, Hsieh et al., 19 Oct 2025)

4. Adaptive MR Estimation and Step Budget

A primary innovation of AD-FlowTSE is MR‐aware initialization. At test time, the MR-predictor $\tau \in [0,1]$0 estimates $\tau \in [0,1]$1, and extraction proceeds from $\tau \in [0,1]$2 towards $\tau \in [0,1]$3, correcting only the residual distortion corresponding to $\tau \in [0,1]$4. This enables adaptive step-sizing:

• For step budget $\tau \in [0,1]$5, extraction is achieved in a single Euler step of width $\tau \in [0,1]$6.
• For $\tau \in [0,1]$7, the update is recursively applied in uniform increments from $\tau \in [0,1]$8 to $\tau \in [0,1]$9, but empirical results show best performance for $$\mathbf{z}_t = (1-t)\,\mathbf{B} + t\,\mathbf{S}, \quad t \in [0,1],$$0 or $$\mathbf{z}_t = (1-t)\,\mathbf{B} + t\,\mathbf{S}, \quad t \in [0,1],$$1 with degradation at large $$\mathbf{z}_t = (1-t)\,\mathbf{B} + t\,\mathbf{S}, \quad t \in [0,1],$$2.
• MR initialization ensures the model does not over- or under-correct for noise, as $$\mathbf{z}_t = (1-t)\,\mathbf{B} + t\,\mathbf{S}, \quad t \in [0,1],$$3-mismatch can degrade SI-SDR significantly (e.g., random $$\mathbf{z}_t = (1-t)\,\mathbf{B} + t\,\mathbf{S}, \quad t \in [0,1],$$4 reduces SI-SDR to ~9.14 dB vs. oracle $$\mathbf{z}_t = (1-t)\,\mathbf{B} + t\,\mathbf{S}, \quad t \in [0,1],$$5 at ~12.85 dB) (Hsieh et al., 19 Oct 2025, Li et al., 11 Mar 2026).

5. Empirical Results and Comparative Analysis

Benchmarking on Libri2Mix (16 kHz, min) and REAL-T datasets yields the following performance profile for one-step AD-FlowTSE (NFE=1):

Method	SI-SDR (dB)	PESQ	ESTOI	SpkSim
AD-FlowTSE	17.49 (clean)/12.70 (noisy)	2.89/2.15	0.90/0.81	0.95/0.87
MeanFlowTSE	18.80/12.85	3.26/2.21	0.93/0.82	0.92/0.73
AlphaFlowTSE	19.17/13.16	3.27/2.28	0.94/0.85	0.93/0.76

Ablation studies demonstrate that removing the MR predictor results in heavy degradation (SI-SDR −4.95 dB, PESQ −19.4 %) for AD-FlowTSE, whereas more advanced methods are less sensitive. On zero-shot real mixtures (e.g., REAL-T) without MR supervision, AD-FlowTSE yields 34% WER and 0.54 SpkSim, underperforming baselines with more robust MR-agnostic formulations (Li et al., 11 Mar 2026).

Efficiency metrics (on NVIDIA L40) indicate AD-FlowTSE achieves RTF = 0.017 and GPU memory ≈1.5 GB, enabling real-time operation (Shimizu et al., 21 Dec 2025).

6. Limitations and Directions for Enhancement

• MR-predictor Dependence: AD-FlowTSE relies critically on an accurate, supervised MR regressor ($$\mathbf{z}_t = (1-t)\,\mathbf{B} + t\,\mathbf{S}, \quad t \in [0,1],$$6). In mismatched or real-world conditions where mixture ratio labels are absent, performance is not robust.
• One-Step Constraint: Although NFE=1 enables low latency, modeling only a single finite-interval update constrains the method to cases where the mixture closely follows the linear background-target interpolation; significant target/background interaction or nonlinear corruption is less well-addressed.
• No Interval-Consistency Teacher: Lack of an interval-consistency (teacher-student) target, as in AlphaFlowTSE, renders AD-FlowTSE less stable under MR uncertainty.
• Future Directions: Improvements could include joint MR/velocity distillation, hybrid one-/multi-step inference, trajectory parameterization beyond linear MR paths, or multimodal conditioning (multi-enrollment, audio-visual) (Li et al., 11 Mar 2026).

7. Relationship to Subsequent Paradigms

AD-FlowTSE provides the structural substrate for one-step generative TSE, with its MR-indexed transport and deterministic velocity matching becoming central in later models. MeanFlow-TSE generalizes the loss to mean-flow (α-flow) objectives with further empirical gains, and AlphaFlowTSE introduces teacher-student consistency in an attempt to regularize and robustify extraction under uncertainty (Hsieh et al., 19 Oct 2025, Shimizu et al., 21 Dec 2025, Li et al., 11 Mar 2026). This situates AD-FlowTSE as both a baseline and a conceptual precursor to higher-performing, more robust generative TSE frameworks.