---
title: 'FlowSE: Flow Matching Speech Enhancement'
url: https://www.emergentmind.com/topics/flowse
type: topic
---

# FlowSE: Flow Matching Speech Enhancement

Searching arXiv for FlowSE-related papers to ground the article in current research.
Querying the arXiv API for papers matching “FlowSE” and closely related speech-enhancement work.
```python
import urllib.parse, urllib.request, xml.etree.ElementTree as ET, textwrap, re, json, sys, os, math
query = 'all:FlowSE speech enhancement'
url = 'http://export.arxiv.org/api/query?search_query=' + urllib.parse.quote(query) + '&start=0&max_results=10'
print(url)
try:
    data = urllib.request.urlopen(url, timeout=20).read().decode('utf-8')
    print(data[:1000])
except Exception as e:
    print("ERR", e)
```

Running the arXiv API query now.
```python
import urllib.parse, urllib.request
query = 'all:FlowSE speech enhancement'
url = 'http://export.arxiv.org/api/query?search_query=' + urllib.parse.quote(query) + '&start=0&max_results=10'
print(url)
try:
    data = urllib.request.urlopen(url, timeout=20).read().decode('utf-8')
    print(data[:1500])
except Exception as e:
    print("ERR", repr(e))
```

FlowSE is the name used for a set of speech enhancement methods that formulate denoising as continuous transport between probability distributions, learned through flow matching rather than iterative reverse diffusion. In these systems, a neural network estimates a velocity field on a continuous path connecting a simple source distribution to a clean-speech target, and enhancement is obtained by numerically integrating an ordinary differential equation with relatively few function evaluations. The designation covers at least two distinct 2025 formulations—one framed as conditional flow matching for speech enhancement, and another framed as an efficient mel-spectrogram generator with optional transcript conditioning—and it is subsequently extended by online reinforcement-learning post-training and contrasted with a time-unconditional autonomous rectified-flow alternative [2508.06840][2505.19476][2601.16483][2606.20001].

## 1. Scope and nomenclature

The 2025 paper "FlowSE: Flow Matching-based Speech Enhancement" defines FlowSE as a conditional continuous normalizing flow for recovering clean speech from noisy observations, with emphasis on low-NFE inference without any additional fine-tuning procedure [2508.06840]. The later 2025 paper "FlowSE: Efficient and High-Quality Speech Enhancement via Flow Matching" uses the same name for a mel-spectrogram-based speech enhancement framework composed of a mel encoder, a latent Diffusion Transformer, and a neural vocoder, optionally conditioned on character transcripts [2505.19476].

A concise summary of the main variants is useful because the shared name masks substantial methodological differences.

| Work | Core formulation | Reported emphasis |
|---|---|---|
| FlowSE [2508.06840] | Conditional flow matching / conditional CNF | Comparable quality at NFE \(=5\) without extra fine-tuning |
| FlowSE [2505.19476] | Flow matching on mel spectrograms with optional text | High quality, low latency, real-time factor reporting |
| FlowSE-GRPO [2601.16483] | Online RL post-training of FlowSE | Metric alignment and reward-design analysis |
| Autonomous Rectified Flow [2606.20001] | Time-unconditional rectified flow | Challenges explicit time conditioning |

The shared conceptual core is a deterministic transport view of speech enhancement: rather than simulating a long stochastic reverse process, the model learns a vector field that maps noisy or latent initial states toward clean speech. A plausible implication is that "FlowSE" is best understood not as a single architecture, but as a family of flow-based SE formulations occupying a common design space.

## 2. Mathematical formulations

In the conditional-CNF formulation, the task is posed as recovering clean speech \(s\) from a noisy observation \(y=s+n\) by learning a continuous-time flow
\[
\frac{dx_t}{dt} = v_t(x_t \mid y), \qquad x_0 \sim \mathcal N(y,\sigma^2 I), \quad t\in[0,1],
\]
with \(x_1 \approx q(s\mid y)\). Training uses the Conditional Flow Matching loss
\[
L_{\mathrm{CFM}}(\theta)=\mathbb{E}_{t\sim U[0,1-t_\delta],\,(x_1,y)\sim \mathrm{data},\,x_t\sim p_t(\cdot\mid x_1,y)}
\left\|v_\theta(x_t,t\mid y)-v_t(x_t\mid x_1,y)\right\|^2 .
\]
For the Gaussian conditional path
\[
\mu_t(x_1,y)=t\,x_1+(1-t)\,y,\qquad \sigma_t=(1-t)\sigma,
\]
the target field is given in closed form by
\[
v_t(x_t\mid x_1,y)=
-\frac{\sigma}{(1-t)\sigma}\Bigl(x_t-(t\,x_1+(1-t)\,y)\Bigr)+(x_1-y).
\]
The same work states that the corresponding diffusion model can be reinterpreted with drift \(f(x,y,t)=(y-x)/(1-t)\) and diffusion \(g(t)=\sqrt{2t\sigma^2/(1-t)}\) [2508.06840].

The mel-spectrogram formulation instead writes flow matching directly between noisy and clean speech distributions. Let \(p_0(z_0)=p(y)\) denote the noisy-speech distribution and \(p_1(z_1)=p(x)\) the clean-speech distribution. With linear interpolant
\[
z_t=\alpha(t)\,x+\bigl[1-\alpha(t)\bigr]\,y,\qquad \alpha(t)=t,
\]
the ground-truth velocity is
\[
v^*(z_t,t)=\frac{dz_t}{dt}=x-y.
\]
Conditioned on noisy input \(y\) and optional transcript \(c\), the estimator is
\[
v_\theta(z_t,t\mid y,c)=\mathcal F_\theta\bigl(z_t,t,\mathcal T(c)\bigr),
\]
and the training objective is
\[
\mathcal L_{\mathrm{flow}}
=
\mathbb E_{t\sim\mathcal U(0,1),\,x,y,c}
\Bigl[
\|v_\theta(z_t,t\mid y,c)-v^*(z_t,t)\|_2^2
\Bigr].
\]
This formulation adds an auxiliary \(\ell_1\) mel-spectrogram reconstruction term, yielding
\[
\mathcal L(\theta)=\mathcal L_{\mathrm{flow}}+\lambda\,\mathcal L_{\mathrm{mel}}
\]
with \(\mathcal L_{\mathrm{mel}}=\mathbb E_{x,y}\|M_x-\hat M_x\|_1\) [2505.19476].

Both formulations use MSE supervision on velocity fields along analytically specified paths. This suggests that FlowSE methods derive much of their efficiency from deliberately choosing paths whose transport geometry is easier to integrate than reverse-SDE denoising trajectories.

## 3. Architectures and conditioning mechanisms

The conditional-CNF FlowSE adopts the same U-Net-style architecture as the BBED score model. The reported components are a time-embedding layer, a convolutional encoder-decoder based on NCSN++, conditioning on \(y\) via FiLM layers or by concatenating the noisy-speech STFT magnitude as an extra channel, and complex-valued STFT frames \((K\times F)\) that are amplitude compressed and processed as real 2-channel inputs. Total model size, feature-map widths, depth, and hyperparameters are stated to match Lay et al. for fair comparison [2508.06840].

The mel-spectrogram FlowSE is explicitly modular. It contains a mel-spectrogram encoder \(\mathcal E\), a flow-matching generative module \(\mathcal F\), and a neural vocoder \(\mathcal V\). Given waveform \(y\in\mathbb R^T\), the model computes
\[
M_y=\mathrm{Mel}(\mathrm{STFT}(y))\in\mathbb R^{F\times T'}.
\]
The core velocity model is a latent Diffusion Transformer operating on log-mel frames with \(N=22\) transformer blocks, 16 attention heads, hidden dimension \(d_{\mathrm{model}}=1024\), and feed-forward dimension \(d_{\mathrm{ff}}=2048\). At each block it injects a sinusoidal time embedding \(\tau(t)\) of dimension 1024 and an optional text embedding \(\phi(c)\in\mathbb R^{512}\) produced by ConvNeXt V2:
\[
v_\theta(M_t,t,c)=\mathrm{DiT}_\theta\bigl(M_t+W_t\,\tau(t),\phi(c)\bigr).
\]
Transcripts are tokenized as character sequences, and text is dropped during training with probability \(p_{\mathrm{drop}}=0.2\) to enable text-free operation. The enhanced mel spectrogram is rendered to waveform through a pre-trained neural vocoder, reported as Vocos or BigVGAN in the main FlowSE paper and HiFi-GAN in the later GRPO extension [2505.19476][2601.16483].

The architectural contrast is substantial. One line remains close to diffusion-style score-model backbones for complex STFT features; the other moves toward latent-transformer generation over mel spectrograms with optional linguistic conditioning. The commonality lies in the learned velocity field, not in a canonical network design.

## 4. Inference, NFEs, and latency

In the conditional-CNF formulation, enhancement is performed by integrating
\[
\frac{dx_t}{dt}=v_\theta(x_t,t\mid y)
\]
from \(x_0\sim\mathcal N(y,\sigma^2 I)\) to \(t=1\). The interval \([0,1]\) is discretized into \(N+1\) time points and integrated with explicit Euler:
\[
x_i=x_{i-1}+v_\theta(x_{i-1},t_{i-1}\mid y)\,\Delta t_i,\qquad \Delta t_i=t_i-t_{i-1}.
\]
Here NFE is simply \(N\), the number of calls to \(v_\theta\). The paper attributes the low-NFE behavior to an optimal-transport-inspired straight-line path in distribution space and states that FlowSE requires as few as \(N=5\) steps, or even fewer, to achieve convergence in distribution, whereas standard diffusion-based SDE solvers typically need 25 to 60 steps [2508.06840].

In the mel-spectrogram formulation, single-pass sampling starts from Gaussian noise \(M_{t=0}\sim\mathcal N(0,I)\) and integrates
\[
M_{t+\Delta t}=M_t+v_\theta(M_t,t,c)\,\Delta t
\]
for \(t\) from 1 down to 0 in \(K\) steps, with \(K=10\) given as an example. Each step costs one DiT forward pass with complexity \(\mathcal O(N\,T'\,d_{\mathrm{model}})\), so total cost is approximately \(K\times\) one pass. The paper reports that \(K\ll100\) relative to diffusion and therefore yields much lower latency; it also reports that quality saturates by \(K=10\), with OVRL drop \(<0.02\) if \(K=5\), while real-time factor halves when \(K\to5\) [2505.19476].

The reported latency measurements make the efficiency claim concrete. On DNS 2021, FlowSE is reported at RTF \(=0.31\), compared with \(3.30\) for CDiffuSE, \(3.27\) for SGMSE, and \(3.49\) for StoRM; the paper summarizes this as \(>10\times\) faster inference [2505.19476]. In the conditional-CNF study, reducing NFE from \(60\) to \(5\) is described as yielding roughly a \(12\times\) speedup in the score/flow model runs, with negligible loss in perceptual quality [2508.06840].

## 5. Reported empirical results

The conditional-CNF FlowSE is evaluated on WSJ0-CHiME3 / VB-DMD. At NFE \(=60\), the BBED diffusion model is reported at PESQ \(\approx 3.00\) and SI-SDR \(\approx 18.98\,\mathrm{dB}\). At NFE \(=5\), BBED without fine-tuning drops to PESQ \(\approx 2.90\) and SI-SDR \(\approx 18.33\,\mathrm{dB}\), while CRP reaches PESQ \(\approx 3.02\) and SI-SDR \(\approx 19.71\,\mathrm{dB}\). FlowSE at NFE \(=5\) is reported as matching or slightly exceeding CRP in most metrics, with PESQ \(\approx 3.03\), SI-SDR \(\approx 18.95\,\mathrm{dB}\), and DNSMOS, ESTOI, and related measures described as comparable; the paper also states that 5-step FlowSE is approximately equivalent to CRP(5) and BBED(60) [2508.06840].

The mel-spectrogram FlowSE is evaluated on DNS Challenge 2021 and a simulated VCTK+WHAM!/DEMAND setup. On DNS 2021, the reported OVRL / speaker-similarity / RTF values are \(3.331 / 0.801 / 0.31\) without text and \(3.340 / 0.809 / 0.31\) with text. For comparison, MaskSR is reported at OVRL \(=3.253\), speaker similarity \(=0.827\), RTF \(=0.25\); TF-GridNet at OVRL \(=2.805\), speaker similarity \(=0.815\), RTF \(=0.10\); and diffusion baselines between OVRL \(=2.190\) and \(2.516\) with RTF near \(3.3\). On the simulated set, FlowSE is reported at WER \(=8.81\) without text and \(8.79\) with text, both at RTF \(=0.31\), compared with \(15.31\) for CDiffuSE, \(13.97\) for SGMSE, and \(14.00\) for StoRM [2505.19476].

The same paper reports that text conditioning gives a small but measurable gain, moving OVRL from \(3.331\) to \(3.340\) and speaker similarity from \(0.801\) to \(0.809\). It also reports that a DiT-Large model with \(N=22\) improves OVRL by \(\sim 0.05\) relative to DiT-Small at \(+20\%\) compute, and that diminishing returns appear beyond \(K=10\) integration steps [2505.19476].

A later post-training study reports that online GRPO can further improve a base FlowSE model. On DNS2020 No-Reverb, the paper reports SIG \(3.685\to3.753\), BAK \(4.201\to4.248\), OVRL \(3.445\to3.549\), SPK \(88.88\to90.43\), and SBS \(86.35\to86.72\). On the With-Reverb test, it reports SIG \(3.601\to3.740\), BAK \(4.102\to4.251\), OVRL \(3.331\to3.530\), SPK \(73.72\to77.75\), and SBS \(73.62\to75.89\). On a real-recording test set, OVRL is reported as \(3.115\to3.356\) [2601.16483].

## 6. Extensions, misconceptions, and open questions

FlowSE-GRPO extends a pretrained flow-matching SE model with online Group Relative Policy Optimization. The paper formulates an MDP with state \(s_t=(c,t,x_t)\), action \(a_t=x_{t-1}\), deterministic transitions, and final reward \(R(s_0,a_0)=r(x_0,c)\). Because GRPO requires stochastic rollouts, it replaces part of the deterministic ODE sampler with an equivalent SDE sampler,
\[
x_{t+\Delta t}
=x_t+
\Bigl[
v_\theta(x_t,t)+\frac{\sigma_t^2}{2(1-t)}\bigl(-x_t+t\,v_\theta(x_t,t)\bigr)
\Bigr]\Delta t
+\sigma_t\sqrt{\Delta t}\,\epsilon,
\]
with \(\sigma_t=a\sqrt{(1-t)/t}\), and restricts stochastic training to a window \(S=[S_{\min},S_{\min}+ws]\) to reduce compute. A central empirical conclusion is that single-metric optimization rapidly increases the chosen reward but often induces reward hacking that degrades audio fidelity despite higher scores. The proposed mitigation is a multi-metric reward combining DNSMOS, speaker similarity, and SpeechBERTScore, normalized by empirical standard deviation and weighted with \(\lambda_1=0.6\), \(\lambda_2=\lambda_3=1.0\) [2601.16483].

A separate line of work, "Time-Unconditional Generative Speech Enhancement via Autonomous Rectified Flow," challenges an assumption implicit in most FlowSE variants: that explicit time-step conditioning is necessary. Under the linear interpolation
\[
x(t)=(1-t)x_0+t(y+\sigma z),
\]
the target vector field is
\[
u=\frac{dx}{dt}=(y+\sigma z)-x_0=n+\sigma z,
\]
which the paper argues is time-invariant. It therefore removes all time-step or noise-scale embeddings from an NCSN++-style U-Net and performs inference with repeated evaluations of the same time-unconditional field. Reported comparisons against FlowSE include PESQ \(=3.11\) vs. \(3.05\) at NFE \(=5\), and PESQ \(=3.00\) vs. \(2.86\) at NFE \(=1\), with RTF \(=0.13\) at NFE \(=5\) and \(0.02\) at NFE \(=1\) [2606.20001].

This time-conditioning debate is the main conceptual controversy surrounding the FlowSE design space. One position emphasizes explicit time-dependent flows, conditional embeddings, and carefully designed probability paths; the other argues that, for a linear interpolation path, temporal conditioning is redundant. The available evidence supports neither a universal equivalence nor a universal superiority claim across all datasets and feature spaces. What it does establish is that speech enhancement by flow transport remains an active methodological frontier, with open directions explicitly identified as adaptive step-size ODE solvers, better coupling functions, distillation or model compression for on-device real-time SE, extension to multi-channel or reverberant scenarios, joint learning of flow and dispersion \(g(t)\), adaptive \(\sigma\) or non-Gaussian perturbations, and combination with universal SE backbones [2508.06840][2606.20001].

Source: https://www.emergentmind.com/topics/flowse