Time-Unconditional Generative Speech Enhancement via Autonomous Rectified Flow

Abstract: Most generative speech enhancement methods rely on explicit time-step embeddings for temporal conditioning. In this paper, we propose the Autonomous Rectified Flow framework, which challenges the necessity of such conditioning. Using a linear interpolation path, we show that the target vector field is inherently time-invariant. We further introduce a time-unconditional network that eliminates explicit time-step information and infers the denoising direction solely from the spatial relationship between the current state and the noisy observation. Predicting this target vector field is equivalent to modeling the noise distribution. By avoiding overfitting to temporal trajectories, the proposed autonomous design significantly improves generation quality, robustness, and inference efficiency.

• The paper introduces ARF, which removes temporal conditioning by demonstrating that linear interpolation between clean and noisy speech yields a constant denoising target.
• The methodology employs autonomous ODE integration, enabling single-step inference and reducing computational overhead, with competitive metrics like PESQ and SI-SDR.
• Empirical results on benchmarks such as VoiceBank+DEMAND and DNS Challenge highlight ARF’s robust generalization and efficiency compared to traditional time-dependent diffusion models.

Time-Unconditional Generative Speech Enhancement via Autonomous Rectified Flow

Overview

The examined paper proposes Autonomous Rectified Flow (ARF), a time-unconditional generative framework for speech enhancement that obviates the need for explicit time-step embeddings typically used in generative models. Traditional generative speech enhancement methods, especially those leveraging diffusion processes and flow matching, rely on temporal conditioning to guide denoising. ARF demonstrates that, under linear interpolation between noisy and clean speech, temporal conditioning is mathematically redundant. The paper details the ARF architecture, its empirical performance, and implications for efficient and robust speech enhancement.

Background and Related Methods

Classical speech enhancement approaches (e.g., Wiener filtering, spectral subtraction) have largely been supplanted by DNN-based discriminative models, but these often poorly generalize to unseen conditions. Generative models, particularly diffusion-based and score-based architectures, have achieved more robust speech recovery by better modeling underlying distributions. However, typical diffusion models suffer from high computational cost (iterative sampling with large NFE) and prior mismatch issues, since the Gaussian prior is often uninformative compared to the noisy mixture.

To address prior mismatch, boundary-anchored stochastic processes (e.g., Brownian Bridge with Exponential Diffusion) and flow-based architectures (e.g., FlowSE) define deterministic paths between the noisy mixture and clean speech. These often require explicit time-step conditioning and learn trajectory-specific denoising directions.

Autonomous Rectified Flow Framework

ARF is founded on the observation that, when the generative path is linear between clean and noisy speech, the optimal denoising direction (the target vector field) is constant and does not depend on $t$. Thus, temporal conditioning is unnecessary and potentially introduces overfitting and inefficiency.

ARF's generative process:

• Defines the interpolated state as $x_t = (1-t)x_0 + t(y + \sigma z)$, where $x_0$ is clean speech, $y$ is noisy, $\sigma$ is noise scaling, and $z$ is Gaussian noise.
• The denoising target, $$u = \frac{d}{dt}x_t = (y + \sigma z) - x_0 = n + \sigma z$$ (where $n$ is additive noise), is invariant to $t$.
• ARF trains a neural network $v_\theta(x_t, y)$ to predict this target vector field, optimizing the loss $x_t = (1-t)x_0 + t(y + \sigma z)$0.
• Generation is achieved via an autonomous ODE: $x_t = (1-t)x_0 + t(y + \sigma z)$1, integrating backward from the noisy observation to clean speech.

  Figure 1: The ARF framework removes the explicit time-step t in generative speech enhancement, modeling linear interpolation between clean and noisy speech.

Comparative Analysis of Flows

The paper positions ARF relative to prior rectified flow and flow matching methods. Figure 2 illustrates key differences:

• Rectified Flow and Conditional Flow Matching (CFM) require time-step or noise schedule embeddings; these condition the model and enforce trajectory-specific denoising.
• Autonomous Rectified Flow (ARF) eliminates explicit temporal conditioning, relying solely on the spatial relationship between state and noisy mixture.

This conceptual change enables ARF to generalize denoising regardless of the integration state's temporal position.

Figure 2: Comparison among Rectified Flow, Conditional Flow Matching, and Autonomous Rectified Flow; ARF uniquely removes explicit time-step dependency, using current position only.

Experimental Results

Core Performance

ARF was evaluated on VoiceBank+DEMAND and DNS Challenge datasets, using standard metrics: PESQ, eSTOI, SI-SDR, MOS, DNSMOS, and Real-Time Factor (RTF).

• At $x_t = (1-t)x_0 + t(y + \sigma z)$2, ARF achieved PESQ 3.11, eSTOI 0.88, SI-SDR 18.02 dB, and MOS 4.27.
• For single-step inference ($x_t = (1-t)x_0 + t(y + \sigma z)$3), ARF delivered PESQ 3.00, SI-SDR 19.91 dB, DNSMOS 3.58, and RTF 0.02.
• Compared to flow matching (FlowSE) and BBED baselines, ARF maintained performance parity or improvement, especially at low NFE (where typical flow-based systems degrade).
• Ablation studies on time-step embeddings showed that removing $x_t = (1-t)x_0 + t(y + \sigma z)$4 (ARF) improved both enhancement quality and inference velocity: e.g., at $x_t = (1-t)x_0 + t(y + \sigma z)$5, ARF reached RTF 0.02, versus FlowSE's 0.05.

Generalization

Cross-dataset evaluation on DNS Challenge demonstrated comparable out-of-distribution generalization between ARF and FlowSE. Both models experienced performance drops in severe reverberation conditions, but ARF did not regress relative to temporally conditioned flows.

Implications and Future Directions

ARF's time-unconditional design has several practical and theoretical implications:

• Inference Efficiency: Removing explicit temporal conditioning and noise schedules allows for low-NFE (single-step) inference with minimal quality loss and dramatically reduced RTF.
• Robustness: Autonomous ODE integration mitigates trajectory overfitting and enhances recovery from small numerical deviations, improving stability.
• Architectural Simplification: Eliminates the overhead and complexity associated with time-step embeddings and corresponding network modules.
• Generalization: Comparable performance in unseen conditions, even in reverberant environments, suggests ARF's suitability for robust speech enhancement.

Theoretically, ARF demonstrates that, for boundary-anchored linear generative tasks, temporal conditioning is mathematically unnecessary and can be detrimental. This should prompt reconsideration of such conditioning in other diffusion/flow-based generative frameworks.

Future research may extend ARF to universal speech enhancement, explore its integration with discriminative architectures, and further optimize autonomous ODE solvers for real-time applications.

Conclusion

Autonomous Rectified Flow establishes a time-unconditional generative paradigm for speech enhancement, obviating explicit temporal conditioning in linear-path scenarios. Empirical results demonstrate competitive speech quality and inference efficiency compared to state-of-the-art temporally conditioned flows, particularly in low-NFE regimes. The findings advocate for broader adoption of autonomous dynamical systems in generative speech enhancement and underscore the mathematical redundancy of time-step conditioning in linearly anchored generative processes.