VeRe-Flow: Guiding Flow Matching toward Clean Speech via Velocity Contrastive Regularization and Representation Alignment for Noise-Robust Bandwidth Expansion

Abstract: Noise-robust bandwidth expansion aims to reconstruct high-fidelity wideband speech from noisy low-resolution inputs. While flow matching has shown strong performance in speech generation, accurately recovering clean speech from noisy inputs remains challenging due to the ambiguity of velocity estimation under noise. In this work, we propose VeRe-Flow, a clean-guided flow matching framework that introduces multi-level clean supervision to guide the generative process toward clean speech. At the velocity level, we introduce velocity contrastive regularization, which attracts the predicted velocity toward the clean trajectory while repelling it from noisy trajectories. At the representation level, we incorporate representation alignment that aligns intermediate features with clean self-supervised learning representations. The results demonstrate that the proposed method achieves the lowest LSD and highest DNSMOS OVRL among all baselines, and the highest MOS among generative baselines.

• The paper introduces a clean-guided regularization framework combining velocity contrastive regularization and representation alignment to steer flow matching toward the clean speech manifold.
• It leverages noise-robust SSL features and a convolutional-transformer architecture to achieve superior spectral (LSD) and perceptual (DNSMOS, MOS) performance on noisy inputs.
• Empirical results on the Valentini-Botinhao dataset demonstrate that the approach outperforms both generative and non-generative baselines in noise-robust bandwidth expansion.

VeRe-Flow: Clean-Supervised Flow Matching for Robust Speech Bandwidth Expansion

Motivation and Background

Noise-robust bandwidth expansion (NR-BWE) presents the dual challenge of reconstructing high-fidelity wideband speech from noisy, low-resolution observations. Traditional bandwidth expansion (BWE) methods assume clean input and primarily focus on reconstructing truncated high-frequency components. Conversely, general speech enhancement methods are optimized for noise suppression, often without explicit high-frequency recovery. Recent flow matching generative models such as FLowHigh have excelled in clean BWE tasks by directly modeling deterministic trajectories in feature space. However, in noisy regimes, flow matching suffers from velocity ambiguity, causing the generative process to deviate from the clean speech manifold.

VeRe-Flow Framework

VeRe-Flow addresses the limitations of flow matching under noisy conditions by introducing explicit clean-guided regularization. The model augments the standard flow matching objective with two supervision mechanisms:

• Velocity Contrastive Regularization (VeCoR): The predicted velocity field is attracted towards the clean speech trajectory and repelled from noisy speech trajectories, explicitly counteracting the drift caused by noisy conditioning.
• Representation Alignment (REPA): Intermediate encoder representations are aligned with clean, noise-robust self-supervised learning (SSL) speech embeddings, mitigating the tendency of the model to rely on noisy input structure.

The architecture is composed of convolutional and transformer blocks assembled in a sandwich structure, incorporating Conv ResBlocks inspired by DiC-style designs for enhanced local feature learning and XEUS-based SSL features for noise-robust conditioning.

Figure 1: Training workflow of VeRe-Flow, illustrating explicit velocity regularization towards the clean speech manifold and embedding representation alignment.

Technical Contributions

Flow Matching and Conditional Velocity Formulation

VeRe-Flow builds upon conditional flow matching (CFM). Given noisy low-resolution mel-spectrograms and associated noise-robust SSL vectors, the model learns a continuous velocity field mapping between a simple source prior (Gaussian) and a high-resolution, clean target distribution in mel-spectrogram space. Unlike traditional CFM, VeRe-Flow’s velocity network receives both the standard flow-matching loss (toward the clean sample) and a contrastive regularization (repulsion from noisy targets). This bidirectional gradient signal sharpens the model’s convergence to the clean speech manifold.

Velocity Contrastive Regularization

Let $u_t^{\mathrm{clean}}$ and $u_t^{\mathrm{noisy}}$ represent clean and noisy target velocity fields, respectively, and $v_\theta$ the predicted velocity. VeCoR optimizes: $$\mathcal{L}_{\mathrm{VeCoR}} = \mathbb{E} \left[ \|v_\theta - u_t^{\mathrm{clean}}\|^2 - \lambda_{\mathrm{VeCoR}} \|v_\theta - u_t^{\mathrm{noisy}}\|^2 \right]$$ where $\lambda_{\mathrm{VeCoR}}$ adjusts the strength of contrastive repulsion, enhancing separation between clean and noisy data distributions.

Representation Alignment

To further decouple the latent modeling from noisy artifacts, REPA projects intermediate transformer outputs to the clean SSL feature space and maximizes framewise cosine similarity. This regularization enforces semantic consistency at the representation level, leveraging the generalization capabilities of SSL models pretrained for noise robustness.

Empirical Evaluation

Comprehensive experiments on the Valentini-Botinhao dataset (multi-condition: 84-speaker, 20-noise setup) demonstrate that VeRe-Flow consistently outperforms both generative (e.g., NU-Wave2, FLowHigh) and non-generative (e.g., EP-WUN, SDNet, codec-based) NR-BWE baselines in both spectral (LSD) and perceptual (DNSMOS, MOS) metrics.

Key results include:

• Lowest LSD and highest DNSMOS OVRL among all compared methods.
• Highest MOS within generative models.

Ablation studies show that XEUS SSL conditioning offers the most significant improvement in LSD, and VeCoR and REPA contribute additively to DNSMOS gains, especially in background noise suppression and perceptual quality. Notably, the model achieves strong performance even when the convolutional backbone is ablated, indicating the primacy of the clean-guided regularization mechanisms.

Model Architecture and Implementation Details

• Input: 80-dimensional mel-spectrograms (20ms frame, 1280pt window), XEUS SSL features (aligned to mel rate, noise-robust via dereverberation/denoising pretraining).
• Network: Stacked Conv ResBlocks (GroupNorm, SiLU, Conv1D), central transformer blocks; sandwich structure.
• Losses: Combined VeCoR (contrastive velocity) and REPA (representation alignment), with loss weights $\lambda_{\text{align}} = 0.25$, $\lambda_{\text{VeCoR}} = 0.05$.
• Training: 400k steps, Adam optimizer ($3 \times 10^{-4}$), cosine annealing, stochastic SNR noise perturbation (5–20 dB SNR).
• Vocoder: BigVGAN for waveform synthesis from predicted mel-spectrograms.

Implications and Future Directions

VeRe-Flow’s introduction of multi-level clean supervision in generative speech bandwidth expansion highlights the value of explicit trajectory regularization in flow-based models, especially for tasks involving adversarial or non-stationary corruptions. The combination of velocity-space and embedding-space alignment provides a generic blueprint potentially extensible to robust generative modeling of other modalities (e.g., music, environmental sounds, cross-domain speech).

Anticipated developments may include adaptive contrastive regularization schedules, more advanced SSL representations, and integration with streaming, low-latency inference for telephony or real-time communication systems. Additionally, the combination of clean-targeted flow matching with multi-scale or hierarchical architectures remains an open avenue for large-scale speech synthesis applications.

Conclusion

VeRe-Flow advances the state-of-the-art in noise-robust speech bandwidth expansion by combining flow matching with clean-targeted contrastive supervision and embedding alignment. The proposed model delivers clear improvements across both spectral and perceptual metrics, supported by rigorous ablation. Its architectural and methodological innovations are directly relevant for robust speech coding, communication, and restoration in noisy environments.