PhASE-Flow: Phonetic-Conditioned Acoustic Flow Matching in SSL Representation Domain for Speech Enhancement

Abstract: Flow matching (FM) enables high-fidelity generation, while self-supervised learning (SSL) speech models provide hierarchical representations spanning acoustic and phonetic levels. However, existing FM-based speech enhancement (SE) methods operate primarily in the spectral domain, treating SSL features only as external conditions rather than modeling directly in the SSL latent space. To fully exploit the structural richness of SSL representations, we propose PhASE-Flow, an FM-based SE framework that operates entirely in the SSL space. It models the conditional distribution of clean acoustic representations given phonetic ones, reconstructing the waveform via a neural vocoder. Experiments show that PhASE-Flow outperforms state-of-the-art baselines in perceptual quality and intelligibility. Notably, it achieves competitive performance with only four sampling steps, enabling highly efficient inference. Audio demos are available at https://anonymous.4open.science/w/phase-flow_demo-E6E1/.

• The paper introduces a novel speech enhancement framework that uses flow matching in the SSL representation domain conditioned on phonetic embeddings.
• It employs disentangled WavLM features and a DiT-based flow matching module to align acoustic detail with linguistic content, enhancing both perceptual quality and speaker similarity.
• Experimental results on DNS 2020 benchmark datasets show improved metrics in perceptual quality, linguistic integrity, and inference efficiency with only four ODE sampling steps.

PhASE-Flow: Phonetic-Conditioned Acoustic Flow Matching in SSL Representation Domain for Speech Enhancement

Motivation and Problem Formulation

Conventional speech enhancement (SE) systems either apply discriminative techniques or generative models in spectral domains (Mel, STFT), but these approaches are limited by their inability to reliably disentangle acoustic, linguistic, and speaker-related features. Recent advances in self-supervised learning (SSL) speech models, such as WavLM, provide hierarchical representations—lower layers encode fine-grained acoustic properties while higher layers capture phonetic content. Flow Matching (FM) has emerged as a generative modeling technique yielding high-fidelity outputs yet typically relies on spectral features as its modeling domain. Prior attempts to introduce SSL representations as conditioning signals merely augment the spectral domain, leaving the representational potential of the SSL latent space underexploited.

PhASE-Flow addresses these limitations by modeling the conditional distribution of clean acoustic SSL representations given phonetic SSL embeddings, operating entirely within the SSL domain. This framework leverages disentangled WavLM features to align semantic content and acoustic detail, with FM as its generative backbone and a neural vocoder for waveform synthesis.

Framework Architecture

PhASE-Flow consists of three principal modules:

• WavLM Encoder: Frozen, used to extract acoustic (lower-layer) and phonetic (upper-layer) representations from noisy speech.
• DiT-based Flow Matching Module: Trains to model the manifold of clean acoustic SSL representations conditioned on phonetic SSL representations. The generative process is formulated as an ODE, leveraging optimal transport conditional vector fields.
• Neural Vocoder (Vocos): Maps enhanced acoustic SSL features back to waveform space, trained independently for optimal fidelity.

The DiT-based backbone ingests noisy acoustic and phonetic representations, intermediate latent states, and flow step indices, with random acoustic feature dropout during training to encourage robust conditioning. Training utilizes the "x-pred" loss (direct mapping to clean acoustic representations), yielding stable convergence and improved performance.

Representational Advantages and Modeling Efficacy

PhASE-Flow's design exploits the structured hierarchy within SSL representations. Acoustic embeddings provide granularity essential for perceptual reconstruction and speaker similarity, while phonetic embeddings supply coherent linguistic conditioning. Operating solely in the SSL domain circumvents the pitfalls of Mel and STFT—loss of phase, entanglement, heavy-tailed distributions, and representational mismatch—improving both modeling stability and output quality.

Ablation studies confirm that SSL acoustic domain modeling provides superior perceptual metrics compared to Mel (Flow-M) or STFT (Flow-S) domains. Phonetic-only variants (Flow-P) improve linguistic metrics but degrade speaker similarity, while flow matching with Mel-phonetic conditioning (Flow-M-P) underperforms SSL-domain modeling in overall quality. The use of a neural vocoder as a generative prior further enhances perceptual quality, outperforming vocoder-free baselines.

Experimental Results and Benchmarking

Evaluations on the DNS 2020 synthetic test sets demonstrate that PhASE-Flow achieves or surpasses state-of-the-art performance across multiple axes:

• Perceptual Quality (DNSMOS, UTMOS): PhASE-Flow consistently achieves high DNSMOS (3.40–3.36) and leads in UTMOS (4.11–3.81).
• Representation Similarity (SpeechBERTScore, LPS, SpkSim): Achieves top SBS (0.93–0.85), LPS (0.97–0.90), and competitive SpkSim (0.94–0.75), confirming robust preservation of both linguistic content and speaker identity.
• Linguistic Integrity (dWER): Exhibits lowest hallucination rates for generative models (2.79–13.19% dWER), outperforming diffusion and LLM-based baselines that frequently degrade transcription accuracy.

Crucially, PhASE-Flow attains these results using only four ODE sampling steps, dramatically improving inference efficiency compared to diffusion-based approaches requiring lengthy iterative sampling.

Implications and Future Directions

PhASE-Flow demonstrates the practical advantage of generative modeling within the SSL representation domain, aligning semantic and acoustic spaces for structured speech enhancement. The results underscore the efficacy of FM applied to SSL features, bridging existing gaps between perceptual quality, speaker similarity, and linguistic fidelity. This approach mitigates the prevalent hallucination artifacts of generative paradigms and drastically reduces inference latency.

Theoretically, this framework sets a precedent for leveraging hierarchical SSL embeddings as principal modeling substrates for generative audio processing. Practically, its efficiency and robustness position PhASE-Flow for deployment in real-time speech enhancement systems and extension to broader speech tasks, such as separation, restoration, and multimodal generation. As SSL models and generative transport frameworks evolve, further optimization and adaptation can be anticipated for low-latency, high-fidelity speech technologies.

Conclusion

PhASE-Flow introduces a fundamentally novel SE paradigm wherein flow matching operates exclusively within the SSL representation domain, specifically modeling acoustic representations conditioned on phonetic context. The method achieves superior perceptual quality, speaker similarity, and linguistic integrity with minimal computational overhead. Its contributions delineate a clear advantage for hierarchical SSL representation modeling, offering both theoretical refinement and formidable practical gains in generative speech enhancement (2606.17806).