SenSE: Semantic-Aware Speech Enhancement

• SenSE is a semantic-aware, high-fidelity universal speech enhancement framework that integrates language-model-derived semantic tokens with flow matching for robust acoustic restoration.
• It employs a two-stage process featuring a semantic-aware speech language model and semantically guided flow matching to mitigate semantic ambiguity and acoustic discontinuity.
• Empirical evaluations demonstrate improvements in perceptual quality, semantic fidelity (dWER, SpeechBERTScore), and speaker similarity (SIM-o) compared to prior methods.

Searching arXiv for the SenSE paper and directly related cited components. {"query":"(Li et al., 29 Sep 2025) SenSE Semantic-Aware High-Fidelity Universal Speech Enhancement", "max_results": 5} {"query":"(Du et al., 2024) CosyVoice tokenizer speech LLM", "max_results": 5} SenSE is a semantic-aware high-fidelity universal speech enhancement framework that integrates language-model-derived semantic information into a flow-matching-based speech enhancement system. It is designed for universal speech enhancement settings in which speech may be degraded by noise, reverberation, clipping, or bandwidth limitation, and it targets a failure mode of generative enhancement systems that learn only an acoustic mapping from degraded speech to clean speech: acoustically plausible outputs that remain semantically ambiguous or acoustically discontinuous. SenSE addresses this by introducing a semantic-aware speech LLM, a semantic guidance mechanism for flow matching, and an optional prompt guidance mechanism for speaker preservation under severe distortions (Li et al., 29 Sep 2025).

1. Problem formulation and motivation

Universal speech enhancement seeks to recover clean, intelligible, high-fidelity speech from inputs affected by diverse distortions. In the formulation adopted by SenSE, recent diffusion-based and flow-matching generative methods are effective at reconstructing continuous acoustic features and thus avoid the heavy quantization loss associated with codec- or token-based approaches, but they typically do not model high-level semantic information explicitly. The paper identifies three resulting difficulties: semantic ambiguity, acoustic discontinuity, and loss of speaker consistency under severe degradation (Li et al., 29 Sep 2025).

The motivation is explicitly cognitive as well as algorithmic. Humans can often comprehend heavily corrupted speech by relying on semantic priors, which the work treats as evidence that semantics are not merely auxiliary cues but a central component of robust enhancement. SenSE therefore reorients generative speech enhancement away from a purely low-level acoustic restoration objective and toward a joint semantic-acoustic reconstruction problem. This positioning is important because the framework does not discard flow matching; rather, it augments a strong generative backbone with semantic conditioning.

2. Semantic-aware speech LLM

The first stage of SenSE is the semantic-aware speech LLM, abbreviated SASLM, whose function is to extract “purified” semantic tokens from degraded speech. The tokenizer is the CosyVoice $\mathcal{S}^3$ Tokenizer, described as being trained under ASR supervision with vector quantization or finite scalar quantization and as producing discrete tokens that are more robust and more closely aligned with textual meaning (Du et al., 2024). These tokens provide the semantic interface between degraded speech and the downstream enhancement model.

The SASLM audio encoder is initialized with a pretrained Whisper-large-v3 encoder followed by an adapter layer. The quantization layer after the encoder is removed, and the model uses unquantized embeddings as a prefix for an autoregressive Transformer. Training is organized in two stages: first the encoder is frozen and only the LLM is trained; then the LLM is frozen and the audio encoder is moderately fine-tuned. The inference objective is to generate semantic tokens that represent clean semantic content inferred from noisy speech (Li et al., 29 Sep 2025).

The token prediction process is specified with the input pattern ["<SOS>", speech embedding, "<ToT>", semantic token, "<EOS>"] and the next-token factorization

$$p(x) = \prod_{i=1}^{n} p\left(s_k \mid s_1, \ldots, s_{k-1}, e_{1..n}\right),$$

where $s$ denotes semantic tokens and $e$ denotes speech embeddings. This design makes the semantic pathway explicit: the model is not asked to predict clean acoustics directly, but to infer a linguistically structured representation that can later constrain acoustic generation (Li et al., 29 Sep 2025).

3. Semantically guided flow matching

The second stage is a flow-matching speech enhancement module conditioned on both acoustic and semantic inputs. Its inputs are purified semantic tokens from SASLM, a noisy mel spectrogram, and optionally reference tokens for speaker guidance. The target of generation is a clean mel spectrogram, which is subsequently rendered into waveform form by a BigVGAN vocoder (Li et al., 29 Sep 2025).

A central implementation issue is that semantic tokens and mel spectrograms have different temporal resolutions. SenSE addresses this by padding semantic tokens with filler tokens to match mel length and then encoding the aligned sequence with ConvNeXt V2. The resulting representation is combined with masked clean and degraded acoustic cues: $m_1 \odot x_1$ for the masked clean signal and $m_2 \odot y$ for the masked degraded signal, where $m_1$ and $m_2$ are binary masks. The paper states that this construction forces the model to exploit both semantic information and the remaining acoustic evidence rather than collapsing onto a purely acoustic shortcut (Li et al., 29 Sep 2025).

The flow-matching objective is summarized as

$$\mathcal{L}_{\text{FM}} = \mathbb{E}_{t, x_0, x_1} \left\| v_\theta\left((1-t)x_0 + tx_1, m_1 x_1, m_2 y, z\right) - (1-m_1)(x_1 - x_0) \right\|^2,$$

where $x_0 \sim \mathcal{N}(0, I)$, $$p(x) = \prod_{i=1}^{n} p\left(s_k \mid s_1, \ldots, s_{k-1}, e_{1..n}\right),$$0, $$p(x) = \prod_{i=1}^{n} p\left(s_k \mid s_1, \ldots, s_{k-1}, e_{1..n}\right),$$1 is the clean mel spectrogram, $$p(x) = \prod_{i=1}^{n} p\left(s_k \mid s_1, \ldots, s_{k-1}, e_{1..n}\right),$$2 is the degraded mel spectrogram, $$p(x) = \prod_{i=1}^{n} p\left(s_k \mid s_1, \ldots, s_{k-1}, e_{1..n}\right),$$3 denotes semantic tokens, and $$p(x) = \prod_{i=1}^{n} p\left(s_k \mid s_1, \ldots, s_{k-1}, e_{1..n}\right),$$4 is the learned velocity field (Li et al., 29 Sep 2025). In this formulation, semantic conditioning is not a post hoc reranking signal; it is injected into the generative dynamics.

A distinctive training device is degraded signal masking, described in the paper as “degrad mask.” Random temporal masking is applied to the degraded input during training. The stated rationale is that, without masking, the model over-relies on degraded acoustic features and may ignore semantics. By obscuring portions of the degraded observation, SenSE forces reconstruction of masked regions through the joint use of semantic tokens and available acoustic cues (Li et al., 29 Sep 2025).

4. Prompt guidance and speaker preservation

SenSE includes an optional prompt guidance mechanism intended to alleviate the loss of speaker similarity under severe distortion conditions. The model accepts a short clean reference utterance from the target speaker and derives both reference semantic tokens and a reference mel spectrogram from it. These are padded and concatenated with the main input so that the enhancement model can condition not only on content semantics but also on speaker-specific acoustic traits (Li et al., 29 Sep 2025).

This mechanism is specifically motivated by cases in which extreme distortions erase speaker identity cues. The paper treats speaker preservation as a distinct objective from semantic fidelity and perceptual quality. In that framing, prompt guidance is not required for ordinary enhancement, but it acts as a targeted conditioning channel when the enhancement problem becomes underdetermined from degraded acoustics alone. The reported effect is improved objective speaker similarity, measured by SIM-o, with further gains when prompt guidance is enabled (Li et al., 29 Sep 2025).

At inference time, the system therefore admits two operating modes. In the standard mode, degraded mel features and generated semantic tokens are sufficient to drive flow matching. In the speaker-guided mode, the model additionally consumes a short reference utterance, which provides a prompt-like constraint on timbre and identity. This dual-mode design preserves the universality of the framework while allowing an explicit intervention for speaker recovery.

5. Evaluation protocol and empirical findings

The evaluation covers speech denoising on the DNS Challenge, composite restoration involving noise, reverberation, clipping, and band limitation, and robustness tests under extreme or unseen distortions, including low-SNR settings such as $$p(x) = \prod_{i=1}^{n} p\left(s_k \mid s_1, \ldots, s_{k-1}, e_{1..n}\right),$$5 to $$p(x) = \prod_{i=1}^{n} p\left(s_k \mid s_1, \ldots, s_{k-1}, e_{1..n}\right),$$6 dB. The reported metrics are DNSMOS and NISQA for reference-free perceptual quality, SpeechBERTScore and dWER for semantic similarity and word error rate behavior, PESQ and ESTOI for waveform-level fidelity, and SIM-o for speaker similarity (Li et al., 29 Sep 2025).

The main empirical claim is that SenSE ensures high perceptual quality while substantially improving speech fidelity and maintaining robustness under severe distortions. On DNSMOS and NISQA, it matches or surpasses the best baselines among both discriminative and generative models. On semantic fidelity, it achieves the best dWER and the highest SpeechBERTScore, and the paper explicitly states that these results outperform even discriminative models. On speaker similarity, SIM-o matches or surpasses previous generative models, with further improvements when prompt guidance is used (Li et al., 29 Sep 2025).

The ablation results are structurally important. Removing semantic guidance or degraded masking causes clear drops in both DNSMOS and semantic metrics. This supports the paper’s central argument that language-model-based semantic conditioning is not an incidental add-on but a causal contributor to both intelligibility and fidelity. The qualitative analysis is consistent with this reading: spectrogram visualizations and ASR transcriptions are reported to show better preservation of high-frequency content and transient consonants than typical discriminative models, together with reduced ambiguities or “hallucinations” in heavily degraded conditions (Li et al., 29 Sep 2025).

6. Interpretation within speech enhancement research

SenSE occupies a specific position within generative universal speech enhancement. It retains the continuous acoustic synthesis advantages of flow matching while introducing an explicit high-level semantic channel. This differentiates it from systems that treat enhancement as a direct mapping from degraded acoustics to clean acoustics and from approaches that rely primarily on discretized acoustic tokens. The architecture thereby links speech enhancement to speech language modeling without collapsing one task into the other (Li et al., 29 Sep 2025).

A common misconception in speech enhancement is that perceptual naturalness alone is sufficient evidence of successful restoration. SenSE directly challenges that view by elevating semantic fidelity to a first-class objective, measured with dWER and SpeechBERTScore, and by treating speaker similarity as a separate axis rather than a by-product. The framework’s design implies that high-quality enhancement must preserve what was said, how coherently it is rendered acoustically, and, in severe cases, who is speaking.

The broader significance is methodological. SenSE suggests that explicit semantic extraction and conditioning can be integrated into high-capacity generative enhancement models without abandoning mel-spectrogram-based continuous generation. In the formulation presented, semantic tokens are not merely auxiliary annotations; they are operational variables that shape the velocity field learned by flow matching. Code and demos are reported as available, which positions the model as both a research result and a reusable framework for future work on semantically grounded speech restoration (Li et al., 29 Sep 2025).