---
title: 'CosyWhisper: Text-to-Whisper Synthesis'
url: https://www.emergentmind.com/topics/cosywhisper
type: topic
---

# CosyWhisper: Text-to-Whisper Synthesis

Searching arXiv for papers explicitly related to “CosyWhisper” and adjacent Whisper-based work.
CosyWhisper most explicitly denotes an open-source text-to-whisper system obtained by fine-tuning CosyVoice3 on the WhispSynth corpus, a multilingual whispered-speech resource constructed through real-data curation and a Differentiable Digital Signal Processing (DDSP)-based pitch-free generative framework. The system is intended to generate natural, intelligible, pitch-free whispered speech from text, addressing the scarcity of high-quality whisper data and the residual pitch artifacts that conventional text-to-speech pipelines often retain when attempting whisper-like synthesis. In the reported evaluation, CosyWhisper is described as achieving speech naturalness on par with ground-truth samples [2603.14853].

## 1. Definition and nomenclatural scope

In the literature represented here, the clearest and most explicit use of the name **CosyWhisper** is the model introduced in "WhispSynth: Scaling Multilingual Whisper Corpus through Real Data Curation and A Novel Pitch-free Generative Framework" [2603.14853]. In that usage, CosyWhisper is a **text-to-whisper synthesis** system rather than an automatic speech recognition or speaker verification model.

The label is, however, not semantically unique across adjacent Whisper-based work. A different usage appears in work on whispered-speech speaker verification, where CosyWhisper denotes a post-processing system built on a fine-tuned ReDimNet-B6 backbone for robustness under normal-versus-whispered and whispered-versus-whispered trials [2604.20229]. In another case, a rare-word contextual-biasing model is more explicitly named **Bias-Whisper (B-Whisper)**, although the accompanying summary also associates it with a “CosyWhisper/B-Whisper” interpretation [2502.11572]. A further summary uses CosyWhisper to name a semi-supervised framework for **speech confidence detection** using Whisper embeddings plus acoustic features [2605.12387]. By contrast, **CarelessWhisper** is a distinct model for causal low-latency streaming ASR and is explicitly not a CosyWhisper system [2508.12301].

This naming overlap suggests that **CosyWhisper** should be interpreted contextually. In the most stable sense, it refers to the CosyVoice3-derived text-to-whisper model in the WhispSynth line of work [2603.14853].

## 2. Problem setting: text-to-whisper synthesis

The CosyWhisper system addresses whisper synthesis under three constraints identified in the WhispSynth study: **scarcity of high-quality whisper data**, **low recording fidelity of real whispers**, and a persistent **bias of TTS models toward phonated speech**, which causes residual periodicity or \(F_0\) leakage in purported whisper outputs [2603.14853]. The goal is therefore not merely to weaken voicing, but to synthesize speech that is simultaneously whispered, natural, intelligible, and acoustically consistent.

The underlying acoustic motivation is reinforced by adjacent whispered-speech research. Whispered speech differs from phonated speech in that it lacks vocal-fold vibration; it also exhibits upward formant shifts, reduced low-frequency energy in voiced consonants, and increased spectral flatness [2604.20229]. These properties help explain why ordinary TTS systems, even when capable of producing whisper-like timbre, may still leave perceptible pitch traces or other phonation-linked artifacts.

Within this problem formulation, CosyWhisper is positioned as a **specialized acoustic adaptation** rather than a full end-to-end redesign of language modeling or semantic generation. The WhispSynth paper states that semantic content remains the same and that the main difference between normal and whispered speech is acoustic, which motivates fine-tuning the acoustic model while leaving other CosyVoice3 components frozen [2603.14853]. A plausible implication is that the system treats whispering primarily as an acoustic-domain transformation conditioned on stable linguistic content.

## 3. Data foundation: WhispReal, WhispNJU, and WhispSynth

The data substrate for CosyWhisper consists of two intermediate resources: **WhispReal** and **WhispSynth**. WhispReal is a curated union of six public whispered-speech datasets plus a newly recorded corpus called **WhispNJU**. The included resources are **AISHELL6-Whisper**, **wTIMIT**, **Whisper40**, **CHAINS**, **Expresso**, **EARs**, and **WhispNJU** [2603.14853].

WhispNJU is the project’s newly constructed paired whisper/normal corpus. The paper reports **77 native Mandarin speakers**, with **37 male** and **40 female**, aged **22–26**, recorded at **44.1 kHz**. Each speaker recorded about **250 sentences**, roughly **1 hour per speaker**, and each sentence was recorded twice: once in normal speech and once in whispered speech. The sentence partitioning follows THCHS-30-style grouping: **A: 1–250**, **B: 251–500**, **C: 501–750**, and **D: 751–1000** [2603.14853].

The paper is internally inconsistent about WhispNJU duration. The prose mentions **85 hours** of paired whispered and normal speech, whereas the tabulated corpus statistics list **45.24 hours** for WhispNJU. The accompanying summary identifies the table as the more precise summary for the released subset [2603.14853]. This inconsistency is relevant because corpus scale is central to the paper’s contribution claims.

At corpus level, the tabulated statistics are:

- **WhispNJU**: **45.24 h**, **77 speakers**, **Mandarin Chinese**, **44.1 kHz**
- **WhispReal**: **117.62 h**, **479 speakers**, **English + Chinese**
- **WhispSynth**: approximately **118 h**, **479 speakers**, **English + Chinese**, **24 kHz** [2603.14853]

WhispSynth is generated from WhispReal through a pitch-free synthesis pipeline and is presented as a cleaner, more consistent training resource than raw whispered recordings. On the dataset evaluation table, **WhispReal** is reported with **DNSMOS 2.80**, **UTMOS 1.44**, **CER/WER 39.30 / 37.58**, and **VTR 0.88**, whereas **WhispSynth** is reported with **DNSMOS 2.89**, **UTMOS 1.46**, **CER/WER 31.16 / 20.98**, and **VTR 0.87** [2603.14853]. This suggests that the synthetic corpus improves intelligibility while keeping naturalness and residual voicing at roughly comparable levels.

## 4. Pitch-free generative framework

The central technical mechanism behind WhispSynth, and thus behind CosyWhisper’s training data, is a **DDSP-based pitch-free post-processing framework** integrated with TTS. The high-level pipeline is described as follows: first generate an initial whispered utterance using **CosyVoice3**; then use **DDSP pitch detection** to identify segments with residual \(F_0\); next decompose problematic segments into **harmonic** and **noise** parts; discard the harmonic component; retain the noise component; and finally reconstruct the waveform using **overlap-add (OLA)**. The paper characterizes this as a **pitch-aware segment replacement** pipeline [2603.14853].

The decomposition is written as
\[
\mathbf{S}[i] = \mathbf{H}[i] + \mathbf{N}[i],
\]
where \(\mathbf{S}[i]\) is the input speech frame, \(\mathbf{H}[i]\) the harmonic component, and \(\mathbf{N}[i]\) the noise component. The harmonic approximation is given by
\[
\widetilde{\mathbf{H}}[i] = \sum_{k=1}^{K} \frac{1}{k}\sin(\phi_k[i]),
\]
with phase accumulated from the estimated pitch \(\widetilde{F_0}[i]\),
\[
\phi_k[i] = 2\pi \sum_{n=0}^{N} k\widetilde{F_0}[i].
\]
The harmonic signal is then shaped by a linear time-varying FIR filter,
\[
\bar{\mathbf{H}}[i] = \widetilde{\mathbf{H}}[i] * \psi_h[i],
\]
while the noise component is modeled as
\[
\bar{\mathbf{N}}[i] = \zeta[i] * \psi_n[i],
\]
with \(\zeta \in [-1,1]\). The predicted parameter set is
\[
\Phi := \{\widetilde{F_0}[i], \psi_h[i], \psi_n[i]\}_{i=1}^{N},
\]
obtained from mel-spectrogram input \(\mathbf{X}\) via
\[
\Phi = f_{\mathrm{NN}}(\mathbf{X}).
\]
These equations formalize a source-filter separation in which residual periodicity can be explicitly removed from candidate whisper outputs [2603.14853].

The DDSP training procedure has three stages. First, the model is trained adversarially on normal speech within a **BigVGAN-style** framework using **Multi-resolution discriminator (MRD)**, **Multi-period discriminator (SPD)**, and **Least-squares GAN loss**. Second, it is continued on whispered speech from WhispReal **without adversarial objectives** for stability and faster training. Third, a **semi-supervised dual-focus training** scheme is used because WhispReal contains samples with residual pitch-like contours; the outputs are treated as harmonic plus stochastic components, and gradients are reweighted so that both whispered and normal speech inform the model [2603.14853].

## 5. CosyWhisper architecture and adaptation strategy

CosyWhisper is derived from **CosyVoice3**, which the paper describes as comprising a **speech LLM**, a **Conditional Flow Matching (CFM)** acoustic model, and a **HiFi-GAN vocoder** [2603.14853]. The adaptation strategy is deliberately narrow: for whisper generation, the authors fine-tune **only the CFM model**, while keeping both the **LLM frozen** and the **vocoder frozen**.

The stated rationale is that “for whispering, the semantic content stays the same; the difference is mostly acoustic” [2603.14853]. This identifies the acoustic model as the principal locus of whisper specialization. The approach therefore treats whispered speech synthesis as an acoustic-style transformation layered atop unchanged semantic and waveform-synthesis backbones.

The implementation required nontrivial modifications to the official CosyVoice3 training script, which initially supported only LLM training. The paper reports three changes for CFM training: **adding a token projection layer**, **replacing the encoder with a direct embedding lookup**, and **revising the conditioning mechanism** [2603.14853]. These modifications indicate that CosyWhisper is not only a dataset-driven fine-tune but also a targeted adaptation of the training interface around the acoustic model.

The reported training setup uses mel-spectrograms with **frame length = 1280** and **hop length = 320**. Batch size is **8** for adversarial training and **32** for standard training, on **8 × V100 32GB GPUs** [2603.14853]. The official implementation and related resources are available at **https://github.com/tan90xx/cosywhisper**.

## 6. Evaluation, empirical results, and limitations

CosyWhisper is evaluated with both subjective and objective criteria. Subjective evaluation uses **W-MOS**, a 5-point whisper-likeness mean opinion score ranging from “totally not similar” to “extremely similar,” based on **20 audio stimuli**, **20 participants**, and **Latin-square ordering** to reduce bias. Objective metrics include **DNSMOS**, **UTMOS**, **MCD**, **CER/WER**, **SpkSim** using **ECAPA-TDNN cosine similarity**, and **VTR** [2603.14853].

In the main text-to-whisper comparison against **Whisper-Effect**, **toWhisper**, **Normal2Whisper**, **SeedVC**, **CosyVoice3**, and **CosyWhisper**, the reported whisper-likeness scores are:

- **CosyVoice3**: **W-MOS = 3.40 ± 0.51**
- **CosyWhisper**: **W-MOS = 4.53 ± 0.20**
- **Ground truth**: **W-MOS = 4.33 ± 0.33** [2603.14853]

This is the basis for the paper’s claim that CosyWhisper achieves whisper-likeness on par with or better than ground-truth test-set whispers in that listening test. The accompanying objective results for CosyWhisper are **DNSMOS 3.08**, **UTMOS 1.48**, **CER/WER 12.76 / 29.22**, **SpkSim 0.80**, and **VTR 0.88** [2603.14853].

Ablation on training data quality compares CosyWhisper fine-tuned on **WhispReal** versus **WhispSynth**. The WhispReal-trained version is reported with **DNSMOS 2.93**, **UTMOS 1.33**, **CER/WER 28.3 / 46.5**, **Cosine 0.75**, and **VTR 0.77**, while the WhispSynth-trained version achieves **DNSMOS 3.08**, **UTMOS 1.48**, **CER/WER 12.8 / 29.2**, **Cosine 0.80**, and **VTR 0.70** [2603.14853]. The paper interprets this as evidence that **synthetic whisper data can outperform noisy real whisper data** for training text-to-whisper models.

The work also includes a small multilingual extension on **Korean** and **Japanese**, using manually curated whispered utterances from YouTube ASMR videos and native-speaker ratings; CosyWhisper outperformed the baseline CosyVoice3 in W-MOS for both languages [2603.14853]. This suggests some cross-lingual transfer, although the evaluation is much narrower than the main English-and-Mandarin setting.

Two limitations are explicitly listed. First, **microphone / hardware variation** was not systematically studied. Second, the released CosyWhisper model will include **watermarking** for responsible use, which may slightly affect acoustics [2603.14853]. These constraints are significant because whispered speech is unusually sensitive to recording conditions, and any added watermarking could interact with the very low-amplitude, noise-dominated character of whisper acoustics.

In sum, CosyWhisper occupies a specific position within Whisper-adjacent research: it is not a recognition system, a speaker verifier, or a streaming ASR method, but a **CosyVoice3-based text-to-whisper generator** trained on a curated-and-synthesized whisper corpus. Its technical identity is defined by three linked elements: **WhispReal** as curated source data, **WhispSynth** as a pitch-free synthetic corpus, and **CFM-only fine-tuning** of CosyVoice3 as the final synthesis mechanism [2603.14853].

Source: https://www.emergentmind.com/topics/cosywhisper