CosyWhisper: Text-to-Whisper Synthesis

• CosyWhisper is a text-to-whisper synthesis system that adapts CosyVoice3 by fine-tuning on curated and DDSP-synthesized whisper corpora.
• It utilizes a DDSP-based pitch-free framework to decompose speech signals and remove residual pitch, enhancing naturalness and intelligibility.
• Evaluations indicate that CosyWhisper achieves whisper-likeness scores on par with or exceeding ground-truth samples, outperforming models trained on raw whisper data.

Searching arXiv for papers explicitly related to “CosyWhisper” and adjacent Whisper-based work. CosyWhisper most explicitly denotes an open-source text-to-whisper system obtained by fine-tuning CosyVoice3 on the WhispSynth corpus, a multilingual whispered-speech resource constructed through real-data curation and a Differentiable Digital Signal Processing (DDSP)-based pitch-free generative framework. The system is intended to generate natural, intelligible, pitch-free whispered speech from text, addressing the scarcity of high-quality whisper data and the residual pitch artifacts that conventional text-to-speech pipelines often retain when attempting whisper-like synthesis. In the reported evaluation, CosyWhisper is described as achieving speech naturalness on par with ground-truth samples (Tan et al., 16 Mar 2026).

1. Definition and nomenclatural scope

In the literature represented here, the clearest and most explicit use of the name CosyWhisper is the model introduced in "WhispSynth: Scaling Multilingual Whisper Corpus through Real Data Curation and A Novel Pitch-free Generative Framework" (Tan et al., 16 Mar 2026). In that usage, CosyWhisper is a text-to-whisper synthesis system rather than an automatic speech recognition or speaker verification model.

The label is, however, not semantically unique across adjacent Whisper-based work. A different usage appears in work on whispered-speech speaker verification, where CosyWhisper denotes a post-processing system built on a fine-tuned ReDimNet-B6 backbone for robustness under normal-versus-whispered and whispered-versus-whispered trials (Gołębiowska et al., 22 Apr 2026). In another case, a rare-word contextual-biasing model is more explicitly named Bias-Whisper (B-Whisper), although the accompanying summary also associates it with a “CosyWhisper/B-Whisper” interpretation (Jogi et al., 17 Feb 2025). A further summary uses CosyWhisper to name a semi-supervised framework for speech confidence detection using Whisper embeddings plus acoustic features (Wynn et al., 12 May 2026). By contrast, CarelessWhisper is a distinct model for causal low-latency streaming ASR and is explicitly not a CosyWhisper system (Krichli et al., 17 Aug 2025).

This naming overlap suggests that CosyWhisper should be interpreted contextually. In the most stable sense, it refers to the CosyVoice3-derived text-to-whisper model in the WhispSynth line of work (Tan et al., 16 Mar 2026).

2. Problem setting: text-to-whisper synthesis

The CosyWhisper system addresses whisper synthesis under three constraints identified in the WhispSynth study: scarcity of high-quality whisper data, low recording fidelity of real whispers, and a persistent bias of TTS models toward phonated speech, which causes residual periodicity or $F_0$ leakage in purported whisper outputs (Tan et al., 16 Mar 2026). The goal is therefore not merely to weaken voicing, but to synthesize speech that is simultaneously whispered, natural, intelligible, and acoustically consistent.

The underlying acoustic motivation is reinforced by adjacent whispered-speech research. Whispered speech differs from phonated speech in that it lacks vocal-fold vibration; it also exhibits upward formant shifts, reduced low-frequency energy in voiced consonants, and increased spectral flatness (Gołębiowska et al., 22 Apr 2026). These properties help explain why ordinary TTS systems, even when capable of producing whisper-like timbre, may still leave perceptible pitch traces or other phonation-linked artifacts.

Within this problem formulation, CosyWhisper is positioned as a specialized acoustic adaptation rather than a full end-to-end redesign of language modeling or semantic generation. The WhispSynth paper states that semantic content remains the same and that the main difference between normal and whispered speech is acoustic, which motivates fine-tuning the acoustic model while leaving other CosyVoice3 components frozen (Tan et al., 16 Mar 2026). A plausible implication is that the system treats whispering primarily as an acoustic-domain transformation conditioned on stable linguistic content.

3. Data foundation: WhispReal, WhispNJU, and WhispSynth

The data substrate for CosyWhisper consists of two intermediate resources: WhispReal and WhispSynth. WhispReal is a curated union of six public whispered-speech datasets plus a newly recorded corpus called WhispNJU. The included resources are AISHELL6-Whisper, wTIMIT, Whisper40, CHAINS, Expresso, EARs, and WhispNJU (Tan et al., 16 Mar 2026).

WhispNJU is the project’s newly constructed paired whisper/normal corpus. The paper reports 77 native Mandarin speakers, with 37 male and 40 female, aged 22–26, recorded at 44.1 kHz. Each speaker recorded about 250 sentences, roughly 1 hour per speaker, and each sentence was recorded twice: once in normal speech and once in whispered speech. The sentence partitioning follows THCHS-30-style grouping: A: 1–250, B: 251–500, C: 501–750, and D: 751–1000 (Tan et al., 16 Mar 2026).

The paper is internally inconsistent about WhispNJU duration. The prose mentions 85 hours of paired whispered and normal speech, whereas the tabulated corpus statistics list 45.24 hours for WhispNJU. The accompanying summary identifies the table as the more precise summary for the released subset (Tan et al., 16 Mar 2026). This inconsistency is relevant because corpus scale is central to the paper’s contribution claims.

At corpus level, the tabulated statistics are:

• WhispNJU: 45.24 h, 77 speakers, Mandarin Chinese, 44.1 kHz
• WhispReal: 117.62 h, 479 speakers, English + Chinese
• WhispSynth: approximately 118 h, 479 speakers, English + Chinese, 24 kHz (Tan et al., 16 Mar 2026)

WhispSynth is generated from WhispReal through a pitch-free synthesis pipeline and is presented as a cleaner, more consistent training resource than raw whispered recordings. On the dataset evaluation table, WhispReal is reported with DNSMOS 2.80, UTMOS 1.44, CER/WER 39.30 / 37.58, and VTR 0.88, whereas WhispSynth is reported with DNSMOS 2.89, UTMOS 1.46, CER/WER 31.16 / 20.98, and VTR 0.87 (Tan et al., 16 Mar 2026). This suggests that the synthetic corpus improves intelligibility while keeping naturalness and residual voicing at roughly comparable levels.

4. Pitch-free generative framework

The central technical mechanism behind WhispSynth, and thus behind CosyWhisper’s training data, is a DDSP-based pitch-free post-processing framework integrated with TTS. The high-level pipeline is described as follows: first generate an initial whispered utterance using CosyVoice3; then use DDSP pitch detection to identify segments with residual $F_0$; next decompose problematic segments into harmonic and noise parts; discard the harmonic component; retain the noise component; and finally reconstruct the waveform using overlap-add (OLA). The paper characterizes this as a pitch-aware segment replacement pipeline (Tan et al., 16 Mar 2026).

The decomposition is written as

$\mathbf{S}[i] = \mathbf{H}[i] + \mathbf{N}[i],$

where $\mathbf{S}[i]$ is the input speech frame, $\mathbf{H}[i]$ the harmonic component, and $\mathbf{N}[i]$ the noise component. The harmonic approximation is given by

$$\widetilde{\mathbf{H}}[i] = \sum_{k=1}^{K} \frac{1}{k}\sin(\phi_k[i]),$$

with phase accumulated from the estimated pitch $\widetilde{F_0}[i]$,

$$\phi_k[i] = 2\pi \sum_{n=0}^{N} k\widetilde{F_0}[i].$$

The harmonic signal is then shaped by a linear time-varying FIR filter,

$$\bar{\mathbf{H}}[i] = \widetilde{\mathbf{H}}[i] * \psi_h[i],$$

while the noise component is modeled as

$F_0$0

with $F_0$1. The predicted parameter set is

$F_0$2

obtained from mel-spectrogram input $F_0$3 via

$F_0$4

These equations formalize a source-filter separation in which residual periodicity can be explicitly removed from candidate whisper outputs (Tan et al., 16 Mar 2026).

The DDSP training procedure has three stages. First, the model is trained adversarially on normal speech within a BigVGAN-style framework using Multi-resolution discriminator (MRD), Multi-period discriminator (SPD), and Least-squares GAN loss. Second, it is continued on whispered speech from WhispReal without adversarial objectives for stability and faster training. Third, a semi-supervised dual-focus training scheme is used because WhispReal contains samples with residual pitch-like contours; the outputs are treated as harmonic plus stochastic components, and gradients are reweighted so that both whispered and normal speech inform the model (Tan et al., 16 Mar 2026).

5. CosyWhisper architecture and adaptation strategy

CosyWhisper is derived from CosyVoice3, which the paper describes as comprising a speech LLM, a Conditional Flow Matching (CFM) acoustic model, and a HiFi-GAN vocoder (Tan et al., 16 Mar 2026). The adaptation strategy is deliberately narrow: for whisper generation, the authors fine-tune only the CFM model, while keeping both the LLM frozen and the vocoder frozen.

The stated rationale is that “for whispering, the semantic content stays the same; the difference is mostly acoustic” (Tan et al., 16 Mar 2026). This identifies the acoustic model as the principal locus of whisper specialization. The approach therefore treats whispered speech synthesis as an acoustic-style transformation layered atop unchanged semantic and waveform-synthesis backbones.

The implementation required nontrivial modifications to the official CosyVoice3 training script, which initially supported only LLM training. The paper reports three changes for CFM training: adding a token projection layer, replacing the encoder with a direct embedding lookup, and revising the conditioning mechanism (Tan et al., 16 Mar 2026). These modifications indicate that CosyWhisper is not only a dataset-driven fine-tune but also a targeted adaptation of the training interface around the acoustic model.

The reported training setup uses mel-spectrograms with frame length = 1280 and hop length = 320. Batch size is 8 for adversarial training and 32 for standard training, on 8 × V100 32GB GPUs (Tan et al., 16 Mar 2026). The official implementation and related resources are available at https://github.com/tan90xx/cosywhisper.

6. Evaluation, empirical results, and limitations

CosyWhisper is evaluated with both subjective and objective criteria. Subjective evaluation uses W-MOS, a 5-point whisper-likeness mean opinion score ranging from “totally not similar” to “extremely similar,” based on 20 audio stimuli, 20 participants, and Latin-square ordering to reduce bias. Objective metrics include DNSMOS, UTMOS, MCD, CER/WER, SpkSim using ECAPA-TDNN cosine similarity, and VTR (Tan et al., 16 Mar 2026).

In the main text-to-whisper comparison against Whisper-Effect, toWhisper, Normal2Whisper, SeedVC, CosyVoice3, and CosyWhisper, the reported whisper-likeness scores are:

• CosyVoice3: W-MOS = 3.40 ± 0.51
• CosyWhisper: W-MOS = 4.53 ± 0.20
• Ground truth: W-MOS = 4.33 ± 0.33 (Tan et al., 16 Mar 2026)

This is the basis for the paper’s claim that CosyWhisper achieves whisper-likeness on par with or better than ground-truth test-set whispers in that listening test. The accompanying objective results for CosyWhisper are DNSMOS 3.08, UTMOS 1.48, CER/WER 12.76 / 29.22, SpkSim 0.80, and VTR 0.88 (Tan et al., 16 Mar 2026).

Ablation on training data quality compares CosyWhisper fine-tuned on WhispReal versus WhispSynth. The WhispReal-trained version is reported with DNSMOS 2.93, UTMOS 1.33, CER/WER 28.3 / 46.5, Cosine 0.75, and VTR 0.77, while the WhispSynth-trained version achieves DNSMOS 3.08, UTMOS 1.48, CER/WER 12.8 / 29.2, Cosine 0.80, and VTR 0.70 (Tan et al., 16 Mar 2026). The paper interprets this as evidence that synthetic whisper data can outperform noisy real whisper data for training text-to-whisper models.

The work also includes a small multilingual extension on Korean and Japanese, using manually curated whispered utterances from YouTube ASMR videos and native-speaker ratings; CosyWhisper outperformed the baseline CosyVoice3 in W-MOS for both languages (Tan et al., 16 Mar 2026). This suggests some cross-lingual transfer, although the evaluation is much narrower than the main English-and-Mandarin setting.

Two limitations are explicitly listed. First, microphone / hardware variation was not systematically studied. Second, the released CosyWhisper model will include watermarking for responsible use, which may slightly affect acoustics (Tan et al., 16 Mar 2026). These constraints are significant because whispered speech is unusually sensitive to recording conditions, and any added watermarking could interact with the very low-amplitude, noise-dominated character of whisper acoustics.

In sum, CosyWhisper occupies a specific position within Whisper-adjacent research: it is not a recognition system, a speaker verifier, or a streaming ASR method, but a CosyVoice3-based text-to-whisper generator trained on a curated-and-synthesized whisper corpus. Its technical identity is defined by three linked elements: WhispReal as curated source data, WhispSynth as a pitch-free synthetic corpus, and CFM-only fine-tuning of CosyVoice3 as the final synthesis mechanism (Tan et al., 16 Mar 2026).