CarelessWhisper: Multi-Domain Whisper Analysis

• CarelessWhisper is a research label applied to multiple domains, including ASR hallucination detection, music transcription via source separation, causal streaming adaptation, and mobile privacy exploitation.
• It demonstrates methodological innovations such as adaptive layer attention, RMS-VAD segmentation, and blocked causal masking to improve robustness and reduce latency.
• The diverse studies reveal both enhanced performance under adverse conditions and exposure of hidden harms like hallucinated content and side-channel privacy vulnerabilities.

CarelessWhisper, also appearing as Careless Whisper, is not a single framework but a reused research label applied to multiple unrelated contributions. In the arXiv record represented here, the name denotes: a study of Whisper speech-to-text hallucination harms (Koenecke et al., 2024); a lyrics-transcription pipeline that combines music source separation with Whisper (Syed et al., 18 Jun 2025); a method for converting Whisper into a causal streaming model (Krichli et al., 17 Aug 2025); and a privacy attack that exploits silent delivery receipts in mobile instant messengers (Gegenhuber et al., 2024). A related Whisper-specific literature extends the hallucination branch with architectural mitigation and post-hoc calibration under noise (Tripathi et al., 18 Nov 2025, Huo et al., 8 Sep 2025).

1. Scope and disambiguation

The shared label spans at least four distinct technical domains: speech-to-text reliability analysis, automatic lyrics transcription, streaming ASR, and mobile-messenger privacy. The overlap is nominal rather than methodological.

Variant	Domain	Core focus
Careless Whisper	ASR evaluation	Whisper hallucination harms
CarelessWhisper	Music information retrieval	Lyrics transcription with source separation
CarelessWhisper	Streaming ASR	Causal, low-latency Whisper
Careless Whisper	Security and privacy	Silent delivery-receipt monitoring

A common misconception is that CarelessWhisper denotes a canonical Whisper modification. In the arXiv usage represented here, that is not the case. Two of the works are about Whisper model behavior in ASR (Koenecke et al., 2024, Krichli et al., 17 Aug 2025), one uses Whisper as the transcription backend for automatic lyrics transcription (Syed et al., 18 Jun 2025), and one is unrelated to ASR and instead targets WhatsApp and Signal receipt protocols (Gegenhuber et al., 2024).

2. Whisper hallucinations, harms, and later mitigation work

One usage of the label originates in a study of full-sentence hallucinations in Whisper transcriptions. That work defines a hallucination as generated text that is “nonsensical, or unfaithful to the provided source input,” operationalized as entire phrases or sentences that were never uttered in the audio. On AphasiaBank interview speech, using sentence-length utterances, the study reports $N = 13{,}140$ segments and $187$ segments yielding at least one hallucinated phrase in the April/May runs, with

$$H = \frac{\text{number of hallucinated utterances}}{\text{total number of utterances}} \simeq 1.4\%.$$

The same work reports that $38\%$ of hallucinations include explicit harms, partitioned into Perpetuation of Violence ($19\%$), Inaccurate Associations ($13\%$), and False Authority ($8\%$) (Koenecke et al., 2024).

The paper also reports disparities associated with aphasia and non-vocal duration. The corpus contains $7{,}805$ control segments and $5{,}335$ aphasia segments; hallucination rates are $1.7\%$ for aphasia speakers and $187$0 for controls, with $187$1. Non-vocal duration is defined as

$187$2

for segment $187$3. Mean $187$4 is $187$5 for aphasia speakers and $187$6 for controls, and hallucinated segments have $187$7 versus $187$8 for non-hallucinated segments. In the reported logistic regression, the coefficient for Share of Duration Non-Vocal is $187$9, alongside $$H = \frac{\text{number of hallucinated utterances}}{\text{total number of utterances}} \simeq 1.4\%.$$0 for Number of Words and $$H = \frac{\text{number of hallucinated utterances}}{\text{total number of utterances}} \simeq 1.4\%.$$1 for Has Aphasia (Koenecke et al., 2024).

Later Whisper-specific work addresses related failure modes under noisy acoustic conditions. One approach modifies the Whisper model itself through a two-stage architecture. Its Adaptive Layer Attention (ALA) stage computes inter-layer similarities

$$H = \frac{\text{number of hallucinated utterances}}{\text{total number of utterances}} \simeq 1.4\%.$$2

over the Whisper encoder’s $$H = \frac{\text{number of hallucinated utterances}}{\text{total number of utterances}} \simeq 1.4\%.$$3 layers, clusters layers into coherent blocks, mean-pools each block as

$$H = \frac{\text{number of hallucinated utterances}}{\text{total number of utterances}} \simeq 1.4\%.$$4

adds sinusoidal positional encodings, and uses learnable multi-head attention with final-layer queries to fuse block summaries before passing $$H = \frac{\text{number of hallucinated utterances}}{\text{total number of utterances}} \simeq 1.4\%.$$5 to the decoder. The second stage applies multi-objective knowledge distillation with encoder cosine loss, decoder cosine loss, MSE on decoder cross-attention maps, and token-level cross-entropy, combined as

$$H = \frac{\text{number of hallucinated utterances}}{\text{total number of utterances}} \simeq 1.4\%.$$6

with $$H = \frac{\text{number of hallucinated utterances}}{\text{total number of utterances}} \simeq 1.4\%.$$7 and $$H = \frac{\text{number of hallucinated utterances}}{\text{total number of utterances}} \simeq 1.4\%.$$8. On Hindi at $$H = \frac{\text{number of hallucinated utterances}}{\text{total number of utterances}} \simeq 1.4\%.$$9 dB / clean, the reported WER / SeMaScore progression is $38\%$0 from Baseline-2 to W-ALA to W-MOKD. On English average across noise levels, WER drops from $38\%$1 to $38\%$2 and then $38\%$3, while English clean WER changes from $38\%$4 to $38\%$5 and $38\%$6 (Tripathi et al., 18 Nov 2025).

A complementary line of work studies overconfidence in noisy Whisper decoding rather than hallucinated content per se. In additive noise below roughly $38\%$7 dB SNR, $38\%$8 of tokens are incorrectly predicted with confidence above $38\%$9. The paper defines token confidence as the maximum softmax probability over Whisper’s $19\%$0-token vocabulary and evaluates calibration with Expected Calibration Error (ECE) and Normalized Cross Entropy (NCE). Its post-hoc framework uses per-token features, a two-layer MLP overconfidence classifier trained with weighted BCE ($19\%$1), and selective temperature scaling

$19\%$2

applied only when $19\%$3. On R-SPIN in the $19\%$4 dB to $19\%$5 dB regime, baseline Whisper has ECE $19\%$6, NCE $19\%$7, and overconfident mass $19\%$8; the selective token-level calibrator yields ECE $19\%$9, NCE $13\%$0, and overconfident mass $13\%$1, with EER unchanged at $13\%$2 and WER unaffected (Huo et al., 8 Sep 2025).

3. CarelessWhisper in automatic lyrics transcription

In music information retrieval, CarelessWhisper denotes a lyrics-transcription system built around music source separation and Whisper. The system uses Hybrid Demucs (mdx and mdx_extra) to extract a vocal stem from full-mix audio. Hybrid Demucs is described as a U-Net–style time-domain separator augmented with spectral (STFT) branches and LSTM blocks, trained with a time-domain $13\%$3 loss plus a multi-resolution spectrogram $13\%$4 loss: $13\%$5 The reported pretrained checkpoints are mdx, trained on MUSDB18 train only with vocals SDR $13\%$6 dB, and mdx_extra, trained on MUSDB train+test+800 songs with SDR $13\%$7 dB (Syed et al., 18 Jun 2025).

The preprocessing pipeline retains a $13\%$8 kHz sampling rate, saves the separated vocals as single-track WAV, and uses them as Whisper inputs for both short-form and long-form transcription. For short-form evaluation, segments are $13\%$9 s. For VAD features, the system computes root-mean-square amplitude

$8\%$0

Whisper integration uses Whisper large-v2 via Faster-Whisper, beam size $8\%$1, language set to ground-truth, and $8\%$2 random seeds whose results are averaged (Syed et al., 18 Jun 2025).

For short-form transcription, the method first merges overlapping lyric lines into non-overlapping “merged-lines,” merging transcripts when overlap is $8\%$3 s and discarding rare segments $8\%$4 s. It then groups merged-lines by splitting whenever the gap exceeds $8\%$5 s, and further partitions each group into sub-groups of $8\%$6 s while maximizing minimum segment duration. These “group” segments have mean duration $8\%$7 s, compared with $8\%$8 s for single lines. The paper states that lines alone yield $8\%$9 absolute worse WER (Syed et al., 18 Jun 2025).

For long-form transcription, CarelessWhisper replaces Whisper’s native timestamp-based segmentation with RMS-VAD plus the “Cut-Merge” algorithm from WhisperX. The thresholds are onset $7{,}805$0, offset $7{,}805$1, $7{,}805$2 s, and $7{,}805$3 s. Adjacent regions separated by less than $7{,}805$4 are merged, and any region longer than $7{,}805$5 s is split at the local VAD minimum closest to the $7{,}805$6 s mark. Prompt conditioning is disabled in long-form mode to allow batching (Syed et al., 18 Jun 2025).

Evaluation uses Jam-ALT ($7{,}805$7 songs, $7{,}805$8 min, backing $7{,}805$9, non-lexical vocables $5{,}335$0) and MUSDB-ALT ($5{,}335$1 songs, $5{,}335$2 min, backing $5{,}335$3, non-lexical $5{,}335$4) in a zero-shot setting. WER is defined as

$5{,}335$5

In short-form transcription on MUSDB-ALT, WER is $5{,}335$6 for mix, $5{,}335$7 for sep-mdx, $5{,}335$8 for sep-mdx_extra, and $5{,}335$9 for vocal stem. In long-form transcription on Jam-ALT, mix/native yields $1.7\%$0, while mix/RMS-VAD yields $1.7\%$1; on MUSDB-ALT, vocal stem/native yields $1.7\%$2 and vocal stem/RMS-VAD yields $1.7\%$3. The paper states that CarelessWhisper (sep-mdx_extra + RMS-VAD on mix) sets a new open-source SOTA on Jam-ALT long form with WER $1.7\%$4, compared with vanilla Whisper at $1.7\%$5 (Syed et al., 18 Jun 2025).

The implementation notes emphasize that deletion rates on backing vocals and non-lexical vocables remain $1.7\%$6 and are an open research problem. This identifies a residual error mode even when source separation and improved segmentation reduce overall WER (Syed et al., 18 Jun 2025).

4. CarelessWhisper as a causal streaming Whisper model

In streaming ASR, CarelessWhisper is a method for turning Whisper into a causal, low-latency model. The central problem is that Whisper is an encoder-decoder Transformer trained offline on fixed-length inputs with $1.7\%$7 frames, and its encoder uses full self-attention. Consequently, when only the first $1.7\%$8 frames are available,

$1.7\%$9

need not match the prefix of the offline representation $187$00, since

$187$01

The paper formalizes this as a barrier to direct low-latency conversion (Krichli et al., 17 Aug 2025).

The proposed solution replaces encoder self-attention with blocked causal attention. With chunk size $187$02 and initial priming chunk $187$03, the mask is

$187$04

The masked self-attention is

$187$05

Theorem 2 is reported to show that for all $187$06, chunked encoding reproduces the offline prefix exactly: $187$07 With cached $187$08 matrices, the encoder complexity becomes $187$09 and memory $187$10 (Krichli et al., 17 Aug 2025).

Adaptation to the causal regime is performed by LoRA fine-tuning. For each projection matrix $187$11,

$187$12

with only $187$13 trained. Reported ranks are $187$14 for Whisper-base/small and $187$15 for Whisper large-v2. Training uses weakly aligned data with sampled chunk-boundary time points $187$16, and minimizes cross-entropy on streaming predictions conditioned on the audio prefix and tokens whose end time is within that prefix (Krichli et al., 17 Aug 2025).

Streaming inference combines the causal encoder with a decoder that can revise unstable tokens. In greedy decoding, a token $187$17 at chunk $187$18 is stable if either

$187$19

or

$187$20

The algorithm checks the last $187$21 tokens, backtracks to the first unstable position, and resumes decoding; once a token remains stable across two chunks it is considered final. Theorem 3 states that the resulting path probability satisfies

$187$22

Beam search generalizes this by requiring beam stability within $187$23, with the reported practical setting $187$24. To avoid hallucinations, if any beam predicts EOT, decoding pauses until a new chunk arrives (Krichli et al., 17 Aug 2025).

On LibriSpeech test-clean/test-other, with chunk sizes $187$25 ms and $187$26, the paper reports that CarelessWhisper outperforms or matches Simul-Whisper and Ufal-Whisper in most settings. One example on large-v2 at $187$27 ms gives test-clean WER $187$28 for CarelessWhisper versus $187$29 for Simul-Whisper, while on test-other at $187$30 ms the reported WERs are $187$31 versus $187$32. Offline Whisper large-v2 is listed at $187$33 on clean and $187$34 on other (Krichli et al., 17 Aug 2025).

Latency and runtime are central results. For large-v2 with $187$35 ms, average latency is reported as $187$36 s for Ufal-Whisper (beam $187$37), $187$38 s for Simul-Whisper (greedy), $187$39 s for CarelessWhisper (greedy), and $187$40 s for CarelessWhisper (beam $187$41). The paper further states that CarelessWhisper is $187$42 faster in RTF than baselines despite using unoptimized OpenAI Whisper code (Krichli et al., 17 Aug 2025).

The same fine-tuning process is also used for word-level timestamps. On TIMIT test, at a $187$43 ms threshold, CarelessWhisper large-v2 with $187$44 ms chunks yields $187$45 precision, $187$46 recall, Start Difference $187$47 ms, and End Difference $187$48 ms. At the stricter $187$49 ms threshold, the paper reports $187$50 precision, compared with Canary’s $187$51 (Krichli et al., 17 Aug 2025).

5. Careless Whisper as a privacy attack on mobile instant messengers

In security and privacy research, Careless Whisper is an attack that exploits delivery receipts in E2EE messengers. The protocol description distinguishes server-ack, device-ack, and read-ack. In a single-device case, the sender receives confirmation that the encrypted packet reached the service front-end, the recipient device confirms successful receipt and decryption, and the server forwards that device-ack back to the sender. In a multi-device client-fanout setup, delivery and device-ack steps are repeated independently for each recipient device, so the sender observes one device-ack per device. The paper states that delivery receipts cannot be disabled by the end user in WhatsApp or Signal (Gegenhuber et al., 2024).

The attack uses specially crafted messages that trigger device acknowledgments without any user-visible notification in many cases. The exploited packet types are reactions, message edits, and message deletions. The paper gives a simplified packet structure

$187$52

and states that even when the referenced message ID is invalid, the client still returns an ack_device (Gegenhuber et al., 2024).

For high-frequency profiling, the attacker sends a stream of stealth pings at rate $187$53 and timestamps the returned acknowledgments. The success probability within deadline $187$54 is modeled as

$187$55

On WhatsApp, the authors report no server-side rate limiting up to $187$56 Hz, whereas on Signal burst rates above $187$57 Hz caused server-side queuing and the experiments therefore used $187$58 Hz. For WhatsApp at $187$59 s and $187$60 Hz, the paper reports $187$61 (Gegenhuber et al., 2024).

The side channel supports several inferences. Online/offline status can be tracked with second-level granularity. Device count is inferred directly from the number $187$62 of independent device acknowledgments per ping. OS and implementation fingerprinting can be derived from whether late receipts are stacked or separate, and from their ordering; the paper states that WhatsApp on iOS reverses stacked receipts, whereas Android does not stack at all. Screen and application state can be inferred from RTTs: screen off yields $187$63 s, screen on with app in background yields $187$64 s, and app in foreground yields $187$65 s. Using thresholds $187$66 s and $187$67 s in the average-RTT classifier gives $187$68 correct classification in the reported tests (Gegenhuber et al., 2024).

The paper also presents resource-exhaustion attacks. If each reaction carries payload $187$69, victim download rate is

$187$70

With $187$71 MB and $187$72 Hz, the reported rate is $187$73 MB/s, or $187$74 GB/h. For battery drain, the paper writes

$187$75

and reports additional battery draw of $187$76 per hour under stealth-ping bombardment, versus $187$77/h idle (Gegenhuber et al., 2024).

The proposed countermeasures are client-side validation of reactions/edits/deletions against causally available message IDs, receipt rate-limiting or random delay, optional receipt disabling, server-side per-recipient packet limits and abuse detection, and batching across devices so that only one acknowledgment is sent. The paper explicitly frames the trade-offs as latency versus privacy, usability versus security, and implementation complexity (Gegenhuber et al., 2024).

6. Comparative significance and recurring themes

Across the Whisper-centered uses of the term, the dominant problem is model behavior under adverse or incomplete context. The adverse conditions differ: long silences and disfluencies in interview speech (Koenecke et al., 2024), additive noise in multilingual ASR (Tripathi et al., 18 Nov 2025), severe low-SNR decoding with misleading confidences (Huo et al., 8 Sep 2025), musical accompaniment and segmentation error in lyrics transcription (Syed et al., 18 Jun 2025), and absence of future context in streaming transcription (Krichli et al., 17 Aug 2025). Each work introduces a different control mechanism: auditing hallucination harms and subgroup disparities, adaptive layer fusion and teacher-student alignment, selective temperature scaling, source separation plus RMS-VAD segmentation, or blocked causal masking with backtracking.

The privacy paper is technically unrelated to Whisper, but it exhibits a parallel structural motif: reliable-looking system signals conceal failure modes that are not apparent to end users. In the messenger case, silent delivery receipts expose activity, device, and resource signals (Gegenhuber et al., 2024); in the ASR cases, aggregate transcription success can coexist with hallucinated phrases, overconfident token errors, or instability under chunked inference (Koenecke et al., 2024, Huo et al., 8 Sep 2025, Krichli et al., 17 Aug 2025). This suggests that the reused label functions less as the name of a stable software artifact than as a marker for hidden brittleness under conditions that standard interfaces obscure.

Taken together, the works associated with CarelessWhisper illustrate three distinct research patterns. First, reliability analysis can reveal harms not captured by conventional accuracy summaries, as in hallucination content analysis and calibration metrics. Second, targeted architectural or pipeline changes can substantially alter failure behavior without requiring a completely new backbone, as shown by ALA+KD, RMS-VAD segmentation, and LoRA-based causal conversion. Third, apparently auxiliary protocol features such as receipts can themselves become attack surfaces. The term therefore indexes a heterogeneous but methodologically serious body of work on failure detection, robustness intervention, and systems-level side channels.