Reducing Hallucinated Transcripts in Whisper via Hallucination Space Projection

Abstract: Whisper is a widely used foundation model for automatic speech recognition (ASR), but its generative decoder can produce fluent hallucinated transcripts for inputs containing little or no speech. We propose a training-free, inference-time method to reduce these hallucinations using low-rank projection of decoder activations. A compact hallucination-associated subspace is estimated from non-speech calibration data, and decoder hidden states are projected away from this subspace during inference. We evaluate two variants: always-on, which applies projection to all inputs, and gated, which applies it only when Whisper predicts that an input is likely non-speech. Across non-speech benchmarks, always-on projection reduces average hallucination rate (HR) from 31.31% to 2.44%, a 92.21% relative reduction, while gated projection reduces HR to 3.74%, an 88.05% relative reduction, with lower false rejection of genuine speech. On LibriSpeech, gated projection increases absolute word error rate (WER) by 0.33-4.39 percentage points and yields false-rejection rates (FRR) of 0.41--9.97% across model and split settings. These results show that low-rank activation projection can substantially suppress Whisper hallucinations without retraining, while providing a controllable trade-off between hallucination suppression and speech recognition performance.

• The paper implements Hallucination Space Projection (HSP), a training-free method for reducing hallucinated transcripts in Whisper by modifying decoder activations during inference.
• The effectiveness of HSP is demonstrated by significant reductions in hallucination rates (HR) across non-speech data while preserving accurate speech recognition as measured by Word Error Rate (WER) and False Rejection Rate (FRR).
• The gated projection mode, which uses Whisper’s own no-speech probability, emerges as the most practical approach, achieving a strong trade-off between speech preservation and hallucination suppression, especially in models of different sizes, known as scales,.

Problem formulation and contribution

Whisper’s generative decoder can emit fluent text for audio containing silence, environmental sounds, music, or other non-speech content. In this setting, the correct transcript is empty, so any non-empty output constitutes a hallucinated transcription. The problem is operationally significant because the generated text may be linguistically plausible and therefore pass downstream validation. The paper addresses this failure mode with Hallucination Space Projection (HSP), a training-free inference-time intervention that modifies Whisper’s decoder activations rather than its parameters, decoding objective, input segmentation, or output text.

The central hypothesis is that hallucinating and correctly rejected non-speech examples occupy distinguishable regions of decoder representation space. Using non-speech calibration data, the method constructs activation differences between examples that produce hallucinated transcripts and examples that correctly produce empty outputs. Singular value decomposition (SVD) extracts a low-rank basis for the dominant difference directions. During inference, the component of a decoder hidden state aligned with this basis is removed. The approach therefore treats hallucination suppression as a representation-editing problem rather than as phrase filtering or threshold calibration.

Two operating modes are evaluated. Always-on projection intervenes on every input, whereas gated projection first uses Whisper’s unmodified no-speech probability and applies the intervention only when that probability exceeds a gate threshold $\gamma$. The final rejection decision still uses Whisper’s no-speech threshold $\tau$, fixed at $0.6$ in the principal experiments. The paper’s main result is that always-on projection can attain very low hallucination rates at substantial speech-recognition cost, while gated projection yields a more usable HR–WER–FRR trade-off (2609.04561).

Method

Let $h_\ell$ denote the hidden state at decoder layer $\ell$. The calibration procedure partitions non-speech examples into hallucinating and correctly empty groups. Pairwise differences between the two groups form a matrix $\Delta_\ell$. The right singular vectors associated with its largest singular values define the basis $B_{\ell,r}$, where $r$ is the retained rank. Because both groups contain non-speech audio, the differencing operation is intended to suppress generic acoustic variation and retain directions associated with hallucination-prone decoder behavior.

At inference time, HSP applies

$$\tilde{h}_\ell = h_\ell-\alpha(h_\ell B_{\ell,r}^{\top})B_{\ell,r},$$

where $\alpha$ controls intervention strength. With an orthonormal basis, the second term is the component of $\tau$0 lying in the estimated hallucination subspace. For $\tau$1, this is an ordinary orthogonal projection away from the subspace; values below or above one produce weaker or stronger suppression.

The gated variant requires two passes for inputs satisfying the gate condition. An initial unmodified pass produces $\tau$2. If this value exceeds $\tau$3, the model is decoded again with the projection hook active. The resulting no-speech probability is compared with $\tau$4 to determine whether the transcript is discarded. This distinction between $\tau$5 and $\tau$6 is important: $\tau$7 controls whether the hidden-state intervention occurs, whereas $\tau$8 controls the final empty-transcript decision. Gating thus uses Whisper’s own uncertainty estimate to limit representation perturbation on likely speech.

The method is evaluated with Whisper small, medium, and large-v3. Calibration and parameter selection use ESC-50 folds 1–3; held-out ESC-50 folds 4–5, UrbanSound8K, and a filtered FSD50K subset test cross-dataset behavior. LibriSpeech validation-clean is used for parameter selection and LibriSpeech test-clean and test-other measure WER and false rejection of speech. The study consequently evaluates both sides of the intervention: empty-output reliability on non-speech and preservation of normal ASR on speech.

Calibration and operating-point selection

The offline sweeps show that layer selection is more consequential than rank selection. Early decoder layers produce limited hallucination suppression, while middle-to-late layers produce much larger reductions. Increasing rank exhibits saturation and can remove speech-relevant information without materially improving non-speech rejection. For large-v3, the selected configuration is $\tau$9, $0.6$0, followed by $0.6$1 and $0.6$2. The corresponding development-set HR falls from $0.6$3 to $0.6$4 under gated projection. On LibriSpeech validation-clean, always-on projection reaches 11.59% WER and 9.69% FRR, whereas gated projection reaches 5.32% WER and 1.85% FRR. This establishes the paper’s principal design rationale: gating is not merely a computational refinement, but a mechanism for limiting speech degradation.

Figure 1: Offline selection of decoder layer $0.6$5 and projection rank $0.6$6 for Whisper large-v3; lower HR and WER are preferred.

The selected settings for small and medium are $0.6$7, $0.6$8, $0.6$9, $h_\ell$0 and $h_\ell$1, $h_\ell$2, $h_\ell$3, $h_\ell$4, respectively. For small, the non-gated configuration reduces development HR from 20.2% to 1.08% while increasing WER from 4.28% to 5.07%; gating yields 1.2% HR and 4.68% WER. For medium, the selected non-gated setting reduces HR from 22.8% to 2.0% with 5.13% WER, while the gated setting yields 2.8% HR and 3.90% WER. These results support the claim that a very low-dimensional intervention can capture a substantial portion of the relevant decoder behavior.

Figure 2: Development-set sweep of projection strength $h_\ell$5 and gate threshold $h_\ell$6 for large-v3 after fixing $h_\ell$7 and $h_\ell$8.

The results also indicate that the selected parameters are model-specific. Decoder depth, hidden dimensionality, and the location of hallucination-related representations differ across Whisper scales, so a single universal layer and rank are not established. The calibration procedure is lightweight, but it still requires model-specific development data and a speech validation set to select an acceptable operating point.

Hallucination suppression and cross-dataset transfer

The strongest evidence for HSP is its performance on held-out non-speech data. The reported baseline hallucination rates are high and heterogeneous: for large-v3, 44.25% on held-out ESC-50, 76.08% on UrbanSound8K, and 21.35% on FSD50K. Always-on projection reduces these values to 1.50%, 0.87%, and 0.18%, respectively. Gated projection produces 8.38%, 2.74%, and 1.15%. Thus, gating sacrifices some rejection performance relative to always-on intervention, but remains substantially better than the unmodified model on all three datasets.

The cross-dataset result is particularly relevant to the proposed representation-level interpretation. The basis is estimated using ESC-50 folds 1–3, yet the intervention transfers to urban sounds and the filtered FSD50K collection. This behavior is difficult to explain as simple phrase memorization: the method does not inspect the generated text and does not require a dataset-specific list of hallucinated phrases. It is consistent with the existence of decoder-level states associated with unsupported generation across diverse acoustic conditions. However, the evidence is still limited to three non-speech benchmarks and one calibration source; it does not establish invariance under arbitrary domain shift.

The paper reports an average reduction from 31.31% to 2.44% for always-on projection and to 3.74% for gated projection in the abstract. The detailed dataset-specific results support the qualitative conclusion, but the manuscript contains inconsistencies in model labels and aggregate values: one displayed result table assigns the 44.25%, 76.08%, and 21.35% baseline values to large-v3, while the corresponding row labels elsewhere appear inconsistent with the small-model results. The headline averages should therefore be interpreted alongside the per-dataset values rather than as independently verifiable summary statistics.

Figure 3: Gated projection shifts no-speech probabilities upward on non-speech audio while leaving the distribution for LibriSpeech speech concentrated near low values.

The no-speech probability analysis provides a mechanism-level account of the aggregate results. Projection moves many ESC-50 examples above $h_\ell$9, converting latent hallucination-prone cases into rejected segments. On LibriSpeech test-clean, the distribution remains concentrated near zero, which explains why gated projection substantially reduces—but does not eliminate—speech false rejection. The result implies that HSP does not simply suppress all decoder activity; it preferentially changes the no-speech decision boundary for cases already judged ambiguous by Whisper.

Speech preservation and the HR–WER–FRR trade-off

HSP’s principal cost is degradation on genuine speech. For large-v3, baseline WER is 4.06% on test-clean and 5.87% on test-other. Gated projection increases these values to 6.17% and 6.57%, while always-on projection increases them to 12.95% and 13.13%. The corresponding FRR values are 2.86% and 1.40% for gated projection, compared with 10.50% and 11.47% for always-on projection. Gating therefore removes most of the extreme degradation caused by universal intervention, but it does not preserve baseline ASR performance.

Across model sizes, the same pattern holds:

Model	Method	Clean WER	Other WER	Clean FRR	Other FRR
Small	Original	4.04	8.38	0.00	0.00
Small	Gated	4.37	9.13	0.41	2.58
Medium	Original	3.66	7.29	0.03	0.27
Medium	Gated	5.47	11.68	4.07	9.97
Large-v3	Original	4.06	5.87	0.04	0.00
Large-v3	Gated	6.17	6.57	2.86	1.40

The medium model exhibits the most severe gated speech degradation, particularly on test-other, where FRR reaches 9.97%. This demonstrates that the trade-off is not determined solely by the nominal intervention strength or rank. It depends on the interaction between decoder scale, layer location, calibration geometry, no-speech estimates, and acoustic difficulty.

The comparison with threshold-only tuning is unfavorable to threshold adjustment as a standalone solution. For original Whisper, changing $\ell$0 from 0.6 to 0.5 reduces average HR only from 47.23% to 45.99%, with little change in WER or FRR. By contrast, projection changes decoder representations before the no-speech decision and produces much larger reductions. The implication is that the failure is not adequately addressed by selecting a more aggressive decision threshold when the underlying decoder state continues to favor fluent text generation.

Projection also compares favorably with the paper’s phrase-based baseline. Bag-of-Hallucinations filtering reduces HR only modestly because it can remove phrases observed during calibration but cannot address novel generations. WhisperX VAD achieves an average HR of 5.24% across the non-speech benchmarks, while gated projection reaches 4.09% without an external VAD. This comparison should be qualified: WhisperX reports lower LibriSpeech WER and FRR in the cited evaluation, so gated projection is not uniformly superior. Its advantage is the absence of a separately deployed segmentation model and the ability to intervene inside Whisper’s decoder.

Ablations and stronger interventions

The threshold ablation confirms that HSP exposes a controllable operating curve rather than a single universally optimal point. For medium, lowering $\ell$1 from 0.6 to 0.4 reduces average HR from 5.09% to 2.89%, but increases average WER from 8.58% to 11.49% and FRR from 7.03% to 10.91%. Raising $\ell$2 to 0.7 lowers WER to 7.28% and FRR to 4.76%, but raises HR to 7.85%. The default $\ell$3 is selected as a compromise, not because it dominates all alternatives.

The multi-layer ablation demonstrates that stronger suppression is technically possible. For medium, applying gated projections at layers 20 and 24 reduces HR from 2.75% to 0.13% on ESC-50, from 0.66% to 0.07% on UrbanSound8K, and from 8.78% to 4.87% on FSD50K. The cost is substantial: test-clean WER increases from 5.48% to 10.62%, test-other WER from 11.68% to 15.71%, and FRR increases from 4.07% to 11.07% and from 9.97% to 13.75%. This result suggests that hallucination-associated directions are distributed across layers, but it also supports the paper’s decision to use a single-layer intervention for the main configuration.

Figure 4: Offline layer–rank selection for Whisper small, showing the dependence of HR and WER on intervention depth and projection rank.

Figure 5: Projection-strength and gate-threshold sweep for Whisper small after fixing $\ell$4 and $\ell$5.

Figure 6: Offline layer–rank selection for Whisper medium, showing lower HR at middle-to-late decoder layers with corresponding speech-recognition costs.

Figure 7: Projection-strength and gate-threshold sweep for Whisper medium after fixing $\ell$6 and $\ell$7.

These ablations establish an important negative result: reducing hallucinations more aggressively is straightforward, but doing so can undermine the speech-recognition function that Whisper’s decoder simultaneously performs. The proposed contribution is therefore not maximal suppression in isolation; it is a parameterized intervention that makes the trade-off explicit and adjustable.

Limitations and open questions

The evaluation is restricted to hallucinations caused by non-speech audio. It does not establish effectiveness for long-form transcription, multilingual speech, code-switching, acoustically ambiguous speech, weak vocal activity, or hallucinations arising from autoregressive context accumulation. The method also depends on Whisper’s internal no-speech probability for gating. If that score is poorly calibrated or systematically low on a class of hallucination-prone inputs, gated projection may not activate when needed.

The calibration basis is estimated from ESC-50 and transferred to other datasets, but broader calibration-to-deployment shifts remain untested. The use of ESC-50 folds 1–3 for both subspace construction and parameter selection also creates a dependence between calibration and development evaluation, even though the final non-speech tests use held-out folds and other datasets. A more extensive study would need multiple calibration sources, confidence intervals across random splits, and evaluation on realistic continuous recordings rather than predominantly segmented benchmark clips.

The paper does not provide a detailed computational analysis of the additional inference cost of the gated two-pass procedure. In deployment, the second decoding pass may be acceptable for batch processing but consequential for latency-sensitive systems. The proposed strategy of retaining and flagging uncertain segments is operationally sensible, but its effectiveness depends on downstream storage, review, and retranscription policies that are not evaluated.

Finally, several reported aggregate statistics and model labels are internally inconsistent in the supplied manuscript. This does not invalidate the qualitative result or the clearly reported per-dataset trends, but it limits confidence in the headline averages until the experimental tables and aggregation procedures are reconciled.

Conclusion

The paper presents a decoder-level, training-free method for suppressing Whisper’s hallucinated transcripts on non-speech audio. A low-rank subspace estimated from hallucinating versus correctly empty calibration examples is removed from selected decoder activations, and a no-speech-conditioned gate limits the intervention on likely speech. The method reduces hallucination rates dramatically across ESC-50, UrbanSound8K, and FSD50K, with always-on projection achieving the strongest suppression and gated projection offering a more practical HR–WER–FRR balance.

The results support the narrower claim that non-speech hallucinations have exploitable decoder-level structure that can be modified without fine-tuning, external VAD, or phrase lists. They do not show that all ASR hallucinations share this structure. The principal open question is whether the same representation-editing framework remains reliable under broader acoustic, linguistic, and deployment shifts while preserving speech recognition at acceptable WER and FRR.