Compression-Step Hallucination

• Compression-step hallucination is a phenomenon where lossy compression causes neural systems to generate high-confidence yet unsupported outputs due to limited memory capacity.
• Empirical studies reveal that diverse modalities—including language, image, and speech—exhibit increased hallucination rates as compression intensifies.
• Mitigation strategies such as alignment-aware training and conditional control effectively reduce spurious outputs while preserving core model performance.

Compression-Step Hallucination refers to the emergence or amplification of hallucinations—confidently asserted but spurious, unsupported, or outright incorrect outputs—caused by aggressive or information-losing compression processes within neural systems. This phenomenon is pervasive across modalities, appearing in neural speech coding, deep generative models for language or vision, and compressed chain-of-thought (CoT) reasoning. As compression limits or degrades the signal available to downstream decoders or inference modules, these systems substitute plausible but fabricated content drawn from learnt priors in place of unattainable or ambiguous details. Compression-step hallucination is thus a direct, information-theoretic consequence of finite model or codebook capacity, and its characteristics and mitigation are domain- and architecture-dependent.

1. Theoretical Foundations: Rate-Distortion, Information Bottlenecks, and Hallucination

The information-theoretic roots of compression-step hallucination are formalized via the rate-distortion tradeoff for membership testing in LLMs (Guo et al., 31 Jan 2026). In this framework, a model must compress a large set of arbitrary facts $K$ within a finite memory state $W$, balancing false negatives on true facts and false positives (hallucinations) on non-facts. The minimal per-fact capacity required to achieve expected error rates $(\epsilon_K, \epsilon_N)$ is given in the sparse limit by the KL-divergence-based rate-distortion function

$$\frac{C(M)}{n}\geq \min_{(\mu_K,\mu_N)} \mathrm{KL}(\mu_K\|\mu_N),$$

with $\mu_K$ and $\mu_N$ the distributions of model confidence scores on facts and non-facts, respectively. When memory is insufficient, the optimal compression “overloads” a fraction $q^*$ of non-facts with high-confidence scores identical to facts—the “hallucination channel.” No post-hoc thresholding can separate facts from hallucinated non-facts: both produce identical high-confidence scores.

This lower bound is not merely a theoretical curiosity. Synthetic experiments demonstrate that even with perfect training and data, the hallucination probability $q^*\sim 2^{-C}$ persists as a direct consequence of the compression bottleneck (Guo et al., 31 Jan 2026). This effect generalizes to any neural reasoning system where random or patternless facts must compete for scarce parametric capacity, including LLM factuality, ASR phoneme recognition, and multimodal models.

In the context of intermediate-chain compression (e.g., CoT, visual token chains), the Information Bottleneck principle (Fang et al., 3 Feb 2026) characterizes the trade-off between salience (signal) and redundancy (noise) in the compressed representation $Z$. Pruning irrelevant reasoning steps can reduce hallucinations if uninformative tokens dominate, but overcompression may discard verification steps critical for resisting hallucination (Zeng et al., 5 Apr 2026).

2. Manifestations by Domain: Language, Vision, Speech, and Multimodal

Compression-step hallucination takes distinct operational forms in different neural architectures:

• LLMs: Space-optimal LLMs hallucinate random facts with high confidence on out-of-distribution or memory-constrained queries, even in “closed world” settings. The phenomenon is especially acute for random or patternless facts, where parameter sharing is unavailable, and defaulting to non-facts is not capacity-optimal. Scaling, regularization, and training loss weighting all modulate the $q^*$ hallucination rate (Guo et al., 31 Jan 2026).
• Chain-of-Thought (CoT) Reasoning: Aggressive compression of reasoning traces—by distillation, token skipping, or RL-driven depth minimization—removes self-checks and verification steps, measurably increasing hallucination rates, as quantified by truthfulness metrics on unanswerable questions (Zeng et al., 5 Apr 2026). However, on strong base models or with alignment-aware compression, hallucination can be preserved or even reduced for a fixed chain length.
• Image Compression: Learned lossy compressors must either produce blurry, exact reconstructions or inject in-distribution details via perceptual or GAN losses. In high-entropy regions where the code cannot recover all original structure, the decoder hallucinates plausible but non-faithful textures (grass blades, fabric details) (Aczel et al., 2024). The net effect is content-sensitive: hallucination may be imperceptible or even preferred in textures, yet highly undesirable for semantic content (text, faces).
• Speech and ASR (Phoneme Hallucination): At ultra-low bitrates, the quantizer codebook cannot preserve all semantic/phonetic features, forcing the generative decoder (e.g., HiFi-GAN) to synthesize a plausible but possibly incorrect phoneme sequence (PH). Such hallucinations occur when the decoder’s conditional entropy $W$0 grows beyond recoverable bounds and it reverts to an unconditional mode (Yi et al., 5 Feb 2026). In deep ASR (e.g., Whisper models), spectral phase transitions in self-attention dynamics lead to “rank-1 collapse” attractors, projecting away genuine acoustic evidence in favor of internally coherent but input-decoupled transcripts (Viakhirev et al., 31 Mar 2026).
• Multimodal Reasoning Models (MLRMs): Long or verbose reasoning chains in MLRMs dilute visual grounding, increasing reliance on language priors and producing spurious “object hallucinations.” Selective compression of low-score tokens and contrastive preference optimization can reduce hallucinations without sacrificing answer accuracy (Fang et al., 3 Feb 2026).

3. Empirical Characterization and Measurement

Quantitative analysis across modalities employs specialized hallucination metrics:

• LLMs and Reasoning: Truthfulness rate on unanswerable samples (FaithEval), refusal rates (HarmBench), and hallucination rate $W$1 measure factual steadfastness under compression (Zeng et al., 5 Apr 2026).
• Speech Codec Evaluation: Semantic deviation is measured via WER (Word Error Rate, using Whisper-large-v3 and wav2vec2.0 oracles), semantic MOS (subjective 7-point scale for transcript match), and objective acoustic metrics (PESQ, WARPQ) (Yi et al., 5 Feb 2026). PH rates increase sharply below 0.4 kbps, but are significantly reduced by LM-driven losses.
• Image Compression: Perceptual metrics (LPIPS, FID), task-specific preference scores (Elo via user studies), and content-driven fidelity/realism judgments distinguish harmless (texture-proof) from semantically dangerous hallucination (Aczel et al., 2024).
• ASR/Whisper Models: Spectral diagnostics involve measurement of the effective rank $W$2, singular value slopes ($W$3 via eigenspectrum tails), and phase diagram mapping between dispersive and attractor regimes (Viakhirev et al., 31 Mar 2026).
• Multimodal Models: Object hallucination is tracked by CHAIR, POPE (object-existence interrogatives), AMBER (attribute/relation existence), GPT-4–assisted sentence hallucination ratio, and, where available, standard multimodal MC tasks (Fang et al., 3 Feb 2026).

4. Mechanistic Explanations: Phase Transitions, Spectral Collapse, and Information Overload

Distinct but convergent mechanisms underlie compression-step hallucination:

• Spectral Phase Transition (ASR/Deep Transformers): As depth and scale increase, the product of layer-wise gains and alignments in Transformer stacks traverses a threshold, entering a “compression-seeking attractor” regime. Here, the Jacobian of output with respect to acoustic context collapses to rank-1, and only perturbations aligned with a single dominant direction propagate. All orthogonal acoustic evidence is suppressed, and the model regresses to its autoregressive prior, hallucinating fluent but ungrounded outputs (Viakhirev et al., 31 Mar 2026).
• Rate-Distortion Channel Overload (LLMs/Facts): Insufficient capacity induces a bottleneck where, to minimize overall loss, the model maps a fractional mass $W$4 of non-facts to the same high-confidence output as true facts. This channelization is provably capacity-optimal, and increasing compression severity ($W$5) inevitably increases $W$6 (Guo et al., 31 Jan 2026).
• Decoder Prior Hallucination (Generative Speech/Image): When compressed latent codes are inadequate, decoders (e.g., HiFi-GAN, image synthesis blocks) sample from a learned prior, visually or acoustically, hallucinating plausible, in-distribution details unrelated to the original. This is exacerbated when the latent space is strongly coupled to perceptual or adversarial loss terms (Aczel et al., 2024, Yi et al., 5 Feb 2026).
• CoT/Reasoning Chain Verification Loss: In chain-of-thought pruning, removing self-check or verification steps mechanically weakens safeguards against unsupported inference, leading to a monotonic increase in hallucination rates with compression (Zeng et al., 5 Apr 2026, Fang et al., 3 Feb 2026).

5. Mitigation Strategies and Systematic Approaches

Several mitigation and control strategies, tailored to domain, have demonstrated efficacy:

• LLM-Driven Losses (Speech Coding): End-to-end LM losses that penalize mismatches in textual or semantic content—either via ASR-based objectives (e.g., using Whisper to induce token-aligned penalties) or by aligning high-dimensional representations between decoded utterances and ground-truth transcripts (Timed-Text Regularizer)—significantly lower phoneme hallucination rates without hurting acoustic quality (Yi et al., 5 Feb 2026).
• Conditional Hallucination Control (Image Compression): Training a lightweight input-dependent predictor to modulate the perceptual (GAN) loss term enables the codec to automatically restrict hallucination in semantically sensitive regions (text, faces) while permitting it in perceptual textures where it is harmless or preferable. The Conditionally Hallucinating Compressor (ConHa) exemplifies this content-adaptive approach (Aczel et al., 2024).
• Alignment-Aware Compression (CoT, Reasoning): Incorporating direct preference optimization (DPO) that prefers shorter, yet accurate, chains, or learning from AI-generated corrections to hallucinated samples, verifies that compression step effectiveness depends on both base model strength and alignment between surviving steps and truthfulness signals. Alignment-aware variants minimize hallucination degradation even as chain length drops by up to 19.3% (Zeng et al., 5 Apr 2026, Fang et al., 3 Feb 2026).
• Contrastive Preference Optimization (Multimodal Reasoning): Combining chain pruning with contrastive training using high-quality “corrected” traces as positive samples and visually or instruction-induced hallucinations as negatives (C3PO) robustly suppresses hallucination in MLRMs (Fang et al., 3 Feb 2026).
• Capacity Augmentation and Retrieval: Increasing model memory (e.g., parameter count, regularization relaxation, targeted fact fine-tuning) or supplementing with external retrieval (removing parametric compression bottlenecks) lowers $W$7 and mitigates space-optimal hallucination (Guo et al., 31 Jan 2026).

6. Practical Design Considerations and Cross-Domain Lessons

Empirical studies indicate several guiding principles:

• Compression and hallucination trade off as a rigorously predictable function of system capacity and task structure. Hallucination is not a bug but a mathematically unavoidable outcome of lossy compression on random or unstructured facts and inputs (Guo et al., 31 Jan 2026).
• Reporting only accuracy, token savings, or distortion is insufficient. Comprehensive metrics that directly capture hallucination rates, especially in unanswerable or adversarial settings, are necessary for evaluating compressed models (Zeng et al., 5 Apr 2026).
• Domain-specific content sensitivity and user preference are critical. In image and speech codecs, humans may prefer hallucinated details in some contexts (textures) but not in others; mechanisms must allow for conditional or input-adaptive control (Aczel et al., 2024).
• Early detection of “phase transitions” or internal collapse (via rank, alignment diagnostics) is a key safety countermeasure in deep speech and ASR systems (Viakhirev et al., 31 Mar 2026).
• Multimodal and reasoning systems benefit from compression-driven bottlenecks when redundant or non-informative content predominates. Systematic pruning coupled with high-quality, contrastive feedback can substantially suppress hallucination without compromising core task accuracy (Fang et al., 3 Feb 2026).
• Regular assessment on specific hallucination benchmarks, ablation of compression rates, and cross-task efficiency scoring are recommended best practices (Zeng et al., 5 Apr 2026).

A plausible implication is that compression-step hallucination represents a fundamental constraint on the design of efficient, robust neural systems. Mitigation is possible through judicious loss design, content-aware adaptivity, and preference-aligned supervision, but the intrinsic link between model compression and hallucination rate cannot be eliminated—it can only be managed and made transparent within system objectives.

---

References

• "Hallucination is a Consequence of Space-Optimality: A Rate-Distortion Theorem for Membership Testing" (Guo et al., 31 Jan 2026)
• "From Hallucination to Articulation: LLM-Driven Losses for Ultra Low-Bitrate Neural Speech Coding" (Yi et al., 5 Feb 2026)
• "From Dispersion to Attraction: Spectral Dynamics of Hallucination Across Whisper Model Scales" (Viakhirev et al., 31 Mar 2026)
• "Shorter, but Still Trustworthy? An Empirical Study of Chain-of-Thought Compression" (Zeng et al., 5 Apr 2026)
• "Conditional Hallucinations for Image Compression" (Aczel et al., 2024)
• "Seeing Through the Chain: Mitigate Hallucination in Multimodal Reasoning Models via CoT Compression and Contrastive Preference Optimization" (Fang et al., 3 Feb 2026)