Hallucination Taxonomy in LFMs

• Hallucination Taxonomy is a structured framework that categorizes instances where LFMs generate content diverging from factual reality across different modalities.
• It details formal definitions, concrete examples, and specific evaluation metrics like token confidence, CHAIR, and FactVC to assess factual consistency.
• It discusses mitigation strategies—including external knowledge integration and confidence filtering—and outlines open research directions such as self-supervised detection and threshold tuning.

Hallucination in the context of large foundation models (LFMs) refers to the generation of content that diverges from factual reality or includes fabricated information not supported by input data. This phenomenon manifests across text, image, video, and audio modalities, each with specific formalizations, empirical prototypes, and evaluation metrics. The following taxonomy systematizes the major types of hallucination studied in LFMs, decomposes them by modality and task, relates their technical signatures, and surveys current evaluation, mitigation strategies, and pressing research challenges (Rawte et al., 2023).

1. Modality-Structured Hallucination Taxonomy

The taxonomy is rooted in four top-level modalities, each further subdivided as follows:

Modality	Subcategories
Text	General LLMs, Multilingual LLMs, Domain-Specific LLMs
Image	Object Hallucination in Vision–LLMs (VLMs)
Video	Hallucination in Video Captioning and Understanding
Audio	Hallucination in Audio Captioning and Generation

This hierarchy delineates not only the input/output channels (text/image/video/audio) but also key application axes (general-purpose, translation, domain-specificity).

2. Taxonomy Branches: Definitions, Examples, and Metrics

2.1 Text-based Hallucination

2.1.1 General LLMs

• Formal definition: Let $x$ be a user prompt, $\hat{y}$ the model's output, $G$ the ground-truth reference. For an output statement $\phi\subset\hat{y}$, $\phi$ is a hallucination if it is not entailed by $G$, i.e., $E(\phi,G)<\tau_e$ for some threshold.
• Illustrative examples: "Marie Curie was born in Berlin." (incorrect), or self-contradictory output.
• Metrics:
  • Token-level confidence $p(\hat{y}_i|x)$, entropy $H(\hat{y}_i|x)$.
  • F1 or accuracy on annotated hallucination spans.
  • Knowledge-F1 (KF₁) for QA, BLEU-4/ROUGE-L for generation, though these weakly correlate with factuality.

2.1.2 Multilingual LLMs

• Formal definition: For source sentence $s$ (lang A), translation $\hat{y}$0 (lang B): $\hat{y}$1. A hallucination if $\hat{y}$2, i.e., $\hat{y}$3.
• Examples: Translation introduces extra objects/details ("juice" → "grape juice").
• Metrics: spBLEU, human adequacy, percent of extra unaligned tokens.

2.1.3 Domain-Specific LLMs

• Definition: Let $\hat{y}$4 (e.g., UMLS for medicine) be the domain ontology. Output $\hat{y}$5 is a hallucination if $\hat{y}$6 or conflicts with domain constraints.
• Examples: Incorrect drug classification, legal misinterpretation.
• Metrics: Med-HALT accuracy, ELO-ranking (ChatLaw), Reasoning-Hallucination-Test (RHT), Memory-Hallucination-Test (MHT).

2.2 Image-based Hallucination

2.2.1 Object Hallucination in VLMs

• Definition: Given image $\hat{y}$7, ground-truth object set $\hat{y}$8, and generated caption $\hat{y}$9 referencing $G$0. Hallucinated objects $G$1.
• Examples: Caption asserts nonexistent objects ("dog chasing a ball"—no dog present).
• Metrics: CHAIR $G$2, POPE (measures object consistency across prompts).

2.3 Video-based Hallucination

2.3.1 Video Captioning & Understanding

• Definition: For video $G$3, caption $G$4, and factual-consistency model $G$5. $G$6 is hallucinatory if $G$7.
• Examples: Temporal/causal errors; describing unseen events or sounds.
• Metrics: FactVC (video–language entailment), human error rate (up to ~57%).

2.4 Audio-based Hallucination

2.4.1 Audio Captioning & Generation

• Definition: Given audio $G$8, generated caption $G$9, audio–text alignment model $\phi\subset\hat{y}$0. Hallucination if $\phi\subset\hat{y}$1.
• Examples: Caption adds sound sources not present ("piano solo" when only drums).
• Metrics: BLEU-n, METEOR, ROUGE-L (text match); mean average precision (mAP) for classification.

3. Cross-Category Relationships and Unifying Themes

• All branches instantiate the fundamental criterion: "Generated content is not supported by the input (text, image, video, or audio)."
• Multilingual and general text hallucinations blend in translation applications.
• Domain-specific models layer ontology-derived constraints atop general patterns.
• Object hallucinations in image and factuality errors in video both map to input–output entailment failures; modality-adapted metrics (e.g., CHAIR, FactVC) generalize text entailment tests.
• Audio captioning borrows text metrics but incorporates alignment signals unique to the audio domain.

4. Prevalence, Impact, and Mitigation Strategies

Prevalence

• Text LLMs: Hallucination rates 10–30% in open generation.
• Vision–LLMs: ~30% object hallucination (InstructBLIP).
• Video captioning: >50% of sentences contain errors.
• Audio captioners: 20–40% misalignment, especially with pseudo-labeling.

Impact

• High-stakes domains (medicine, law): Even infrequent hallucinations present legal/clinical risk.
• Multimodal assistants: Hallucinated facts degrade user trust, impact downstream automation (e.g., robotics).

Mitigation

• External Knowledge Integration: Chain-of-Knowledge, LLM-Augmenter, citation/provenance tracking.
• Iterative Prompting and Formal Methods: Repeated generation/verification cycles.
• Data-centric Interventions: Curated in-domain benchmarks, adversarial augmentation.
• Confidence Filtering: Use of token- or sequence-level confidence to reject low-support outputs.

5. Open Research Questions and Future Directions

• Unified Metrics: Defining modality-agnostic entailment or consistency metrics across text, image, video, and audio (e.g., universal $\phi\subset\hat{y}$2).
• Probabilistic Modeling: Developing systems where low $\phi\subset\hat{y}$3 triggers automatic hallucination alerts.
• Task-Specific Thresholds ($\phi\subset\hat{y}$4): Customizing decision thresholds to account for domain-specific risk, e.g., stricter for clinical outputs.
• Self-Supervised Detection: Can models autonomously detect their own hallucinations via internal contradiction or uncertainty signals?
• Human-in-the-Loop Correction: Designing UI/UX paradigms allowing robust post-generation correction by domain experts.
• Ethical/Legal Responsibility: Formalizing procedures and legal norms for cases where LFMs hallucinate in consequential settings.

This operational taxonomy, with its formal definitions, canonical evaluation metrics, representative failure cases, and mitigation practices, frames the study and ongoing development of reliable, domain-appropriate, and safe large foundation models across application domains (Rawte et al., 2023).