ViHallu: Mitigating Hallucinations in Multimodal AI

• ViHallu is a unified framework that defines and categorizes hallucinations in multimodal AI with a detailed taxonomy and theoretical underpinnings.
• It integrates benchmark datasets, rigorous annotation protocols, and performance metrics to assess hallucination rates in VLMs, LVLMs, and LLMs.
• The framework advances mitigation strategies through data-driven fine-tuning, controlled visual interventions, and post-processing pipelines, enhancing trustworthiness.

ViHallu is a unified framework and a suite of methodologies and benchmarks focused on the identification, quantification, and mitigation of hallucinations in multimodal AI systems, with primary emphasis on Vision-LLMs (VLMs), Large Vision-LLMs (LVLMs), and LLMs in low-resource languages. The term "ViHallu" encompasses three distinct but interlinked research lines: a fine-grained theoretical and empirical taxonomy for visual hallucination in VLMs, a vision-centric intervention framework for reducing hallucinations via controlled visual variations, and, separately, a shared task and dataset—DSC2025 ViHallu—for hallucination detection in Vietnamese language LLMs. These coordinated efforts collectively address the fundamental challenges of trustworthiness, reliability, and safety in next-generation multimodal and multilingual AI.

1. Foundational Concepts and Formal Definitions

Visual hallucination is defined in the ViHallu paradigm as the generation of textual content by a VLM or LVLM that cannot be corroborated by the ground-truth entities, attributes, or relations present in the visual input (typically, an image or video). Given an image $x$, a model-generated text $F(x)$, and a set of validated entities $S(x)$ derived from the image, a token $t \in F(x)$ constitutes a hallucination if $t \notin S(x)$. The occurrence of any such token in $F(x)$ flags the sample as hallucinated:

$$H(F(x), x) = \begin{cases} 1 & \text{if } \exists t \in F(x): t \notin S(x) \ 0 & \text{otherwise} \end{cases}$$

This approach generalizes across modal boundaries (e.g., language, image, video) and underscores the need for systematic human or algorithmic annotation of ground-truth visual entities.

Textual hallucination in the context of LLMs is similarly conceptualized as the generation of fluent but factually incorrect, unsupported, or fabricated outputs relative to a given context. The ViHallu Challenge formalizes this via categorical labels: "no hallucination," "intrinsic hallucination" (contradicts source), or "extrinsic hallucination" (introduces unsupported but possibly true facts) (Nguyen et al., 8 Jan 2026, Rani et al., 2024).

2. Taxonomy of Hallucination Types

ViHallu elucidates a rigorously defined taxonomy of hallucination phenomena, especially for VLM-generated text. Eight mutually-exclusive categories are delineated for image captioning and Visual Question Answering (VQA) tasks:

• Contextual Guessing: Fabrication of entirely unrelated content.
• Identity Incongruity: Erroneous attribution of identities.
• Geographical Erratum: Misstatement of locations.
• Visual Illusion: Misinference due to visual confounders (e.g., shadows).
• Gender Anomaly: Incorrect assignment of gender.
• VLM as Classifier: Overconfident forced classification not grounded in pixels.
• Wrong Reading: Errors in transcribing or paraphrasing textual elements in the image.
• Numeric Discrepancy: Miscounts of objects or attributes.

Each hallucination event is annotated with the appropriate category, ensuring granular error analysis (Rani et al., 2024).

For LLMs (specifically in the ViHallu Challenge), hallucinations are partitioned into "intrinsic" (semantically or logically contradictory to the context) and "extrinsic" (assertions unsupported by context, regardless of global factuality), with all samples also validated for "no hallucination" (Nguyen et al., 8 Jan 2026).

3. Benchmark Datasets and Annotation Protocols

VHILT for Visual Hallucination

The VHILT (Visual HallucInation eLiciTation) dataset comprises 2,000 image-text pairs, evenly sampled from ten years of New York Times tweets and annotated by both experts and crowdworkers. Each instance involves either (i) free-form captioning by leading VLMs (Kosmos-2, MiniGPT-v2, Sphinx) or (ii) VQA by LLaVa, MiniGPT-4, and others. Annotations specify both presence/absence and the hallucination category, ensuring a balanced corpus for quantification and downstream evaluation (Rani et al., 2024).

ViHallu-Instruction for LVLM Hallucination Mitigation

ViHallu-Instruction is a vision-centric QA dataset designed for counterfactual fine-tuning, constructed by generating controlled visual variations of images. Images are selected, masked, and then locally modified (e.g., changing an attribute, object, background element) using techniques such as ControlNet++ guided by human-vetted captions. For each image–variation pair, a pipeline produces QA pairs probing sensitivity to attribute, presence, relationship, and counting changes; only high-confidence QA pairs (verified by multiple LVLMs) are included. The resulting ~50,000 QA pairs span fine-grained visual reasoning tasks over 6,770 images (Dai et al., 29 Jul 2025).

DSC2025 ViHallu for Vietnamese LLM Hallucination

The DSC2025 ViHallu Challenge corpus consists of 10,000 context–prompt–response triplets extracted from 4,351 Vietnamese passages (UIT-ViQuAD 2.0). Samples are systematically stratified by prompt type—factual, noisy, adversarial—and hallucination category, with uniform class balance. Annotation involves detailed guidelines, peer review, adjudication by senior annotators, and optional confidence scoring (Nguyen et al., 8 Jan 2026).

4. Evaluation Metrics and Experimental Results

Quantification of hallucination incidence and mitigation effectiveness occurs via standardized metrics:

• Hallucination Rate (HR): Fraction of samples exhibiting at least one hallucination event.
• Average Hallucination Count (AHC): Mean number of hallucinated tokens per sample across corpus.
• Macro-F1 (ViHallu-Challenge): Mean F1 across "no," "intrinsic," and "extrinsic" categories.
• Object Hallucination Probes (POPE): Accuracy and F1 under random, popular, and adversarial negative sampling (Dai et al., 29 Jul 2025).
• Aggregate Reasoning Benchmarks: LLaVA-Bench, MMHal-Bench—overall and per-category scores and hallucination rates.
• Statistical Consistency: Macro-averaged classwise metrics are reported to guard against class imbalance.

In Vietnamese LLM detection, baseline encoder-only models (PhoBERT) achieve 32.83% macro-F1, while instruction-tuned LLM ensembles reach up to 84.80% on the private test split, with intrinsic hallucination detection being the limiting factor (Nguyen et al., 8 Jan 2026).

LVLMs fine-tuned with ViHallu-Instruction attain marked improvements on standard and adversarial benchmarks. For instance, Qwen2-VL shows accuracy/F1 increases (e.g., 91.07%/90.44% on POPE random), and hallucination rates fall from 0.51 to 0.48 on MMHal-Bench (Dai et al., 29 Jul 2025). Performance consistently surpasses text-centric augmentation methods.

Empirical radar plots and qualitative evaluations reveal across-the-board gains in object, attribute, relation, counting, and adversarial reasoning following ViHallu-based fine-tuning (Dai et al., 29 Jul 2025).

5. Vision-Centric Hallucination Mitigation with Controlled Variations

ViHallu's distinguishing methodological advance is the automatic generation of visually controlled counterfactuals for LVLM fine-tuning:

• For an image $x$, segmentation masks $m$ (via MobileSAM) and descriptive captions $c$ (via Tag2Text) are edited to produce $F(x)$0, triggering a modified image $F(x)$1 via ControlNet++.
• Modification is restricted to mask-selected regions, preserving global layout ($F(x)$2 regularization).
• Candidate images are retained only if $F(x)$3 as assessed by a VQA module.
• Subsequent QA instruction construction employs LVLM-generated detailed captions, object tag extraction, and cross-model answer verification.
• Fine-tuning proceeds with a joint loss over both original and variation QA pairs, balancing $F(x)$4 and $F(x)$5.

This vision-centric paradigm guides models towards robust visual-semantic alignment and fine-grained visual understanding, in contrast to prior preference-based, text-only augmentation approaches (Dai et al., 29 Jul 2025).

6. Error Analysis, Failure Modes, and Mitigation Strategies

Error breakdowns show persistent challenges:

• Intrinsic hallucinations in LLMs, requiring advanced NLI capabilities to identify contradictions, are underdetected relative to extrinsic types (Nguyen et al., 8 Jan 2026).
• LVLMs struggle with subtle foreground modifications against steady backgrounds and temporal sequence inversion, signaling overreliance on context or background style (Huang et al., 30 Jun 2026).
• Multi-model analyses confirm that foreground semantic changes (rather than global scene statistics) are the dominant axis for wrong predictions (Dai et al., 29 Jul 2025).

ViHallu situates mitigation approaches into three main families:

• Data-driven: fine-tuning with annotated hallucination corpora or large-scale synthetic visual instruction datasets.
• Training objectives: methods such as Visual Contrastive Decoding (robustness to input perturbations), object-masked MLM, and multimodal feature fusion (CLIP+DINOv2).
• Post-processing: pipeline systems (Woodpecker, VOLCANO, LURE) for hallucination diagnosis and output refinement.

Contrastive fine-tuning on background-controlled, adversarial (foreground-varying) video pairs is recommended for LVMs, according to video analysis benchmarks (Huang et al., 30 Jun 2026).

7. Broader Implications and Future Directions

ViHallu's multilevel framework has catalyzed a shift toward vision-centric, fine-grained, and multilingual hallucination research. Open research directions include span-level localization of hallucinated segments, retrieval-augmented output verification, confidence estimation/calibration for selective abstention, adaptive weighting schedules during variation-based fine-tuning, and cross-lingual transfer to underserved languages (Nguyen et al., 8 Jan 2026, Dai et al., 29 Jul 2025, Rani et al., 2024).

For LVLMs, scaling controlled-variation methodologies to video, dynamic scenes, and real-time streams (incorporating temporal consistency and error recovery as explored in DIQ-H) is a recognized need. More adaptive curricula, automatic difficulty calibration, and self-correction processes undergird ongoing innovation (Lin et al., 3 Dec 2025).

The ViHallu framework—encompassing taxonomy, measurement, remediation, and benchmark construction—constitutes a blueprint for systematically reducing hallucination rates and enhancing factual reliability across language, vision, and multimodal AI domains.