AnetHallu-117K: Video Hallucination Dataset

• The paper presents AnetHallu-117K, a dataset using controlled hallucination injection and correction for enhanced factuality judgment in video-caption alignment.
• It outlines a pipeline that generates dense captions from ActivityNet clips, creates negative samples via a two-stage hallucinate-then-correct process, and applies bias suppression.
• The dataset fine-tunes a Factual-Aware Evaluator that achieves improved accuracy and reduced bias on external benchmarks in hallucination evaluation.

Searching arXiv for the specified paper and related context. AnetHallu-117K is a large-scale hallucination judgment preference dataset introduced in the video-understanding framework "Alternating Perception-Reasoning for Hallucination-Resistant Video Understanding" (Pu et al., 23 Nov 2025). It is constructed from human-annotated ActivityNet segments and is designed specifically to tune the Factual-Aware Evaluator (FAE), an evaluator that scores whether a caption about a video clip is factual or hallucinated. The dataset centers on short video clips paired with candidate captions, factual descriptions, and binary judgments, and it is further organized into preference pairs suitable for ORPO-based tuning of a multimodal evaluator (Pu et al., 23 Nov 2025).

1. Dataset role within hallucination-resistant video understanding

AnetHallu-117K is not presented as a general-purpose video captioning corpus. Its stated role is to support hallucination judgment and preference learning for FAE, which is then used as an anti-hallucination reward source in the broader Video-PLR framework (Pu et al., 23 Nov 2025). In that framework, hallucination is tied to insufficient or imprecise visual evidence, and the evaluator is intended to encourage sufficient and precise video evidence during reasoning.

The dataset is therefore best understood as a supervision resource for factuality discrimination in video-caption alignment. Each example concerns a short clip $V_{[t_1,t_2]}$ rather than a full unsegmented video, reflecting the paper’s emphasis on localized evidence and temporally grounded perception. A plausible implication is that the dataset operationalizes hallucination at the level of clip-specific factual consistency rather than at the level of open-ended answer helpfulness or stylistic quality.

2. Construction pipeline

The base material is drawn from the human-annotated ActivityNet dataset. Each example begins with a short clip $V_{[t_1,t_2]}$ and its original coarse labels (Pu et al., 23 Nov 2025). From this starting point, the construction procedure produces a high-fidelity dense caption, controlled hallucinated variants, and corresponding repaired captions.

First, Qwen2.5-VL-32B ingests the original ActivityNet annotation and generates a ground-truth dense description $C_{gt}$ of the visual content. This dense caption serves as the factual reference for subsequent perturbation and correction.

Second, negative captions $C_{neg}$ are created with a two-stage "Hallucinate-then-Correct" pipeline. The model is prompted with $(C_{gt} + \text{error definition})$ to produce a hallucinated caption of a specified type while preserving length and style. The five hallucination types are:

1. Attribute Modification
2. Quantity Modification
3. Action Substitution
4. Detail Conflation
5. Temporal Reordering

These categories define the controlled error model used throughout the dataset. Because the hallucinations are injected from factual dense captions while preserving surface characteristics, the negatives are designed to remain close to the positives in form, differing primarily in factual validity.

Third, corrected positive captions $C_{pos}$ are produced by giving the model both $C_{neg}$ and $C_{gt}$ and asking it to repair $C_{neg}$ back into a factual caption. The paper states that these paired $C_{neg}/C_{pos}$ examples share surface statistics (Pu et al., 23 Nov 2025). This design is central to the dataset’s preference-learning utility, since it reduces the chance that models can solve the task through superficial lexical cues alone.

3. Bias suppression and judgment protocol

A distinctive component of AnetHallu-117K is its explicit text-bias elimination procedure. To remove spurious word-frequency cues, the dataset computes a vocabulary-bias ratio and a caption-level penalty score:

$V_{[t_1,t_2]}$0

$V_{[t_1,t_2]}$1

$V_{[t_1,t_2]}$2

The procedure then iteratively removes the top $V_{[t_1,t_2]}$3 highest-scoring captions over $V_{[t_1,t_2]}$4 iterations with vocabulary set size $V_{[t_1,t_2]}$5, and repeats the process symmetrically for the positive side (Pu et al., 23 Nov 2025). The explicit goal is to suppress lexical artifacts that could let a classifier distinguish positive and negative examples without genuine video grounding.

The dataset also uses what the paper calls CoT-Enhanced Low-Bias Judgement. Each candidate caption $V_{[t_1,t_2]}$6 is paired with its video clip, and the model is forced first to generate a short factual description $V_{[t_1,t_2]}$7 of the clip before voting "Yes/No" on whether $V_{[t_1,t_2]}$8 is correct. Only examples for which the model’s accuracy is strictly between $V_{[t_1,t_2]}$9 and $C_{gt}$0 are retained, yielding the final preference exampleset of approximately $C_{gt}$1K examples (Pu et al., 23 Nov 2025).

This protocol is notable because it does not treat caption verification as a purely text-side classification task. Instead, it inserts an intermediate factual description step, which plausibly constrains the judgment process to attend to visual evidence before producing the binary label.

4. Example schema and preference structure

The paper defines each retained example as a tuple

$C_{gt}$2

where $C_{gt}$3 (Pu et al., 23 Nov 2025). Here, $C_{gt}$4 is the clip, $C_{gt}$5 is the candidate caption to be judged, $C_{gt}$6 is the intermediate factual description generated under the CoT-enhanced protocol, and $C_{gt}$7 is the final binary factuality label.

For FAE training, the dataset is further organized into preference pairs of the form $C_{gt}$8, where $C_{gt}$9, $C_{neg}$0 is the preferred correct response, and $C_{neg}$1 is the less preferred incorrect response (Pu et al., 23 Nov 2025). The paper specifies two subtypes:

• Answer-Accuracy: direct answer vs. direct answer
• Reasoning-Accuracy: CoT reasoning chain vs. CoT reasoning chain

This distinction matters because the dataset does not supervise only the correctness of final labels. It also supports preference learning over reasoning traces when chain-of-thought responses are present. A plausible implication is that AnetHallu-117K is intended to shape both verdict quality and the internal evidential structure of multimodal judgments.

5. FAE tuning and optimization objective

The Factual-Aware Evaluator is built on Qwen2.5-VL-7B with LoRA adapters applied to its vision encoder-language decoder (Pu et al., 23 Nov 2025). The paper states that the full $C_{neg}$2K examples are used to fine-tune FAE; no separate train/val/test splits are specified.

The reported training regimen is:

• single epoch of fine-tuning on $C_{neg}$3K preference examples
• learning rate $C_{neg}$4
• ORPO objective with $C_{neg}$5
• max frames $C_{neg}$6
• total pixels $C_{neg}$7 (Pu et al., 23 Nov 2025)

The ORPO loss is given as:

$C_{neg}$8

$C_{neg}$9

$(C_{gt} + \text{error definition})$0

with

$(C_{gt} + \text{error definition})$1

and $(C_{gt} + \text{error definition})$2 (Pu et al., 23 Nov 2025).

Within the larger Video-PLR system, the paper notes that the anti-hallucination reward $(C_{gt} + \text{error definition})$3 is later computed by the evaluator $(C_{gt} + \text{error definition})$4 as a weighted sum over each perceived evidence, but that mechanism pertains to FAE’s downstream usage rather than to dataset construction itself (Pu et al., 23 Nov 2025).

6. Statistics, benchmark behavior, and interpretive scope

The dataset statistics reported for AnetHallu-117K are summarized below (Pu et al., 23 Nov 2025).

Property	Value
Total examples	$(C_{gt} + \text{error definition})$5K caption-segment pairs
Positive labels	$(C_{gt} + \text{error definition})$6
Negative labels	$(C_{gt} + \text{error definition})$7
Direct answer labeling	$(C_{gt} + \text{error definition})$8
CoT reasoning followed by label	$(C_{gt} + \text{error definition})$9

The hallucination-type distribution is also reported:

Hallucination type	Share
Attribute Modification	$C_{pos}$0
Detail Conflation	$C_{pos}$1
Quantity Modification	$C_{pos}$2
Action Substitution	$C_{pos}$3
Temporal Reordering	$C_{pos}$4

The distribution is comparatively even across four categories, with Temporal Reordering somewhat lower than the others. This suggests that the dataset was designed to avoid collapse onto a single hallucination mode, though the paper does not claim exact uniformity.

On three external hallucination benchmarks, the FAE tuned on AnetHallu-117K achieves the following results (Pu et al., 23 Nov 2025):

Benchmark	Reported result
VideoHallucer (binary Y/N)	Acc $C_{pos}$5, Diff gap $C_{pos}$6
HEAVEN reformatted as caption task	Acc $C_{pos}$7, Diff $C_{pos}$8
VidHalluc MCQ	Acc $C_{pos}$9

The paper compares these against Qwen2.5VL baselines of $C_{neg}$0, $C_{neg}$1, and $C_{neg}$2, respectively, and states that the results match or exceed GPT-4o quality while reducing bias, with the accuracy gap between positive and negative categories dropping by $C_{neg}$3–$C_{neg}$4 (Pu et al., 23 Nov 2025).

Two interpretive points follow from the reported setup. First, AnetHallu-117K is specialized for factuality judgment under clip-caption alignment rather than for arbitrary multimodal reasoning tasks. Second, because no separate train/val/test splits are specified and the full $C_{neg}$5K examples are used to fine-tune FAE, the primary evidence for generalization comes from the external hallucination tests rather than from an internal held-out partition.

7. Significance and common misunderstandings

AnetHallu-117K is significant chiefly as an infrastructure dataset for evaluator training. Its purpose is not to directly train a generative video reasoner, but to tune an evaluator that supplies an anti-hallucination signal inside a loop-based perception-reasoning framework (Pu et al., 23 Nov 2025). Confusing it with a standard captioning benchmark would miss its preference-learning orientation and its explicit coupling to reward design.

Another potential misunderstanding is to treat the dataset as a purely synthetic caption corruption resource. While the negative captions are generated through controlled hallucination injection, the base clips come from human-annotated ActivityNet segments, the ground-truth dense descriptions are anchored in those annotations, and the construction includes explicit bias filtering and a video-conditioned judgment protocol (Pu et al., 23 Nov 2025). The intended effect is to make factuality judgments depend on visual evidence rather than on textual artifacts.

Finally, the coexistence of direct-answer and CoT-formatted responses indicates that the dataset spans both label-level and reasoning-level supervision. This suggests a broader conception of hallucination control in which correct final answers and evidentially grounded intermediate reasoning are both relevant targets.