Semantic Aggregation Hallucination (SAH)
- SAH is a failure mode where semantically accurate local units (e.g., frames or tokens) are incorrectly combined into misleading global narratives.
- Benchmarks like ELV-Halluc reveal that the SAH ratio increases with event complexity and rapid semantic changes in long video contexts.
- Mitigation strategies such as enhanced positional encoding and Direct Preference Optimization (DPO) have proven effective in reducing SAH errors.
Semantic Aggregation Hallucination (SAH) is a failure mode observed in multimodal and LLMs, wherein errors arise not through failure of local recognition (e.g., frame-level or token-level semantics), but during the process of aggregating these locally-correct semantics into global, contextually-coherent representations. SAH is especially prevalent in settings requiring the integration of information across temporal events (in long videos) or across multiple semantic units (in text or structured contexts), and may manifest as misattribution, semantic drift, or mixing of details between unrelated events or concepts. Key research on SAH has focused on diagnosis, benchmarking, and mitigation—most notably in long video understanding, but with implications for other aggregation-rich tasks.
1. Definition and Distinctive Characteristics
Semantic Aggregation Hallucination refers to a class of model errors where semantically correct local units (e.g., frame-level object detections, sentence-level facts) are incorrectly synthesized into higher-level aggregates (such as event-level summaries or passage-level narratives). In the context of long video understanding, SAH specifically denotes cases where the model accurately perceives frame-level semantics but produces errors at the event-grouping level—for example, assigning a visual element from Event to Event , despite each event possessing its own coherent semantic structure (Lu et al., 29 Aug 2025). Unlike hallucinations stemming from missing data or biased priors, SAH arises from temporal or discourse-level confusion: errors are introduced as the model attempts to combine, align, or summarize multiple pieces of information. This failure mode is critical in domains where the sequence and contextual relationships are integral to meaning.
2. Underlying Causes and Contributing Factors
The occurrence of SAH is closely tied to the complexity and rapidity of semantic changes across aggregated units. In long-video models, increased event count and event complexity exacerbate SAH: models become more prone to misattributing frame-level details when required to handle multiple, semantically-intense events (Lu et al., 29 Aug 2025). Rapid changes in fine-grained visual aspects—such as color, shape, or spatial relationships—further increase the likelihood of such errors. While traditional video hallucination benchmarks have attributed hallucinations to strong language priors or vision-language encoder biases, these explanations are insufficient for SAH. SAH instead emerges from failures in temporal integration, aggregation mechanisms (such as positional encoding), and the lack of explicit event-level semantic binding.
Mathematically, the SAH ratio is formalized to isolate errors in aggregation: where OutAcc is accuracy on out-of-video (fully fabricated) hallucination pairs, and InAcc is accuracy on in-video (misattributed event) hallucination pairs.
3. Benchmarking and Evaluation: ELV-Halluc
The ELV-Halluc benchmark is a dedicated framework for systematic investigation of SAH in long-video models (Lu et al., 29 Aug 2025). ELV-Halluc utilizes an "Event-by-Event" dataset, where each video is segmented into defined semantic events. The benchmark generates adversarial query–answer pairs:
- In-video hallucinated captions: details borrowed from other events within the same video,
- Out-of-video hallucinated captions: fully fabricated content not present in any event.
By comparing model accuracy on these pairs, ELV-Halluc quantifies the model's susceptibility to semantic misaggregation. Evaluation spans visual detail, object, action, and declarative aspects. Experimental findings confirm that the SAH ratio grows with the number of events—indicating that aggregation-induced errors become more prevalent as temporal-semantic complexity increases. Models were also shown to be particularly sensitive to perturbations where fine-grained semantics change rapidly, revealing that SAH is orthogonal to overall hallucination rates prevalent in short-video or single-event benchmarks.
4. Mitigation Strategies
To alleviate SAH, two architectural and training approaches have demonstrated efficacy:
- Strengthening Positional Encoding: Enhanced temporal positional encodings (e.g., VideoRoPE) enable the model to maintain clear temporal-event boundaries. Comparative experiments demonstrated reduced SAH ratios for VideoRoPE versus vanilla RoPE and other variants, suggesting that fine-grained positional information aids in accurate frame-to-event mapping.
- Direct Preference Optimization (DPO): Training with adversarial data pairs—especially those where in-video hallucinated captions require the model to disambiguate plausible but semantically misattributed information—proved highly effective. DPO training led to a 27.7% reduction in SAH ratio, as well as marginal improvements on overall video understanding benchmarks such as VideoMME (Lu et al., 29 Aug 2025). This approach incentivizes the model to prefer correct event-level aggregation over plausible-looking, but incorrect, misattributions.
Improvements were shown to be specifically tied to aggregation mechanisms; scaling model size or increasing the number of frames merely improved global accuracy, not SAH directly.
5. Dataset Construction, Adversarial Evaluation, and Experimental Findings
The ELV-Halluc benchmark is underpinned by a curated set of 8,000 adversarial QA pairs, with 348 videos fully annotated and 200 selected for detailed evaluation. Each video is segmented into events by a semi-automated pipeline (using horizon models such as Gemini 2.5 Flash, with manual refinement), then used to derive ground truth, in-video, and out-video caption pairs. Experiments across 14 open-source and 2 closed-source models, evaluated on four semantic aspects, found that models' vulnerability to SAH scales with semantic complexity and rate of semantic change—not with model size or average performance on simpler benchmarks. These results establish SAH as a distinct and nontrivial challenge, decoupled from previously reported hallucination rates.
6. Broader Implications and Directions for Future Research
SAH delineates a failure axis in aggregation-dependent tasks beyond long video: similar mechanisms can arise in text summarization (confusing narrative events), document-level QA (misattributing facts across paragraphs), and multimodal discourse analysis. Mitigation thus requires not only improved local recognition but also architectural provisions for correct event, temporal, or semantic binding. Future work may focus on:
- Advanced event-level aggregation modules (e.g., dynamic temporal attention, hierarchical event encoding)
- Scaling datasets and benchmarks to a wider variety of real-world long videos with diverse semantic complexities
- Integrating regularization or fine-tuning (DPO or related adversarial techniques) that reward correct aggregation while penalizing plausible but incorrect associations.
Extension of the benchmark to dynamic or cross-domain events will allow more comprehensive analysis and tailored mitigation of SAH, potentially leading to models with robust, contextually faithful aggregation mechanisms across modalities and temporal spans.
7. Summary Table: Dimensions of SAH in Long Video Understanding
| Aspect | Description | Impact on SAH |
|---|---|---|
| Frame-level semantics | Local recognition (objects, attributes) | Correct frame-level does not guarantee event-level accuracy |
| Event segmentation | Temporal grouping of frames | Increased events → higher SAH |
| Semantic complexity | Density and variability of events | Higher complexity → more SAH |
| Aggregation mechanism | Positional encoding, temporal attention | Weak encoding increases SAH |
| Training data | Adversarial in-video pairs | Improved aggregation (lower SAH) |
SAH represents a persistent challenge in any system requiring semantic integration across locally coherent units. Addressing SAH thus necessitates not only improvements in base recognition but, critically, the explicit design and training of aggregation mechanisms that preserve the integrity of event, temporal, or discourse-level structure.