Multi-Agent Visual Hallucination Snowballing

• The paper identifies multi-agent visual hallucination snowballing as a failure mode where initial misinterpretations in one agent are propagated and amplified by subsequent agents via textual communication.
• It details how visual attention decays across agent turns, with studies showing a 62% drop and loss of key unimodal visual tokens that preserve grounding.
• The study introduces ViF, a lightweight mitigation strategy that preserves visual evidence and reallocates attention, reducing hallucination propagation by up to 40%.

Searching arXiv for the cited papers and closely related work on multi-agent visual hallucination snowballing. arxiv_search(query="(Zhong et al., 2024) multimodal hallucination snowballing", max_results=5, sort_by="relevance") Multi-agent visual hallucination snowballing denotes a failure mode in which hallucinations about visual content are seeded by one vision-language agent and then propagated and amplified by subsequent agents, chiefly because visual information is relayed through textual messages rather than preserved as visual evidence. In the broader multimodal literature, the precursor phenomenon was defined for multi-turn interaction within a single Large Vision-LLM (LVLM): previously generated hallucinations can bias later turns even when the correct visual information is present (Zhong et al., 2024). In Multi-Agent Systems (MAS) powered by Visual LLMs (VLMs), the same basic failure is re-expressed at system scale, where inter-agent communication itself becomes the conduit through which visual errors accumulate (Yu et al., 26 Sep 2025).

1. Conceptual lineage and definition

The single-model antecedent is multimodal hallucination snowballing, introduced as a conversational phenomenon in which hallucinations generated by LVLMs—responses misaligned with the visual input—can influence subsequent turns. If a later query references or is related to the earlier hallucination, the model may reinforce the mistake even though it could have answered correctly without the misleading context. The original formulation distinguishes multimodal hallucinations from snowballing: the former are responses not grounded in the given visual information, while the latter is the accumulation and propagation of these errors through multistep conversation, resulting in a compounding effect (Zhong et al., 2024).

The multi-agent extension is multi-agent visual hallucination snowballing, described as a novel failure term in MAS powered by VLMs. Here, hallucinations are seeded in a single agent and amplified by following ones due to the over-reliance on textual flow to relay visual information. This differs from single-agent hallucination because the failure is not confined to one model’s output distribution; it is embedded in the communication protocol of the system. Each downstream agent increasingly relies on prior textual outputs rather than direct visual evidence, so the system loses the capacity for self-correction that might otherwise come from re-grounding against the image (Yu et al., 26 Sep 2025).

This definition has immediate practical force for multi-turn, multi-agent deployments such as assistive agents or collaborative tools, where early mistakes can cascade and be amplified. A plausible implication is that reliability in visual MAS cannot be inferred from single-agent visual QA accuracy alone, because the dominant failure may arise from how agents relay evidence rather than from how any one agent initially perceives it (Zhong et al., 2024).

2. Mechanistic basis: textual relay, visual attention decay, and token degradation

The central mechanistic claim in VLM-based MAS is that textual flow gradually displaces visual grounding. Through turn-, layer-, and token-wise attention analyses, the system-level diagnosis identifies hallucination snowballing with a reduction of visual attention allocation. In one illustrative analysis on LLaVA-NeXT-7B, the average attention allocation to visual tokens drops from 0.165 in the first agent turn to 0.063 in the 20th agent turn, a 62% reduction. The middle layers undergo the steepest decline, at approximately 60%, compared with shallow layers at 21% and deep layers at 30% (Yu et al., 26 Sep 2025).

A more specific finding concerns a subset of vision tokens whose attention profile exhibits a unimodal peak in the middle layers. In the first turn, these tokens show a “mountain-shaped” attention pattern and are identified as the tokens that best preserve visual evidence. Ablation studies reported that removing these unimodal tokens causes much deeper performance drops than removing random, inactive, or monotonic-rise/fall tokens, especially in middle layers. Over successive agent turns, however, the proportion of unimodal tokens declines rapidly, while other, less informative visual tokens slightly increase in proportion. The reported conclusion is that loss of this unimodal attention peak is both a marker and a likely driver of visual information loss and subsequent hallucination snowballing (Yu et al., 26 Sep 2025).

This attention-based account aligns with the earlier conversational LVLM results. There, the failure was attributed to over-reliance on linguistic context: once seeded with plausible, contextually consistent hallucinations, models neglect real visual evidence. In the single-model setting, the effect appears as susceptibility to misleading dialogue history; in MAS, it appears as susceptibility to inter-agent textual relay. This suggests that multi-agent visual hallucination snowballing is not merely more of the same error, but the consequence of a communication bottleneck in which textual summaries become privileged over native visual tokens (Zhong et al., 2024).

3. Experimental signatures and measurement

The precursor evaluation framework is MMHalSnowball, which formalizes the phenomenon in a controlled two-turn setting. It uses the GQA balanced validation set and Visual Genome datasets, and considers four hallucination types: Existence, Attribute, Relation, and Imagination. Each sample contains an image, a ground-truth fact, and a visual question. A two-turn conversation is then constructed: in Turn 1 the model is given a hallucinated description contradicting the image, and in Turn 2 it is asked a targeted question that leverages this hallucinatory context. The same question is also posed without the hallucinatory context as CleanConv (Zhong et al., 2024).

The framework defines Accuracy as

$$\text{Acc}(Y, \hat{Y}) = \frac{1}{n} \sum_{i=1}^n \text{Score}_i(y_i, \hat{y}_i),$$

where $\text{Score}_i = 1$ if answer $y_i$ is in generated response $\hat{y}_i$, else $0$. It also defines Flip Rate (FR) as the proportion of cases where, having previously answered correctly in CleanConv., the model flips in HalluConv. and gives the hallucinated answer:

$$FR = \frac{\sum_{i \in D^+}\text{Score}_i(y_i^-, \hat{y}_i^-)}{\text{Acc}(Y^+, \hat{Y})}.$$

A Weak Flip Rate (WFR) further generalizes FR by counting answers that conflict with the original regardless of whether they exactly match the seeded hallucination (Zhong et al., 2024).

The quantitative results are severe. Most open-source LVLMs’ accuracy drops by at least 31% in a hallucinatory conversational context. Reported examples include LLaVA-1.5, whose CleanConv. accuracy is 71.24% and HalluConv. accuracy 14.96%, ShareGPT4V at 71.81% versus 15.91%, and mPLUG-Owl2 at 60.47% versus 7.82%. Flip rates are often 70%–90%, and imagination hallucinations yield nearly 100% flip rate, meaning that invented objects seeded earlier are accepted almost universally when referenced later. A control result is equally important: longer but hallucination-free contexts do not cause such accuracy drops, confirming that the degradation is due to contextual hallucination rather than context length (Zhong et al., 2024).

At MAS scale, the 2025 formulation introduces a dedicated Hallucination Snowballing (HS) Score:

$$HS = \frac{1}{N} \sum_{i=1}^N \frac{1}{1 + \exp\left( \frac{D}{2} - d_i \right)} h_i,$$

where $d_i$ is hallucination propagation distance from origin, $h_i$ is severity of hallucination, $N$ is the number of agents, and $\text{Score}_i = 1$0 is the maximum depth in the MAS. This metric was evaluated across eight benchmarks based on four common MAS structures and ten base models. The reported structural result is that circular topologies, which induce the most interaction, amplify snowballing most severely; baseline systems and competing methods degrade sharply as the number of agent turns increases (Yu et al., 26 Sep 2025).

4. ViF and the preservation of visual evidence

The principal mitigation proposed specifically for MAS is ViF, a lightweight, plug-and-play mitigation paradigm that relays inter-agent messages with Visual Flow powered by selected visual relay tokens and applies attention reallocation to amplify the desired pattern. Its first step is visual relay token selection: at each turn, the system selects the subset of visual tokens exhibiting the unimodal attention peak in the middle layers. If $\text{Score}_i = 1$1 is the full set of visual tokens and $\text{Score}_i = 1$2 the selected relay tokens, those tokens are then contextualized with instruction tokens using a lightweight Transformer block:

$\text{Score}_i = 1$3

The contextualized relay tokens are inserted at the start of the token sequence for the next agent, before fresh vision and instruction tokens (Yu et al., 26 Sep 2025).

A second component is attention reallocation. In middle layers, ViF amplifies dynamic vision-token attention trends using temperature scaling,

$\text{Score}_i = 1$4

with $\text{Score}_i = 1$5 to sharpen attention. Attention mass is reallocated from inactive visual tokens and instructions to the remaining active vision tokens; in deep layers the process is reversed, gradually returning more weight to instruction tokens as visual integration wanes. For models that do not expose explicit attention weights under Flash-Attention, a Key-Norm alternative is provided and described as highly correlated with attention-based selection (Yu et al., 26 Sep 2025).

The empirical effect is consistent. ViF improves average performance by 2.4–3.8% across baselines, models, MAS structures, and tasks, with larger VLMs gaining over 4%. It reduces HS by at least 30%, and by up to 40% in circular structures. It is also described as the only evaluated method that directly targets and effectively suppresses snowballing propagation; methods aimed primarily at single-agent hallucination often perform no better—or even worse—than baseline in MAS because they do not intervene in the cross-agent propagation pathway (Yu et al., 26 Sep 2025).

5. Related mitigation paradigms

Visual hallucination snowballing has also been approached from adjacent perspectives, often outside the specific MAS relay setting. Residual Visual Decoding (RVD) is a training-free inference-time method proposed for conversational LVLMs. It revises the output distribution of the model with one derived from the residual visual input, thereby giving the model direct access to visual information when the dialogue history is misleading. On LLaVA-1.5, HalluConv. accuracy improves from 14.96% to 32.84%, and flip rate drops from 78.21% to 53.52%; on mPLUG-Owl2, accuracy improves from 7.82% to 22.54%, and flip rate falls from 86.63% to 39.15% (Zhong et al., 2024). Although developed for a single conversational agent, the method addresses the same underlying asymmetry between linguistic context and visual grounding.

Other multi-agent methods pursue verification or deliberation rather than visual-token relay. “Interpreting and Mitigating Hallucination in MLLMs through Multi-agent Debate” uses a conservative debater, an imaginative debater, and a judge, combining self-reflection for slow-thinking with debate for divergent-thinking, and reports generalized hallucinations-mitigating performance across several MLLMs (Lin et al., 2024). MUG reframes multimodal debate as a social deduction game that detects hallucinating agents through multimodal counterfactual tests; on HallusionBench, Qwen2.5VL-7B with MUG reaches 53.8% average versus 37.8% for MAD-Vote and 50.2% for MAD-Judge (Liang et al., 14 Nov 2025). InEx combines entropy-based introspective reasoning with external cross-modal collaboration among a decision agent, textual self-reflection agent, image editing agent, and visual self-reflection agent, and reports 4%–27% gains on general and hallucination benchmarks (Yang et al., 2 Dec 2025).

A separate but related line treats localization failure as the source of visual hallucination. DocLens addresses long visual document understanding with a tool-augmented multi-agent framework that navigates from full documents to specific visual elements and then uses sampling-adjudication to produce a single answer. With Gemini-2.5-Pro, it reports 97.3% page recall and improves accuracy on the unanswerable subset of MMLongBench-Doc from 59.9% to 72.2%, a gain of 13.8% (Zhu et al., 14 Nov 2025). In embodied 3D settings, 3D-VCD performs training-free visual contrastive decoding over original and distorted 3D scene graphs; under Scene-Task Contradiction, it reduces hallucination rate from 53.9% to 1.5% (Ogunleye et al., 9 Apr 2026). These methods do not define multi-agent visual hallucination snowballing in the same way as ViF, but they support the general conclusion that preserving or reintroducing grounded visual evidence is decisive for mitigation.

6. Boundary conditions, misconceptions, and broader implications

A central misconception is that multi-agent systems necessarily amplify hallucination. Evidence from text-only cascades complicates that claim. In “Hallucination Cascade: Analyzing Error Propagation in Multi-Agent LLM Systems”, 3-agent chains reduce the normalized hallucination score from 0.422 at the first agent to 0.272 at the final agent, with an amplification factor of 0.644, indicating net attenuation rather than snowballing. This attenuation is accompanied by a drop in factual accuracy from 0.789 to 0.769, so hallucination suppression is traded against factual preservation (Jamshidi et al., 6 Jun 2026). The authors explicitly report that no systemic hallucination snowballing was observed in sequential multi-agent LLM cascades.

That contrast is instructive. It suggests that hallucination propagation is strongly dependent on modality, grounding mechanism, and communication substrate. In purely textual cascades, sequential refinement may weaken unsupported claims. In visual MAS, by contrast, agents often communicate visual content through text while their native visual evidence decays across turns. The result is a modality-specific amplification channel in which the system becomes progressively less visual as it becomes more conversational (Yu et al., 26 Sep 2025).

This interpretation is reinforced by work on structured multi-agent visual reasoning. Visual Para-Thinker++ attributes hallucination partly to early perceptual commitment in single-chain reasoning and addresses it with role-conditioned Main, Worker, and Summary Agents, context isolation among Workers, and trace-level reconciliation by the Summary Agent. On HallusionBench, the 3B system reaches 64.0%, a gain of 7.9 points over Para-Thinker (Xu et al., 8 Jun 2026). The architectural lesson is that visual reliability in multi-agent reasoning depends not only on whether agents debate, but on whether the system preserves independent visual hypotheses long enough for a later mechanism to reconcile them.

From a system-design perspective, the literature converges on several implications. First, evaluation must track propagation pathways rather than only final outputs: MMHalSnowball uses flip-based metrics for conversational corruption, while ViF introduces a turn-aware HS score (Zhong et al., 2024). Second, mitigation must preserve or reinsert grounded visual evidence rather than rely exclusively on better prompting. Third, MAS design should distinguish between helpful deliberation and lossy textual relay. Where the former can improve reasoning, the latter can convert a local visual error into a global system failure.