HaloProbe: Bayesian Hallucination Detector

• HaloProbe is a Bayesian hallucination detector that leverages internal decoding signals and external caption structure features to identify object-level errors.
• It overcomes confounding effects from token position and repetition that mislead coarse attention averages, thereby avoiding Simpson’s paradox.
• The method trains on a balanced dataset to isolate internal evidence and supplies a non-invasive scoring signal that enhances captioning accuracy and fluency.

HaloProbe is a Bayesian hallucination detector and mitigation probe for large vision-LLMs (LVLMs) that targets object hallucinations in open-ended image captioning and description generation. It estimates token-level hallucination probabilities by combining internal decoding evidence with external caption-structure features, then uses the resulting posterior as an external scoring signal for non-invasive mitigation. Its central methodological claim is that coarse-grained attention-based analysis is unreliable because token position and object repetition act as hidden confounders, producing Simpson’s paradox when attention statistics are aggregated; HaloProbe therefore factorizes internal and external evidence rather than treating raw attention as a sufficient signal (Zohrabi et al., 7 Apr 2026).

1. Problem setting and prediction target

HaloProbe is formulated at the level of individual object tokens in a generated caption $c$. For each object token $c_t$, it defines a binary label

$y(c_t) \in \{0,1\},$

where $y(c_t)=1$ denotes an object that is correct or grounded in the image, and $y(c_t)=0$ denotes an object that is hallucinated. The detection problem is to estimate

$p\big(y(c_t)\mid x_i(c_t),x_e(c_t)\big),$

with $x_i(c_t)$ representing internal features extracted from the LVLM during decoding and $x_e(c_t)$ representing external features derived from caption structure (Zohrabi et al., 7 Apr 2026).

The external features used by HaloProbe are token position $t$, object repetition count $r(c_t)$, and a first-versus-non-first occurrence indicator $c_t$0. The internal features include fine-grained attention across layers and heads, together with decoder confidence or logit statistics. This feature split is substantive rather than cosmetic: external features encode regularities of description structure, while internal features are intended to capture grounding evidence available during token generation.

This setup places HaloProbe in a narrower category than generic caption-level factuality scoring. It is explicitly a token-level object-hallucination framework. A plausible implication is that its resolution is well matched to decoding-time intervention, because the inference target is aligned to the unit actually produced by the autoregressive model.

2. Confounding structure and the failure of coarse attention signals

A core contribution of HaloProbe is the claim that global attention averages are not reliable hallucination detectors. Let $c_t$1 denote averaged image attention for token $c_t$2. The method identifies two external variables as hidden confounders: token position and object repetition or occurrence. These variables affect both attention values and hallucination labels, so aggregated attention trends can be misleading (Zohrabi et al., 7 Apr 2026).

For token position, the paper analyzes conditional expectations

$c_t$3

Empirically, for most positions $c_t$4,

$c_t$5

Conditioned on position, hallucinated tokens can therefore appear equally or even more attended than correct tokens. However, after marginalizing over position,

$c_t$6

the trend reverses: $c_t$7 This reversal is the paper’s explicit instance of Simpson’s paradox.

A similar confounding effect arises from repetition. First mentions tend to receive more attention, while hallucinated objects are more likely to be first mentions: $c_t$8 If attention is averaged without conditioning on occurrence, structurally distinct regimes are mixed together. HaloProbe’s critique is therefore not that attention is useless, but that naively aggregated attention is entangled with caption-structure variables and with severe class imbalance. The resulting shortcut signals can yield deceptively strong aggregate statistics while failing to isolate internal grounding evidence.

3. Bayesian factorization and posterior recovery

HaloProbe factorizes the inference problem into an internal evidence model and an external prior model. The main classifier is trained on a balanced distribution and is defined as

$c_t$9

Because the balanced training set is constructed so that

$y(c_t) \in \{0,1\},$0

the classifier does not directly learn the natural posterior. Instead, it learns a balanced posterior that can be interpreted as an internal likelihood-ratio surrogate (Zohrabi et al., 7 Apr 2026).

Under the balanced distribution, Bayes’ rule gives

$y(c_t) \in \{0,1\},$1

Since the balanced prior is $y(c_t) \in \{0,1\},$2 for both classes,

$y(c_t) \in \{0,1\},$3

Therefore,

$y(c_t) \in \{0,1\},$4

which implies the likelihood ratio

$y(c_t) \in \{0,1\},$5

A separate model estimates the natural prior from external features alone: $y(c_t) \in \{0,1\},$6 Combining the balanced internal estimator with the learned prior yields the true posterior

$y(c_t) \in \{0,1\},$7

or equivalently,

$y(c_t) \in \{0,1\},$8

This posterior is the final HaloProbe score.

4. Balanced training as a mechanism for isolating internal evidence

The balanced training scheme is one of HaloProbe’s defining design choices. If the detector is trained directly on the natural data distribution, it can exploit simple correlations: early positions are often correct, first occurrences behave differently, and correct tokens are more common. In that regime, the detector may learn caption-structure shortcuts rather than visual grounding evidence (Zohrabi et al., 7 Apr 2026).

HaloProbe addresses this by balancing the training data with respect to the label conditioned on external features. Operationally, for each $y(c_t) \in \{0,1\},$9, the classes are upsampled so that

$y(c_t)=1$0

This forces the main estimator $y(c_t)=1$1 to focus on internal signals. The external information is not discarded; it is relocated into the prior model $y(c_t)=1$2, which is trained on the natural, imbalanced distribution.

This decomposition gives the method a clear interpretation. The balanced classifier answers, in effect, what the model’s internal signals indicate once class-imbalance bias tied to $y(c_t)=1$3 is removed. The prior model answers what the caption structure alone suggests under the natural data distribution. HaloProbe then recombines these components by Bayes’ rule. This suggests that the method is intended less as an attention heuristic than as a calibrated posterior-reconstruction procedure.

5. HaloProbe-guided mitigation and non-invasive decoding

HaloProbe is used არა მხოლოდ as a detector but also as a mitigation module. Its mitigation strategy is explicitly non-invasive: it does not modify model internals or intervene on attention dynamics. Instead, it supplies an external scoring signal that ranks decoding candidates or edits outputs after generation (Zohrabi et al., 7 Apr 2026).

In hallucination-aware beam search, at decoding step $y(c_t)=1$4, the beam contains candidates

$y(c_t)=1$5

Each candidate is scored using the number of hallucinated mentions $y(c_t)=1$6, the number of correct mentions $y(c_t)=1$7, the summed hallucination confidence $y(c_t)=1$8, and the summed correctness confidence $y(c_t)=1$9. The paper defines

$y(c_t)=0$0

Lower $y(c_t)=0$1 is preferred, and $y(c_t)=0$2 controls the trade-off between suppressing hallucinations and preserving object coverage. The decoding algorithm itself remains standard; HaloProbe only ranks candidates externally.

For greedy or nucleus decoding, the paper also describes a post-processing pipeline. A caption is first generated, HaloProbe then detects hallucinated objects, the detected objects are marked with a special symbol such as $y(c_t)=0$3, and the marked caption is sent to an external LLM editor with instructions to remove only the marked objects and make minimal grammatical fixes. This is again non-invasive: mitigation occurs through output filtering and editing rather than parameter-level intervention.

The contrast with intervention-based approaches is central. Attention intervention methods alter internal decoding dynamics, whereas HaloProbe uses token-level hallucination probabilities as an external control signal.

6. Empirical performance and comparative results

On LLaVA-1.5-7B, HaloProbe is reported to outperform prior detectors including IC, UT, EAZY, and DIML. The reported detection results are Accuracy 90.00, AUROC 93.50, Precision 92.50, Recall 95.80, and F1 94.10. The paper also reports strong transfer to other LVLMs: Shikra achieves Accuracy 90.2 and F1 94.4, while MiniGPT-4 achieves Accuracy 91.0 and F1 95.0 (Zohrabi et al., 7 Apr 2026).

Mitigation results are reported using the CHAIR metrics. For LLaVA-1.5 under greedy decoding, the baseline is $y(c_t)=0$4 and $y(c_t)=0$5, while HaloProbe plus post-process yields $y(c_t)=0$6 and $y(c_t)=0$7. Under beam search, the baseline is $y(c_t)=0$8 and $y(c_t)=0$9, while HaloProbe plus beam yields $p\big(y(c_t)\mid x_i(c_t),x_e(c_t)\big),$0 and $p\big(y(c_t)\mid x_i(c_t),x_e(c_t)\big),$1. Similar patterns are reported for Shikra and MiniGPT-4.

The paper further argues that HaloProbe preserves utility and fluency better than intervention-based methods. Attention intervention is said to increase redundancy and repetition sharply and to cause longer captions, more repetition, lower lexical diversity, and larger repeated spans. HaloProbe’s advantage is framed as reducing hallucinations without harming fluency as much, because the underlying LVLM decoding dynamics are not directly perturbed. The method is also reported to transfer to newer models such as Qwen3-VL and InternVL3.5, where it continues to improve CHAIR scores.

A plausible implication is that HaloProbe’s empirical strength derives from decoupling two tasks that earlier methods conflated: extracting internal evidence and modeling the base-rate structure of caption generation. The reported gains in both detection and mitigation are consistent with that interpretation.

7. Limitations, assumptions, and terminological scope

HaloProbe assumes that object mentions can be identified and aligned to tokens; the paper states that this is done using CHAIR and token-word alignment. It also assumes that the external features $p\big(y(c_t)\mid x_i(c_t),x_e(c_t)\big),$2 are informative and that their train-test distribution is reasonably aligned. The Bayesian factorization further depends on the balanced training setup, because that construction underwrites the likelihood-ratio interpretation of $p\big(y(c_t)\mid x_i(c_t),x_e(c_t)\big),$3 (Zohrabi et al., 7 Apr 2026).

The practical constraints are explicit. Beam-search mitigation increases compute with beam size, and the post-processing route depends on an external editor LLM. The method is designed for token-level object hallucinations; extending it to attributes or relations is described as possible, but it would require redefining the prediction unit and feature design. The paper also emphasizes a methodological caveat: coarse attention averages are unreliable, but fine-grained layer- and head-level attention remains useful. In evaluation, raw accuracy is treated cautiously because hallucination is highly class-imbalanced; AUROC and F1 are presented as more informative.

The name itself is narrow in scope within the supplied literature. In the LVLM setting, “HaloProbe” specifically denotes the Bayesian framework described above (Zohrabi et al., 7 Apr 2026). Other supplied papers discuss conceptually unrelated systems that are explicitly named differently, including a “portable autonomous probing system (APS)” for collecting water-quality measurements at depth from small autonomous surface vehicles (Huang et al., 2021) and a “directional enhanced probe” for side-illumination tip-enhanced spectroscopy (Shen et al., 2015). This suggests that, in current arXiv usage, HaloProbe is not a generic label for halo-related probes, but the specific designation of a Bayesian hallucination detector and mitigation method for vision-LLMs.