HalluShift++: Hallucination Detection & QMC Shifts

• HalluShift++ is a dual-purpose framework encompassing both hierarchical hallucination detection in MLLMs and a CBC algorithm for p-adic shifted Halton sequences in QMC settings.
• The MLLM approach utilizes cross-layer consistency, attention concentration, and token-pattern features to effectively identify and mitigate hallucinations.
• The QMC component constructs optimally shifted Halton sequences to minimize worst-case error in weighted Sobolev spaces, enhancing computational efficiency.

HalluShift++ refers to two independent and domain-specific methodologies: (1) a hierarchical, internal representation–shift-based approach for hallucination detection in multimodal LLMs (MLLMs) (Nath et al., 8 Dec 2025), and (2) a component-by-component method for constructing $p$-adically shifted Halton quasi-Monte Carlo rules in weighted anchored Sobolev spaces (Kritzer et al., 2015). This entry focuses on the technical underpinnings, algorithms, architectures, and empirical properties of both methods as presented in their original sources.

1. Internal Representation–Shift Metrics in MLLMs

HalluShift++ hypothesizes that hallucination in MLLMs is marked by measurable shifts in hidden-state and attention-weight distributions across transformer decoder layers and token positions. For each token $t$ and layer $l$, let $h_t^{(l)} \in \mathbb{R}^d$ denote the hidden state and $A_t^{(l)} \in \mathbb{R}^{H \times N}$ the cross-attention tensor.

Core Features

• Cross-Layer Consistency Features (LCF): Early ($\ell_e$) and late ($\ell_\ell$) layer hidden states are compared via cosine similarity,

$$c_t = \frac{\cos\left(h_t^{(\ell_e)}, h_t^{(\ell_\ell)}\right)+1}{2},\quad f_\text{consistency}=c_t,\quad f_\text{inconsistency}=1-c_t,$$

where lower $c_t$ values indicate greater representational drift across the model depth under hallucination.

• Attention Concentration Features (ACF): For each of the final three decoder layers, the Gini coefficient of sorted, flattened cross-attention weights $a_t^{(\ell)}$ is computed,

$$G_t^{(\ell)} = 2\sum_{i=1}^n \frac{\sum_{j=1}^i s_j}{n\sum_{i=1}^n s_i} - 1$$

with $$f_{\text{ACF-mean}} = \text{mean}_{\ell=L-3..L-1}|G_t^{(\ell)}|$$ and $f_{\text{ACF-std}}$ analogously.

• Perplexity and Confidence Features: For each token,

$$p_\text{max}^{(t)} = \max(p_t),\quad \text{ppl}_t = 1 / p_\text{max}^{(t)},$$

then, the mean, standard deviation, trend (slope), mean confidence, and fraction of low-confidence tokens ($p_\text{max}^{(t)}<0.5$) are collected.

• Token-Pattern Features: Unique and bigram repetition ratios, as well as normalized unique tokens, are computed to characterize diversity. Altogether, HalluShift++ aggregates 74 features per token.

2. MLLM Architecture and Activation Extraction

HalluShift++ is model-agnostic but evaluated on multiple MLLMs employing:

• Vision Encoder: Frozen, e.g., ViT or ResNet, providing dense embeddings.
• Adapter/Q-Former: Projects vision features into $M$ visual queries.
• Text Decoder: Large LLM (e.g., LLaMA-3.2-11B), with $L\approx32$ transformer layers and cross-modal fusion at every decoder layer via cross-attention.

During greedy decoding (max $T=64$ tokens), the approach records all $h_t^{(l)}$, $A_t^{(l)}$, and logits $\ell_t$ needed to compute feature vectors for each output token.

3. Hierarchical Hallucination-Scoring Pipeline

The core scoring algorithm, as implemented, proceeds through semantic chunk extraction, chunk-level feature aggregation, and hierarchical hallucination classification. The operational pseudocode is:

def HalluShiftPP(image, prompt, ground_truth): y = model.generate(image, prompt) # Greedy decoding, T steps chunks = SemanticChunkExtraction(y) S = [0, 0, 0, 0] # S_Correct, S_Category, S_Attribute, S_Relation for c_i in chunks: T_i = token_positions_for_chunk(c_i) F_c = aggregate_features_over_chunk(T_i) phi_c = [CWC, CRP, CPI] # chunk-word count, rel. position, per-image stats s_i = f_theta([F_c, phi_c]) # 3-layer NN, R^4 outputs pi_i = softmax(s_i) # categorical prob. for {Correct, Cat, Attr, Rel} # Collect pi_i for aggregation for y_j in [Category, Attribute, Relation]: S_j = mean([pi_i[y_j] for all i]) S_Correct = 1 - sum(S_j for j in [Category, Attribute, Relation]) return [S_Correct, S_Category, S_Attribute, S_Relation]

Semantic chunks are labeled using a hierarchical ground-truth matching protocol (object $\rightarrow$ attribute $\rightarrow$ relation). Scores are aggregated per category to yield a four-dimensional hallucination vector, supporting both soft and hard assignment.

4. Experimental Design and Empirical Performance

Datasets and Annotation

• Datasets: MS-COCO validation set (≈40,000 images, 5 human captions each); LLaVA vision–language instructions.
• Ground Truth: Manual chunk-level verification with hierarchical matching to ground-truth captions/annotations.

Metrics and Training

• Primary metric: AUC-ROC; also token-level precision, recall, F1; per-class ROC curves for taxonomy adherence.
• Class rebalancing: SMOTE oversampling and weighted cross-entropy for classes with few attribute hallucinations (<5%).
• Training: 3-layer membership network with AdamW, learning rate $1\times10^{-4}$, cosine annealing, early stopping on validation AUC.

Results Table (excerpt)

Method	MS-COCO AUC-ROC	Precision	F1
ITI (external LLM)	49.8%	48.3%	57.2%
HalluShift	52.1%	52.1%	60.5%
HalluShift++	86.1%	77.4%	62.6%

Across eight MLLMs and two datasets, HalluShift++ improved AUC-ROC by 27.7–64.1% over prior external and internal evaluation approaches, reaching over 90% in large models. On text-only QA (TruthfulQA), HalluShift++ reached 92.7% AUC-ROC, exceeding HalluShift by 2–3 points. All results are significant by paired $t$-test ($p<0.01$) (Nath et al., 8 Dec 2025).

5. Limitations and Future Directions in Hallucination Detection

Limitations

• Requires access to all hidden states, cross-attention, and logits—prohibitive in closed-API models.
• Overhead: 74 features per token/chunk, leading to storage and computational costs.
• Semantic chunking may miss nested or global scene-level hallucinations, and rare attributes remain challenging despite oversampling.
• Chunk extraction fails on highly abstract, metaphorical, or extremely short captions.
• Domain adaptation is limited; strong visual-language fusion priors may not generalize (e.g., to medical or satellite imagery).

Future Directions

• Incorporate anomaly detection on raw visual-encoder outputs (early fusion).
• Extend chunking to scene graphs for relational hallucination detection.
• Explore contrastive pretraining of the hallucination head.
• Devise probing methods for closed-weight APIs (e.g., Gini from surprisals).
• Study streaming/video settings to leverage temporal consistency for hallucination drift detection (Nath et al., 8 Dec 2025).

6. Component-by-Component Construction of Shifted Halton Rules

An unrelated but historically prior HalluShift++ construct appears in quasi-Monte Carlo integration (Kritzer et al., 2015). Here, HalluShift++ denotes an explicit component-by-component (CBC) search for optimal $\boldsymbol{p}$-adically shifted Halton sequences minimizing worst-case error in weighted anchored Sobolev spaces.

Definitions

• Let $H_{s, \boldsymbol{\gamma}}$ be a weighted anchored Sobolev space with reproducing kernel

$$K_{s, \gamma}(x, y) = \prod_{j=1}^{s} \left[1 + \gamma_j \min(1-x_j, 1-y_j)\right].$$

• Points are shifted Halton nodes: $$x_{n, \sigma, j} = \varphi_{p_j}\big(\varphi_{p_j}^{-1}(x_{n,j}) +_{\mathbb{Z}_{p_j}} \varphi_{p_j}^{-1}(\sigma_j)\big)$$, with $m_j$ such that $p_j^{m_j}>N$, and $\sigma_j$ taken from a finite candidate set $Q(p_j^{m_j})$.

Algorithm Outline

The CBC algorithm proceeds dimension by dimension:

1. For each $j=1,\ldots,s$, set $m_j = \min\{m: p_j^m > N\}$ and $Q_j = \{k/p_j^{m_j}: 0 \leq k < p_j^{m_j}\}$.
2. For $d$ from 1 to $s$, select $$\sigma^*_d = \arg\min_{t\in Q_d} e_{N,d}^2(\sigma^*_1,\ldots, \sigma^*_{d-1}, t)$$, where $e_{N,d}^2$ is given by

$$e_{N,d}^2(\sigma_1,\ldots,\sigma_d) = A_d - \frac{2}{N}\sum_{n=0}^{N-1}\prod_{j=1}^d [1+\gamma_j(1-x_{n,\sigma,j})] + \frac{1}{N^2}\sum_{n,k=0}^{N-1}\prod_{j=1}^d [1+\gamma_j\min(1-x_{n,\sigma,j},1-x_{k,\sigma,j})].$$

1. Repeat until $\sigma^* = (\sigma^*_1, \ldots, \sigma^*_s)$ is chosen.

Complexity and Practical Guidance

• Naive runtime is $O(s^2N^3)$, but with transforms and precomputed tables, this can be significantly reduced.
• Weights $\gamma_j$ should decay to reflect dimension/smoothness: a typical choice is $\gamma_j = j^{-\alpha}$ for $\alpha>1$, ensuring summability.
• For practical $N$, compute and store radical inverses $\varphi_{p_j}(n)$ and their $p$-adic preimages.
• Worked examples illustrate explicit shifts selected and the resulting worst-case error.

7. Distinctions and Cross-Domain Usage

The two approaches sharing the HalluShift++ moniker are unrelated in technical approach and domain:

• In MLLMs, HalluShift++ denotes hierarchical hallucination detection via internal feature shifts (Nath et al., 8 Dec 2025).
• In QMC integration, HalluShift++ refers to a CBC algorithm for optimal $p$-adic shifts in Halton sequences (Kritzer et al., 2015).

The identical label is a result of independent proposal and does not indicate methodological overlap.