---
title: 'HalluShift++: Hallucination Detection & QMC Shifts'
url: https://www.emergentmind.com/topics/hallushift
type: topic
---

# HalluShift++: Hallucination Detection & QMC Shifts

HalluShift++ refers to two independent and domain-specific methodologies: (1) a hierarchical, internal representation–shift-based approach for hallucination detection in multimodal large language models (MLLMs) [2512.07687], and (2) a component-by-component method for constructing $p$-adically shifted Halton quasi-Monte Carlo rules in weighted anchored Sobolev spaces [1501.07377]. This entry focuses on the technical underpinnings, algorithms, architectures, and empirical properties of both methods as presented in their original sources.

## 1. Internal Representation–Shift Metrics in MLLMs

HalluShift++ hypothesizes that hallucination in MLLMs is marked by measurable shifts in hidden-state and attention-weight distributions across transformer decoder layers and token positions. For each token $t$ and layer $l$, let $h_t^{(l)} \in \mathbb{R}^d$ denote the hidden state and $A_t^{(l)} \in \mathbb{R}^{H \times N}$ the cross-attention tensor.

### Core Features

- **Cross-Layer Consistency Features (LCF):** Early ($\ell_e$) and late ($\ell_\ell$) layer hidden states are compared via cosine similarity,
  $$
  c_t = \frac{\cos\left(h_t^{(\ell_e)}, h_t^{(\ell_\ell)}\right)+1}{2},\quad f_\text{consistency}=c_t,\quad f_\text{inconsistency}=1-c_t,
  $$
  where lower $c_t$ values indicate greater representational drift across the model depth under hallucination.

- **Attention Concentration Features (ACF):** For each of the final three decoder layers, the Gini coefficient of sorted, flattened cross-attention weights $a_t^{(\ell)}$ is computed,
  $$
  G_t^{(\ell)} = 2\sum_{i=1}^n \frac{\sum_{j=1}^i s_j}{n\sum_{i=1}^n s_i} - 1
  $$
  with $f_{\text{ACF-mean}} = \text{mean}_{\ell=L-3..L-1}|G_t^{(\ell)}|$ and $f_{\text{ACF-std}}$ analogously.

- **Perplexity and Confidence Features:** For each token,
  $$
  p_\text{max}^{(t)} = \max(p_t),\quad \text{ppl}_t = 1 / p_\text{max}^{(t)},
  $$
  then, the mean, standard deviation, trend (slope), mean confidence, and fraction of low-confidence tokens ($p_\text{max}^{(t)}<0.5$) are collected.

- **Token-Pattern Features:** Unique and bigram repetition ratios, as well as normalized unique tokens, are computed to characterize diversity. Altogether, HalluShift++ aggregates 74 features per token.

## 2. MLLM Architecture and Activation Extraction

HalluShift++ is model-agnostic but evaluated on multiple MLLMs employing:

- **Vision Encoder:** Frozen, e.g., ViT or ResNet, providing dense embeddings.
- **Adapter/Q-Former:** Projects vision features into $M$ visual queries.
- **Text Decoder:** Large LLM (e.g., LLaMA-3.2-11B), with $L\approx32$ transformer layers and cross-modal fusion at every decoder layer via cross-attention.

During greedy decoding (max $T=64$ tokens), the approach records all $h_t^{(l)}$, $A_t^{(l)}$, and logits $\ell_t$ needed to compute feature vectors for each output token.

## 3. Hierarchical Hallucination-Scoring Pipeline

The core scoring algorithm, as implemented, proceeds through semantic chunk extraction, chunk-level feature aggregation, and hierarchical hallucination classification. The operational pseudocode is:

```python
def HalluShiftPP(image, prompt, ground_truth):
    y = model.generate(image, prompt)  # Greedy decoding, T steps
    chunks = SemanticChunkExtraction(y)
    S = [0, 0, 0, 0]  # S_Correct, S_Category, S_Attribute, S_Relation
    
    for c_i in chunks:
        T_i = token_positions_for_chunk(c_i)
        F_c = aggregate_features_over_chunk(T_i)
        phi_c = [CWC, CRP, CPI]  # chunk-word count, rel. position, per-image stats
        s_i = f_theta([F_c, phi_c])  # 3-layer NN, R^4 outputs
        pi_i = softmax(s_i)  # categorical prob. for {Correct, Cat, Attr, Rel}
        # Collect pi_i for aggregation
        
    for y_j in [Category, Attribute, Relation]:
        S_j = mean([pi_i[y_j] for all i])
    S_Correct = 1 - sum(S_j for j in [Category, Attribute, Relation])
    return [S_Correct, S_Category, S_Attribute, S_Relation]
```

Semantic chunks are labeled using a hierarchical ground-truth matching protocol (object $\rightarrow$ attribute $\rightarrow$ relation). Scores are aggregated per category to yield a four-dimensional hallucination vector, supporting both soft and hard assignment.

## 4. Experimental Design and Empirical Performance

### Datasets and Annotation

- **Datasets:** MS-COCO validation set (≈40,000 images, 5 human captions each); LLaVA vision–language instructions.
- **Ground Truth:** Manual chunk-level verification with hierarchical matching to ground-truth captions/annotations.

### Metrics and Training

- Primary metric: AUC-ROC; also token-level precision, recall, F1; per-class ROC curves for taxonomy adherence.
- Class rebalancing: SMOTE oversampling and weighted cross-entropy for classes with few attribute hallucinations (<5%).
- Training: 3-layer membership network with AdamW, learning rate $1\times10^{-4}$, cosine annealing, early stopping on validation AUC.

### Results Table (excerpt)

| Method                | MS-COCO AUC-ROC | Precision | F1    |
|-----------------------|-----------------|-----------|-------|
| ITI (external LLM)    | 49.8%           | 48.3%     | 57.2% |
| HalluShift            | 52.1%           | 52.1%     | 60.5% |
| HalluShift++          | 86.1%           | 77.4%     | 62.6% |

Across eight MLLMs and two datasets, HalluShift++ improved AUC-ROC by 27.7–64.1% over prior external and internal evaluation approaches, reaching over 90% in large models. On text-only QA (TruthfulQA), HalluShift++ reached 92.7% AUC-ROC, exceeding HalluShift by 2–3 points. All results are significant by paired $t$-test ($p<0.01$) [2512.07687].

## 5. Limitations and Future Directions in Hallucination Detection

### Limitations

- Requires access to all hidden states, cross-attention, and logits—prohibitive in closed-API models.
- Overhead: 74 features per token/chunk, leading to storage and computational costs.
- Semantic chunking may miss nested or global scene-level hallucinations, and rare attributes remain challenging despite oversampling.
- Chunk extraction fails on highly abstract, metaphorical, or extremely short captions.
- Domain adaptation is limited; strong visual-language fusion priors may not generalize (e.g., to medical or satellite imagery).

### Future Directions

- Incorporate anomaly detection on raw visual-encoder outputs (early fusion).
- Extend chunking to scene graphs for relational hallucination detection.
- Explore contrastive pretraining of the hallucination head.
- Devise probing methods for closed-weight APIs (e.g., Gini from surprisals).
- Study streaming/video settings to leverage temporal consistency for hallucination drift detection [2512.07687].

## 6. Component-by-Component Construction of Shifted Halton Rules

An unrelated but historically prior HalluShift++ construct appears in quasi-Monte Carlo integration [1501.07377]. Here, HalluShift++ denotes an explicit component-by-component (CBC) search for optimal $\boldsymbol{p}$-adically shifted Halton sequences minimizing worst-case error in weighted anchored Sobolev spaces.

### Definitions

- Let $H_{s, \boldsymbol{\gamma}}$ be a weighted anchored Sobolev space with reproducing kernel
  $$
  K_{s, \gamma}(x, y) = \prod_{j=1}^{s} \left[1 + \gamma_j \min(1-x_j, 1-y_j)\right].
  $$
- Points are shifted Halton nodes:
  $x_{n, \sigma, j} = \varphi_{p_j}\big(\varphi_{p_j}^{-1}(x_{n,j}) +_{\mathbb{Z}_{p_j}} \varphi_{p_j}^{-1}(\sigma_j)\big)$,
  with $m_j$ such that $p_j^{m_j}>N$, and $\sigma_j$ taken from a finite candidate set $Q(p_j^{m_j})$.

### Algorithm Outline

The CBC algorithm proceeds dimension by dimension:

1. For each $j=1,\ldots,s$, set $m_j = \min\{m: p_j^m > N\}$ and $Q_j = \{k/p_j^{m_j}: 0 \leq k < p_j^{m_j}\}$.
2. For $d$ from 1 to $s$, select $\sigma^*_d = \arg\min_{t\in Q_d} e_{N,d}^2(\sigma^*_1,\ldots, \sigma^*_{d-1}, t)$, where $e_{N,d}^2$ is given by
   $$
   e_{N,d}^2(\sigma_1,\ldots,\sigma_d) = A_d - \frac{2}{N}\sum_{n=0}^{N-1}\prod_{j=1}^d [1+\gamma_j(1-x_{n,\sigma,j})] + \frac{1}{N^2}\sum_{n,k=0}^{N-1}\prod_{j=1}^d [1+\gamma_j\min(1-x_{n,\sigma,j},1-x_{k,\sigma,j})].
   $$
3. Repeat until $\sigma^* = (\sigma^*_1, \ldots, \sigma^*_s)$ is chosen.

### Complexity and Practical Guidance

- Naive runtime is $O(s^2N^3)$, but with transforms and precomputed tables, this can be significantly reduced.
- Weights $\gamma_j$ should decay to reflect dimension/smoothness: a typical choice is $\gamma_j = j^{-\alpha}$ for $\alpha>1$, ensuring summability.
- For practical $N$, compute and store radical inverses $\varphi_{p_j}(n)$ and their $p$-adic preimages.
- Worked examples illustrate explicit shifts selected and the resulting worst-case error.

## 7. Distinctions and Cross-Domain Usage

The two approaches sharing the HalluShift++ moniker are unrelated in technical approach and domain:

- In MLLMs, HalluShift++ denotes hierarchical hallucination detection via internal feature shifts [2512.07687].
- In QMC integration, HalluShift++ refers to a CBC algorithm for optimal $p$-adic shifts in Halton sequences [1501.07377].

The identical label is a result of independent proposal and does not indicate methodological overlap.

Source: https://www.emergentmind.com/topics/hallushift