HalluRNN: Mitigating Hallucinations in LVLMs

• The paper presents HalluRNN, an architecture-level modification that integrates a Dual-Gated Depth Propagation Unit to reduce representational drift in LVLMs.
• It fine-tunes only about 1% of parameters by inserting lightweight DG-DPU cells between Transformer layers, ensuring efficient and plug-and-play adaptation.
• Benchmark evaluations demonstrate improved hallucination metrics, highlighting that depth-wise recurrent reasoning stabilizes updates and enhances visual grounding.

HalluRNN is an architecture-level modification for large vision-LLMs (LVLMs) designed to mitigate hallucinations—specifically, textually plausible outputs that lack visual grounding. It introduces cross-layer recurrent reasoning into standard Transformer-based LVLMs through a specialized recurrent unit, enabling efficient, robust hallucination mitigation by fine-tuning only a small set of parameters. HalluRNN’s principal innovation lies in treating inter-layer transitions as a depthwise recurrent process mediated by the Dual-Gated Depth Propagation Unit (DG-DPU), a lightweight RNN cell inserted between all adjacent Transformer layers and shared across depths (Yu et al., 21 Jun 2025).

1. Architecture and Motivation

The baseline LVLM architecture consists of a vision encoder $V$ that extracts image features $v=V(I)$ and a LLM $L$ (e.g., LLaVA) that embeds the input prompt $x$ into $h_0$. A stack of $N$ Transformer layers $f_1,\ldots,f_N$ generates intermediate hidden states $h_i$, with the final state $h_N$ fed into a prediction head $\phi$ for next-token inference: $v=V(I)$0 Hallucinations in such decoders commonly result from representational drift, where hidden states progressively lose alignment with visual evidence over multi-step decoding.

HalluRNN addresses this challenge by treating the layerwise transitions $v=V(I)$1 as a recurrent process through depth. The core mechanism is the DG-DPU, an RNN cell that refines each layer’s update based on both the raw increment and a cross-layer memory. Crucially, the DG-DPU’s parameters are shared at every Transformer depth and constitute only about 1% of the total model parameters, enabling highly efficient (resource-light) adaptation.

2. Dual-Gated Depth Propagation Unit (DG-DPU)

The DG-DPU operates between every pair of Transformer layers, taking as input the raw increment $v=V(I)$2 and the recurrent memory $v=V(I)$3. Its outputs are a refined increment $v=V(I)$4 used to update the hidden state. The DG-DPU comprises two sequential gating mechanisms:

• Constraint Gate: Detects and attenuates abrupt differences between the raw increment and historical memory for coarse filtering of drift.

$v=V(I)$5

• Correction Gate: Learns fine-grained modulation between the new update and smoothed state.

$v=V(I)$6

The DG-DPU’s parameters $v=V(I)$7, $v=V(I)$8, and $v=V(I)$9 are globally shared across all depths. The updated hidden state is then: $L$0 This approach ensures layer-to-layer consistency and directly counters representational drift.

3. Integration and Training Procedure

HalluRNN leaves the underlying architecture (e.g., LLaVA-1.5 7B) and vision encoder completely frozen, training only the DG-DPU parameters ($L$1, $L$2, $L$3). The fine-tuning process uses 17,000 instruction examples from LLaVA-Instruct-150, relying solely on the standard next-token cross-entropy loss: $L$4 Optimization employs AdamW with a learning rate of $L$5, batch size 10, over three epochs on four RTX 4090 GPUs, requiring approximately three hours. No auxiliary loss or explicit regularizer for cross-layer consistency is introduced. Inference is realized by interposing DG-DPU refinement in each decoding step at every Transformer layer.

Integration Workflow

Component	Modification	Trainable
Vision Encoder	No change, frozen	No
Transformer Stack	Insert DG-DPU after each layer, share across depths	No
DG-DPU	Refines inter-layer transitions, tuned on instruction set	Yes ($L$61%)

4. Impact on Hallucination Metrics and Benchmarks

HalluRNN achieves consistently strong performance on multiple LVLM benchmarks, outperforming or closely matching more computationally intensive approaches:

Benchmark	Metric	Vanilla	HalluRNN	DAMO (best)
MME	Total (↑)	1490.69	1511.35	1514.26
POPE (Random)	Acc / F1 (↑)	87.1 / 88.1	89.05 / 90.18	—
POPE (Popular)	Acc / F1	80.36 / 83.43	85.28 / 87.41	—
POPE (Adv.)	Acc / F1	72.5 / 77.8	74.47 / 78.82	—
CHAIR	$L$7/$L$8 (↓)	50.8 / 14.2	33.4 / 11.2	—
MM-Vet	Total % (↑)	31.1	33.0	—
MMMU	Overall (↑)	34.2	34.6	—

Reductions in object hallucination rate (e.g., CHAIR) and improvements in adversarial and popular POPE settings indicate robust mitigation of visually-ungrounded outputs. Ablations replacing DG-DPU with a standard GRU result in a 4.6% drop in object hallucination accuracy on POPE (random), and removing either of the DG-DPU's gating mechanisms also degrades performance ($L$9 acc. w/o Constraint Gate, $x$0 w/o Correction Gate).

A plausible implication is that the adaptivity and depth-wise gating of DG-DPU are critical for both stability and cross-layer corrective behavior.

5. Mechanisms for Reducing Drift and Hallucination

Representational drift arises from the accumulation of spurious patterns within the Transformer stack, progressively weakening visual grounding. The DG-DPU’s Constraint Gate filters sudden discrepancies between current and accumulated depth memories, anchoring updates toward more stable representations. The Correction Gate further modulates this update, allowing the model to selectively leverage novel evidence or revert to stabilized historical information.

By tying DG-DPU weights across depths, the model learns a global criterion for regulating update dynamics at all layers. This shared structure contrasts with prior approaches that rely on fixed weighting, explicit re-accumulation, or hyperparameter tuning (e.g., DAMO, DeCO).

Visualizations reveal that, for questions where vanilla models lock into an erroneous answer after early layers, HalluRNN progressively shifts layerwise confidence via its gates—demonstrated in tasks like visual verification (e.g., “Is there a tree in the image?”), where the deeper DG-DPU-modulated representations correct earlier misclassifications.

6. Comparative Analysis and Transferability

HalluRNN’s architecture-level design distinguishes it from data-centric and decoding-based hallucination interventions, which often require either expensive end-to-end retraining or task-specific configuration. Only ≈1% of parameters are fine-tuned, resulting in minimal computation and strong plug-and-play transfer potential: DG-DPU weights trained on LLaVA yield consistent gains when transferred to similar model backbones (e.g., INF-MLLM1).

In ablation studies, adaptive DG-DPU gating yields consistently higher robustness relative to non-learnable or fixed-depth blending schemes. This suggests that depth-wise recurrent reasoning with shared parameterization is an efficient inductive bias for combating drift-induced hallucinations in large-scale multimodal models.