ChainMPQ: Interleaved Reasoning in LVLMs

• ChainMPQ is a training-free method for mitigating relation hallucinations in LVLMs by decomposing relational inference into visually grounded steps.
• It employs a three-module structure—text-guided attention enhancement, multi-perspective question construction, and an interleaved text-image reasoning chain—to improve reasoning accuracy.
• Empirical evaluations on benchmarks like MMRel and R-Bench demonstrate that ChainMPQ increases accuracy and precision, effectively reducing false-positive relation predictions.

ChainMPQ, short for Multi-Perspective Questions guided Interleaved Chain of Image and Text, is a training-free mitigation method for relation hallucinations in large vision-LLMs (LVLMs). It addresses the case in which a model identifies the relevant entities in an image but misstates the relationship between them, such as confusing standing on with riding. The method decomposes relational inference into a sequence of visually grounded steps and reuses both textual intermediate answers and visual attention memory across those steps. In the formulation reported in "ChainMPQ: Interleaved Text-Image Reasoning Chains for Mitigating Relation Hallucinations" (Wu et al., 7 Oct 2025), the procedure is built from three modules: text-guided attention enhancement, multi-perspective question construction, and an interleaved text-image reasoning chain.

1. Problem setting and motivation

ChainMPQ is motivated by the claim that hallucinations in LVLMs fall into three categories: object hallucination, attribute hallucination, and relation hallucination (Wu et al., 7 Oct 2025). In this taxonomy, object hallucination refers to failure to recognize whether an entity exists; attribute hallucination refers to incorrect properties such as color, shape, or size; relation hallucination refers to cases in which the entities are recognized but the relationship between them is predicted incorrectly.

The method is specifically targeted at relation hallucination because the paper argues that it is both prevalent and underexplored. Prior work is summarized as suggesting that relation errors account for nearly 40% of all hallucinations in LVLMs, while most mitigation methods have concentrated on object and attribute failures through preference optimization, contrastive decoding, and internal-layer edits (Wu et al., 7 Oct 2025). The paper further argues that relation hallucination is more reasoning-intensive than the other two categories because the model must not only localize entities but also infer how they interact, for example distinguishing left of from right of or holding from touching.

A central premise of ChainMPQ is that one-step inference is structurally fragile for relational questions. Existing methods are described as often expecting the model to identify objects and infer the relation simultaneously. ChainMPQ instead adopts the view that relational reasoning should proceed in stages: localize the entities, inspect the interaction, and then answer the original question. This suggests that relation hallucination is not treated merely as a decoding artifact, but as a failure of reasoning structure.

2. Core architecture and question decomposition

The first module, text-guided attention enhancement, begins by extracting subject and object keywords from the relational question using spaCy (Wu et al., 7 Oct 2025). The paper states that these keywords are usually two tokens, one for the subject and one for the object. Given image features $V \in \mathbb{R}^{M \times d_v}$ and keyword text features $X \in \mathbb{R}^{N \times d_t}$, the model applies cross-attention so that image tokens attend to the extracted keywords, thereby strengthening the image regions likely to correspond to the subject and object. The intended effect is to bias the visual representation toward subject/object regions before relational reasoning begins.

The second module, multi-perspective question construction, decomposes the original relational query into five complementary questions centered on the three core components of a relationship: subject $[S]$, object $[O]$, and relation $[R]$ (Wu et al., 7 Oct 2025). The five questions are:

1. Where is [S]?
2. Where is [O]?
3. What is [S] [R]? with the object masked
4. What is [R] [O]? with the subject masked
5. What is the relationship between [S] and [O]? with the relation masked

The first two subquestions are localization-oriented, while the remaining three are relation-oriented through masking. The paper gives the example “Does the dog chase a disc in the image?” and explains that the subquestions force separate inspection of the dog, the disc, and their interaction. According to the reported interpretation, this reduces reliance on language priors and creates a more stable inference path than a single direct relation query.

3. Interleaved text-image reasoning chain

The third module is the interleaved text-image reasoning chain, which is the mechanism that connects the five subquestions into a progressive reasoning process (Wu et al., 7 Oct 2025). The subquestions are not answered independently. Instead, they are fed to the LVLM sequentially, and each step contributes memory that conditions later steps in two forms:

$\mathcal{T} = \{(Q_1,A_1), (Q_2,A_2), \ldots\}$

$\mathcal{V} = \{M_3, M_4, M_5, \ldots\}$

Here, $\mathcal{T}$ is textual memory, storing accumulated question-answer pairs, and $\mathcal{V}$ is visual memory, storing attention-derived masks.

For Questions 1 and 2, the subject and object localization questions are answered directly using the enhanced visual tokens $V'$, with no prior context. Their answers are appended to textual memory. For Questions 3 to 5, the model uses both accumulated text context and visual memory. The paper computes attention over keyword tokens from the last $X \in \mathbb{R}^{N \times d_t}$0 decoder layers, with the appendix specifying $X \in \mathbb{R}^{N \times d_t}$1 last decoder layers:

$X \in \mathbb{R}^{N \times d_t}$2

The aggregated attention is converted into an adaptive visual memory via an entropy-based top-$X \in \mathbb{R}^{N \times d_t}$3 rule. The paper sets $X \in \mathbb{R}^{N \times d_t}$4, described as about 10% of visual patches. The idea is explicit: if attention is concentrated, fewer tokens are retained; if attention is diffuse, more tokens are retained. The selected tokens define a normalized mask $X \in \mathbb{R}^{N \times d_t}$5, which is then reused as a bias in later attention.

This reuse is weighted by a confidence-based coefficient, with the paper setting $X \in \mathbb{R}^{N \times d_t}$6. The attention bias for later steps is proportional to the confidence of earlier answers, so high-confidence intermediate answers exert stronger guidance on subsequent relational reasoning. For multi-round history, earlier masks are fused by a weighted average rather than using only the most recent mask. The resulting chain is “interleaved” in the precise sense that text influences future visual attention and visual attention shapes future text generation. The paper’s key insight is that an LVLM should retain not only what it said, but also where it looked.

4. Algorithmic procedure and operational characteristics

The appendix summarizes ChainMPQ as a structured, training-free pipeline (Wu et al., 7 Oct 2025). In prose, the procedure is:

1. extract subject/object keywords from the original question;
2. encode the image into visual tokens $X \in \mathbb{R}^{N \times d_t}$7 and the keywords into text tokens $X \in \mathbb{R}^{N \times d_t}$8;
3. apply cross-attention to obtain enhanced visual tokens $X \in \mathbb{R}^{N \times d_t}$9;
4. decompose the original question into the five subquestions;
5. initialize textual and visual memories as empty;
6. answer the first two localization questions directly using $[S]$0;
7. for each later question, compute aggregated attention over the last $[S]$1 decoder layers, select top-$[S]$2 visual tokens using entropy-based adaptive $[S]$3, build a bias mask $[S]$4, answer using $[S]$5, textual memory, and the mask, and append the resulting answer and mask to memory;
8. answer the original question using the accumulated multimodal context.

The method is training-free in the strict sense reported by the paper: it requires no fine-tuning or extra supervised learning. This is an important point of scope. ChainMPQ is not a new LVLM architecture and not a retraining procedure; it is an inference-time mitigation strategy that wraps an existing LVLM with a structured questioning and memory mechanism.

A common misconception would be to interpret ChainMPQ as a general hallucination remedy. The paper instead positions it as a method for relation hallucination specifically. It is evaluated on relation-hallucination benchmarks rather than on benchmarks focused mainly on object hallucinations, and its core design choices—entity localization, relation masking, and multimodal memory transfer—are explicitly tuned to relational inference rather than general factual correction.

5. Empirical evaluation

The reported experiments evaluate ChainMPQ on two open-source LVLMs, LLaVA-1.5-7B and InstructBLIP-7B, and on two relation-hallucination benchmarks, MMRel and R-Bench (Wu et al., 7 Oct 2025). The paper notes that benchmarks such as POPE and CHAIR focus mainly on object hallucinations and are therefore not sufficient for this task. Baselines include Vanilla LVLM, Constraint-Aware Prompting / Prompting, and Detect-then-Calibrate; the baselines were reimplemented in the same setup for fair comparison.

The main accuracy results are as follows:

Setting	Best baseline accuracy	ChainMPQ accuracy
MMRel, LLaVA-1.5	63.50	65.20
MMRel, InstructBLIP	64.52	65.14
R-Bench, LLaVA-1.5	75.86	76.04
R-Bench, InstructBLIP	73.65	75.86

On MMRel, LLaVA-1.5 improves from 59.02 for the vanilla model to 65.20 for ChainMPQ, with Precision 64.75 and F1 71.21. On the same benchmark, InstructBLIP improves from 57.58 to 65.14, with Precision 64.12 and F1 74.12 (Wu et al., 7 Oct 2025).

On R-Bench, LLaVA-1.5 rises from 71.23 for the vanilla model to 76.04 for ChainMPQ; its Precision rises to 72.03, which the paper states is about 4.17 points better than the best baseline, and F1 reaches 81.54. InstructBLIP rises from 69.31 to 75.86, with Precision 70.59 and F1 81.12 (Wu et al., 7 Oct 2025).

The paper identifies precision as especially important because it indicates fewer false-positive relation predictions. On that reading, the gains are not limited to overall accuracy; they are also consistent with a reduction in relation hallucinations in the stricter sense of erroneous relational assertions.

6. Ablation studies, sensitivity analysis, and interpretive findings

The ablation study on MMRel with LLaVA-1.5 isolates the contribution of the three main modules (Wu et al., 7 Oct 2025). The full model achieves 65.20 accuracy. Removing text-guided attention enhancement yields 64.06, a drop of 1.14 points. Removing multi-perspective questions yields 61.52, a drop of 3.68 points. Removing interleaved reasoning yields 62.12, a drop of 3.08 points.

These ablations support a clear ranking of component importance within the reported experiments. The paper states that multi-perspective question decomposition is the most important component, followed closely by interleaved multimodal memory transfer, while attention enhancement provides an additional but smaller boost. The paper also notes that even the ablated variants outperform the vanilla baseline, suggesting that each module contributes positively.

The sensitivity analysis varies $[S]$6 and $[S]$7. The tested settings are $[S]$8 at roughly 5%, 10%, 30%, and 50% of image patches, and $[S]$9. The best performance occurs at $[O]$0 and $[O]$1 (Wu et al., 7 Oct 2025). The accompanying interpretation is explicit: too large a $[O]$2 introduces irrelevant visual noise, too small a $[O]$3 omits useful context, too large a $[O]$4 overlocks the model onto previous memories, and too small a $[O]$5 weakens the guidance effect.

The case studies are consistent with the benchmark and ablation findings. For an action relation, the example “Does a man stand on a surfboard?” is answered incorrectly by a baseline that confuses standing on with riding, whereas ChainMPQ first localizes the man and surfboard and then infers the correct relation. For a spatial relation, the example “Is a chair to the left of a trash bin?” is answered incorrectly by a baseline, while ChainMPQ localizes the objects and determines that the chair is actually to the right of the bin. The paper reports that the attention maps become more concentrated on relevant regions and suppress irrelevant background.

7. Limitations, failure modes, and broader implications

The paper identifies several limitations of ChainMPQ (Wu et al., 7 Oct 2025). First, attention is only a proxy for reasoning. The method assumes that attention maps reflect useful evidence, but attention may not fully capture true causal reasoning. Second, spatial granularity remains challenging because visual tokens may not align cleanly with real object boundaries; the paper notes lower performance in the spatial category of MMRel. Third, inference cost is higher: because multiple guiding questions are asked per original question, inference is about 4× slower than the base model, although the method remains training-free and lightweight compared with retraining.

The paper further implies that failures can occur when object boundaries are too fine-grained for patch-level attention, when the model over-relies on prior memories, or when the attention distribution is too diffuse or too concentrated. These are not presented as contradictions of the method’s premise, but as operating limits of an attention-mediated, multi-step inference procedure.

In broader methodological terms, ChainMPQ suggests a general principle for multimodal reasoning: decompose complex visual inference into sub-questions, accumulate structured memory across steps, and interleave text and vision rather than treating them as one-shot input-output (Wu et al., 7 Oct 2025). A plausible implication is that relation hallucination in LVLMs reflects not only insufficient grounding, but also insufficient procedural structure in inference. Within that interpretation, ChainMPQ is significant less as a standalone prompt engineering trick than as a concrete demonstration that relational faithfulness can be improved by staging the reasoning process and preserving both intermediate linguistic commitments and intermediate visual focus.