MCA-LLaVA: Manhattan Causal Attention

• The paper introduces MCA-LLaVA, a method that replaces raster-scan positional bias with a two-dimensional Manhattan distance formulation to mitigate hallucination in LVLMs.
• It restructures token interactions by employing Manhattan causal attention, ensuring balanced communication between instruction and image tokens.
• Experimental results demonstrate significant improvements on hallucination benchmarks like POPE and CHAIR while also maintaining strong overall multimodal performance.

Searching arXiv for MCA-LLaVA and closely related hallucination-reduction LVLM papers to ground the article with current references. MCA-LLaVA is a large vision-LLM (LVLM) method that targets hallucination by modifying positional modeling inside causal attention rather than by intervening only at decoding time or through additional alignment supervision. It is introduced in “MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-LLMs” (Zhao et al., 12 Jul 2025). The method identifies a mechanistic source of multimodal misalignment in Rotary Position Encoding (RoPE): under long-term decay, instruction tokens unevenly perceive image tokens in the flattened one-dimensional sequence, prioritizing tokens near the bottom-right region of the image because those tokens are positionally closer to the instruction tokens. MCA-LLaVA addresses this “image alignment bias” by replacing raster-scan-relative positional bias with a two-dimensional Manhattan-distance formulation and a corresponding Manhattan causal mask, thereby extending long-term decay from a one-dimensional sequence to two-dimensional, multi-directional spatial decay (Zhao et al., 12 Jul 2025). The name should not be conflated with “MC-LLaVA: Multi-Concept Personalized Vision-LLM,” which addresses multi-concept personalization rather than hallucination reduction (An et al., 2024).

1. Problem formulation and motivation

MCA-LLaVA begins from the observation that LLaVA-style systems encode image patches into tokens and concatenate them with instruction tokens in a causal Transformer. In that formulation, the visual input is processed as a long token sequence. RoPE is then used for positional modeling, and its long-term decay makes attention strength decrease as relative token distance grows (Zhao et al., 12 Jul 2025).

The paper argues that this behavior is suitable for text but problematic for images. Images are inherently two-dimensional, whereas RoPE operates over a one-dimensional raster-scan ordering. When image tokens are flattened row-by-row and instruction tokens are appended after them, later image tokens—typically corresponding to the bottom-right region—are closer in sequence distance to the instruction tokens than earlier tokens, which often correspond to the top-left region. As a result, instruction tokens attend more strongly to bottom-right image regions and more weakly to earlier image tokens (Zhao et al., 12 Jul 2025).

This phenomenon is termed image alignment bias. In the paper’s causal account, hallucination is linked to intrinsic multimodal misalignment inside the attention mechanism: uneven interaction between instruction tokens and image regions leads to incomplete visual understanding, which increases the likelihood of generating objects or attributes that are not present. This framing explicitly rejects the narrower view that hallucination is only a decoding-time issue or only a data/alignment issue (Zhao et al., 12 Jul 2025).

A common misconception is that hallucination in LVLMs is reducible to generation heuristics. MCA-LLaVA instead treats hallucination partly as a positional-geometry problem. This suggests that improving internal token interaction can change hallucination behavior even without redesigning the decoder or introducing a new data curation pipeline.

2. RoPE long-term decay and the origin of image alignment bias

The paper analyzes attention between an instruction query token $Q_i$ and an image key token $K_j$ under RoPE. RoPE applies a rotation matrix $R^d_{\theta,m}$ at position $m$:

$$R^d_{\theta,m} = \begin{pmatrix} \cos(m\theta_1) & -\sin(m\theta_1) & \cdots \ \sin(m\theta_1) & \cos(m\theta_1) & \cdots \ \vdots & \vdots & \ddots \end{pmatrix}, \qquad \theta_i = 10000^{-2(i-1)/d}$$

The attention score is written as:

$$Attn_{i,j} = \text{softmax}\left( \frac{Q_i^T (R^d_{\theta,i})^T R^d_{\theta,j} K_j}{\sqrt{d} } \right) = \text{softmax}\left( \frac{Q_i^T R^d_{\theta,(j-i)} K_j}{\sqrt{d} } \right)$$

In this form, attention depends on the relative distance $j-i$. As that distance increases, attention decays. In a multimodal sequence where image tokens appear before instruction tokens, image tokens closer to the end of the raster-scan are closer to the appended instruction tokens and therefore receive stronger attention (Zhao et al., 12 Jul 2025).

The paper characterizes the resulting information flow as dense in the lower-right region and sparse elsewhere. Crucially, it interprets this not as a learned semantic preference but as a mechanistic bias induced by one-dimensional positional modeling. The consequence is insufficient image-instruction interaction over substantial portions of the image grid, weakening multimodal alignment (Zhao et al., 12 Jul 2025).

This analysis is central to MCA-LLaVA because it redefines hallucination reduction as an architectural intervention on how positional distance is represented. A plausible implication is that any RoPE-based causal LVLM using flattened image tokens may inherit analogous biases unless its positional scheme is adapted to the two-dimensional structure of the visual input.

3. Manhattan Causal Attention

The core proposal is Manhattan Causal Attention (MCA), which preserves the causal sequence structure but changes how relative distance is defined for image-token relations. The standard RoPE relative distance is written as

$D_{RoPE}\{Q_i,K_j\} = \gamma(j) - \gamma(i)$

where $\gamma(\cdot)$ denotes the raster-scan index. MCA replaces this with a two-dimensional spatial model:

$$D_{Manhattan}\{Q_i,K_j\} = (x_j - x_i) + (y_j - y_i)$$

The paper then introduces Manhattan position assignment

$K_j$0

which yields

$K_j$1

This reformulation retains the RoPE-style relative-distance form while changing the meaning of distance from raster-scan separation to spatial Manhattan distance (Zhao et al., 12 Jul 2025).

The image token grid is reassigned into four partitions, with the four corners treated as origin points $K_j$2. Coordinates then grow inward along the grid axes from each corner. The stated purpose is to produce a symmetric coordinate layout in which Manhattan distance reflects spatial locality rather than scan order. The paper also notes that the number of unique Manhattan position indexes is reduced relative to raw raster-scan indexing, which can reduce the effective image-to-instruction positional gap and facilitate interaction (Zhao et al., 12 Jul 2025).

MCA extends long-term decay from a unidirectional one-dimensional process to multi-directional spatial decay. Instead of favoring tokens late in the raster-scan, it makes decay depend on two-dimensional locality in multiple directions on the image plane. This is meant to improve the instruction side’s perception of image tokens at different spatial locations and thereby alleviate image alignment bias (Zhao et al., 12 Jul 2025).

4. Attention scoring and Manhattan causal masking

MCA-LLaVA modifies both the positional index and the masking logic. The updated attention score is given as

$K_j$3

The model therefore remains formally compatible with RoPE-style attention, but the relative position term reflects two-dimensional spatial displacement rather than one-dimensional sequence distance (Zhao et al., 12 Jul 2025).

The masking component is the Manhattan causal mask. The paper distinguishes it from both the default causal mask and the CCA-LLaVA-style center-biased mask. Rather than allowing an image token to focus on all previous tokens in one-dimensional order, MCA structures the allowed attention so that image tokens focus on neighboring image tokens in four directions, while preserving locality and maintaining causality (Zhao et al., 12 Jul 2025).

The paper’s causal chain is explicit:

1. RoPE long-term decay over one-dimensional sequence order creates image alignment bias.
2. Image alignment bias causes uneven image-token information flow.
3. Uneven information flow weakens multimodal alignment.
4. Weak alignment leads to hallucinations.

Under this account, hallucination reduction follows from broadening and balancing image-token interaction rather than from post hoc output filtering. The information-flow visualizations described in the paper are consistent with that interpretation: MCA shows more balanced flow over the image than LLaVA1.5 and CCA-LLaVA, rather than a concentration in the lower-right region (Zhao et al., 12 Jul 2025).

5. Experimental evaluation

The evaluation spans hallucination-focused and general multimodal benchmarks. The hallucination benchmarks are POPE and CHAIR, with CHAIR reported at sentence level $K_j$4 and instance level $K_j$5. The general benchmarks are GQA, VizWiz, ScienceQA (SQA), MMBench / MMB, MMStar, VQAv2, and SEED variants $K_j$6, $K_j$7, and $K_j$8. The paper also evaluates on the hallucination subset of MME, with metrics for object existence, count, position, and color (Zhao et al., 12 Jul 2025).

Using LLaVA-1.5-7B as the baseline under greedy decoding, MCA-LLaVA reports the following gains on POPE and CHAIR:

Benchmark	Metric	Result
POPE	F1	86.0
POPE	Accuracy	86.5
CHAIR	$K_j$9	38.0
CHAIR	$R^d_{\theta,m}$0	10.9

The paper states that, relative to the baseline, POPE F1 improves by +6.7%, POPE accuracy improves by +6.7%, CHAIR sentence-level hallucination decreases by 9%, and CHAIR instance-level hallucination decreases by 2.9%. These results are described as being among the best in comparison with DoLa, ITI, VCD, OPERA, DOPRA, HALC, Less is More, CCA-LLaVA, TAME, and SID (Zhao et al., 12 Jul 2025).

On the MME hallucination subset, MCA-LLaVA achieves the highest total score in the reported table:

Metric	Score
Existence	190.00
Count	163.33
Position	126.67
Color	170.00
Total	650.00

The paper highlights that MCA improves the Count metric substantially compared with CCA-LLaVA, which it interprets as evidence of better global scene understanding and fewer counting hallucinations. On general benchmarks, MCA-LLaVA improves or matches performance on GQA, VizWiz, SQA, MMB, MMStar, VQAv2, and SEED benchmarks, with highlighted improvements of MMB: +1.5% and MMStar: +6.5% (Zhao et al., 12 Jul 2025).

These results matter because they suggest the method is not only optimizing for hallucination-specific benchmarks. The reported gains on general multimodal tasks support the paper’s claim that improved positional geometry can strengthen broader visual perception.

6. Ablations, implementation, and relation to adjacent work

The ablation study tests whether the coordinate reassignment itself matters. The paper compares raster-scan, reverse raster-scan, CCA, a variant of MCA with different origin settings, reverse MCA, and full MCA. The raster-scan baseline is reported as POPE 79.3 / 79.8 and CHAIR $R^d_{\theta,m}$1, $R^d_{\theta,m}$2. Full MCA gives the best or near-best reported results, namely POPE F1 86.0, POPE accuracy 86.5, CHAIR $R^d_{\theta,m}$3, and CHAIR $R^d_{\theta,m}$4 (Zhao et al., 12 Jul 2025).

The ablations support three claims made in the paper: the coordinate scheme matters; corner-origin Manhattan assignment is beneficial; and heuristic reindexing alone is not sufficient unless it is aligned with the two-dimensional distance model. Generalization experiments also apply the method to LLaVA-1.5-13B and InternVL-7B, where MCA continues to improve performance, supporting the claim that the method is robust and architecture-agnostic provided that the underlying model uses RoPE-like positional modeling (Zhao et al., 12 Jul 2025).

The implementation details reported are specific: training uses 8×A800 GPUs; the visual encoder is CLIP ViT-L/14; the LLM is Vicuna-7B; and training follows a two-stage procedure consisting of pretraining on CC-558K for 1 epoch with batch size 256 and instruction tuning on a 665k multi-turn conversation dataset for 1 epoch with batch size 128 (Zhao et al., 12 Jul 2025).

In the landscape of related work, MCA-LLaVA is positioned against three broad classes of approaches. According to the paper, decoding-based methods intervene at generation time, alignment-tuning approaches require extra supervision, and attention-calibration methods bias attention heuristically. MCA-LLaVA instead modifies the positional foundation of image-token and instruction-token interaction (Zhao et al., 12 Jul 2025). This makes it conceptually distinct from MC-LLaVA, whose objective is multi-concept personalization through joint training, concept-token initialization, and personalized prompts rather than hallucination mitigation through positional remodeling (An et al., 2024).

7. Significance, limitations, and interpretation

The broader significance of MCA-LLaVA lies in its reframing of hallucination reduction as partly a positional geometry problem. The paper argues that when an LVLM uses a one-dimensional positional model for a two-dimensional image, attention can become structurally biased before semantic reasoning occurs. MCA-LLaVA is presented as a principled method for bringing the model’s internal geometry into closer correspondence with the geometry of the input (Zhao et al., 12 Jul 2025).

Several limitations are stated or implied. The method still relies on a causal Transformer framework and RoPE-style modeling. The Manhattan reindexing is designed for grid-like image patches, so generalization to irregular visual token layouts may require adaptation. The paper does not claim to solve hallucination fully. It also notes, implicitly, that the coordinate reassignment is engineered and may require careful implementation to preserve compatibility with existing model architectures (Zhao et al., 12 Jul 2025).

A plausible implication is that MCA-LLaVA’s contribution is less a complete anti-hallucination solution than a demonstration that multimodal hallucination can emerge from the positional substrate of attention itself. In that sense, the paper broadens the design space for LVLM reliability: improvements may come not only from better supervision, safer decoding, or stronger alignment objectives, but also from rethinking how positional bias structures image-language interaction.