Layer-Wise Criss-Cross Attention

• Layer-Wise Criss-Cross Attention is a mechanism that aggregates context along two orthogonal axes to efficiently combine intra-layer and inter-layer information.
• It is applied in diverse domains, such as vision-language models, medical image segmentation, and EEG decoding, to enhance reasoning accuracy and computational efficiency.
• Evaluations, particularly in MAP, show that incorporating a broadcast update significantly improves factual accuracy by consolidating scattered semantic signals.

Layer-Wise Criss-Cross Attention denotes a family of attention mechanisms that aggregate context along two structured axes rather than over an unrestricted fully connected set. In its most explicit cross-depth form, the mechanism was introduced in “MAP: Mitigating Hallucinations in Large Vision-LLMs with Map-Level Attention Processing,” where the hidden states of a large vision-LLM are interpreted as a 2D semantic map and each decoding layer is refined by aggregating information from both intra-layer token positions and inter-layer representations of the same token (Li et al., 3 Aug 2025). Related criss-cross formulations appear in medical image segmentation, EEG foundation models, and document-level relation extraction, where the two axes are spatial rows and columns, spatial and temporal stripes, or subject and object axes of an entity-pair matrix (Tang et al., 2019, Wang et al., 2024, Zhang et al., 2022).

1. Conceptual scope

Criss-cross attention is defined by the choice of a structured neighborhood composed of two orthogonal traversal directions. Rather than attending to every location in a 2D grid or every token in a sequence, the operator attends along a row-like axis and a column-like axis. In MAP, those axes are “other tokens in the current layer” and “the same token position across previous layers.” In XLSor, they are the horizontal and vertical directions of a feature map. In CBraMod, they are spatial stripes across EEG channels and temporal stripes across time segments. In Dense-CCNet, they are the row and column of an entity-pair matrix, corresponding to fixed subject and fixed object reasoning paths (Li et al., 3 Aug 2025, Tang et al., 2019, Wang et al., 2024, Zhang et al., 2022).

Work	Structured domain	Criss-cross axes
MAP	$$\mathcal H_\ell \in \mathbb R^{\ell \times T \times D}$$	same layer across tokens; same token across layers
XLSor	$F \in \mathbb R^{C \times H \times W}$	same row; same column
CBraMod	$X \in \mathbb R^{C \times n \times d}$	spatial stripes; temporal stripes
Dense-CCNet	$M \in \mathbb R^{N_e \times N_e \times d}$	subject row; object column

This pattern suggests that “criss-cross” is best understood as an axis-factorized attention topology. What changes across domains is not the underlying principle of structured bidirectional aggregation, but the semantic meaning of the two axes.

2. MAP formulation on a 2D semantic map

In MAP, the decoder has $T$ total tokens at each step and $N$ transformer layers. The hidden state of token position $u$ at decoder layer $\ell$ is $h_{u,\ell}\in\mathbb R^D$, and the layer sequence is $$H_\ell=[h_{1,\ell},\dots,h_{T,\ell}]\in\mathbb R^{T\times D}$$. The central construction is the 2D semantic map $F \in \mathbb R^{C \times H \times W}$0, which collects all hidden states from layer $F \in \mathbb R^{C \times H \times W}$1 up to $F \in \mathbb R^{C \times H \times W}$2 so that $F \in \mathbb R^{C \times H \times W}$3 (Li et al., 3 Aug 2025).

For a focus token $F \in \mathbb R^{C \times H \times W}$4, Layer-Wise Criss-Cross Attention defines the criss-cross semantic neighborhood

$F \in \mathbb R^{C \times H \times W}$5

The first set is intra-layer: all other tokens in the current layer. The second set is inter-layer: the same token position in earlier layers. In a QKV notation,

$F \in \mathbb R^{C \times H \times W}$6

The compatibility score between the focus token and a neighbor $F \in \mathbb R^{C \times H \times W}$7 is normalized by a softmax over the entire criss-cross neighborhood,

$F \in \mathbb R^{C \times H \times W}$8

and the aggregated vector is

$F \in \mathbb R^{C \times H \times W}$9

The paper also gives an equivalent cosine-similarity form,

$X \in \mathbb R^{C \times n \times d}$0

This dual presentation makes clear that the mechanism can be interpreted either as a conventional attention module or as a direct similarity-weighted aggregation over the 2D semantic map.

3. Broadcast update and layer-wise propagation

A distinctive feature of MAP’s Layer-Wise Criss-Cross Attention is the broadcast update. Rather than replacing only the hidden state of the focus token, MAP broadcasts the aggregated vector $X \in \mathbb R^{C \times n \times d}$1 to all positions in layer $X \in \mathbb R^{C \times n \times d}$2: $X \in \mathbb R^{C \times n \times d}$3 where $X \in \mathbb R^{C \times n \times d}$4 is a hyperparameter controlling original versus aggregated contribution (Li et al., 3 Aug 2025).

Operationally, for each decoding step $X \in \mathbb R^{C \times n \times d}$5 and for each decoding layer $X \in \mathbb R^{C \times n \times d}$6 from a user-chosen start_layer up to $X \in \mathbb R^{C \times n \times d}$7, the method builds a criss-cross index set from the row $X \in \mathbb R^{C \times n \times d}$8 and the column $X \in \mathbb R^{C \times n \times d}$9, computes normalized attention scores, forms $M \in \mathbb R^{N_e \times N_e \times d}$0, applies the broadcast update to all token positions in layer $M \in \mathbb R^{N_e \times N_e \times d}$1, replaces $M \in \mathbb R^{N_e \times N_e \times d}$2 with the updated states, and passes $M \in \mathbb R^{N_e \times N_e \times d}$3 forward into layer $M \in \mathbb R^{N_e \times N_e \times d}$4 (Li et al., 3 Aug 2025).

The rationale given for hallucination mitigation is explicitly layer-wise. Factual signals in large vision-LLMs are described as “scattered” across token positions and intermediate layers, rather than confined to the final layer or to local heads. Criss-cross attention therefore collects both intra-layer and inter-layer cues in one operation; broadcasting ensures that the entire layer benefits from the consolidated factual summary; and progressive propagation to deeper layers grounds next-token prediction on a more coherent, visually anchored semantic map, suppressing language-only priors (Li et al., 3 Aug 2025).

4. Quantitative evidence in hallucination mitigation

MAP reports an ablation on the MME “perception” benchmark that isolates the effect of map-level operations and, specifically, the contribution of Layer-Wise Criss-Cross Attention and broadcasting (Li et al., 3 Aug 2025).

Map-level operation	Total score
Vanilla decoding	1491.56
Global attention only	1509.71
Local square (7×7)	1502.44
Criss-cross (single layer)	1507.92
LW-CCA w/o broadcast	1500.08
LW-CCA w/ broadcast (full MAP)	1520.33

The paper states that the jump from $M \in \mathbb R^{N_e \times N_e \times d}$5 $M \in \mathbb R^{N_e \times N_e \times d}$6, together with the drop to $M \in \mathbb R^{N_e \times N_e \times d}$7 for “w/o broadcast,” confirms that both the criss-cross design and the broadcasting step are critical to the overall gain in factual accuracy. At the benchmark level, MAP is reported to improve truthfulness and performance on POPE, MME, and MMHal-Bench, and on object-hallucination benchmarks it consistently outperforms prior inter-layer and intra-layer inference methods (Li et al., 3 Aug 2025).

These results are specific to MAP’s training-free decoding setting. A plausible implication is that the gain arises not merely from using additional context, but from the particular topology by which the model consolidates information distributed across both token positions and decoder depth.

5. Domain-specific variants of criss-cross attention

In XLSor, criss-cross attention is applied to chest X-ray segmentation rather than autoregressive decoding. The backbone is a ResNet-101 pretrained on ImageNet, modified so that the final feature-map stride is $M \in \mathbb R^{N_e \times N_e \times d}$8, producing $M \in \mathbb R^{N_e \times N_e \times d}$9 for a $T$0 input. Two CCA modules are stacked in a recurrent fashion with shared weights, inserted immediately after the final ResNet block and before the segmentation head. For each pixel position $T$1, the criss-cross path $T$2 is the union of all positions in the same row and the same column, excluding $T$3 itself, so each pixel attends to $T$4 positions. Because each pixel attends to $T$5 locations rather than $T$6, the module is more memory- and compute-efficient than a full non-local block. The paper reports that XLSor outperformed a U-Net baseline by up to 12 percent Dice on the challenging NIH abnormal set, and that removing the CCA modules caused a drop in Dice by $T$7–$T$8 percent on unseen abnormal CXRs (Tang et al., 2019).

In Dense-CCNet, the input is an entity-pair matrix $T$9. For each pair $N$0, one CCA layer attends horizontally over $N$1 and vertically over $N$2, producing

$N$3

Multiple CCA layers are then densely connected via

$N$4

The paper explicitly interprets one CCA layer as single-hop reasoning through paths $N$5, while stacked layers enable multi-hop reasoning such as $N$6. Dense connections preserve lower-hop information while progressively enabling longer reasoning chains (Zhang et al., 2022).

In CBraMod, the criss-cross principle is implemented inside a Transformer backbone for EEG decoding. Each block uses a pre-norm layout, and Criss-Cross Attention splits the $N$7 attention heads into two equal groups: spatial-attention heads attend across channels at fixed time-slices, and temporal-attention heads attend across time-steps within a single channel. The total per-layer complexity is $N$8, compared with $N$9 for full 2D attention. CBraMod further uses asymmetric conditional positional encoding generated by a depth-wise 2D convolution with $u$0, for example $u$1 and $u$2. The paper reports that CBraMod’s dual-map criss-cross yields the best accuracy and Cohen’s $u$3 on emotion recognition and motor-imagery, and that a 4:4 split of spatial and temporal heads outperforms 2:6 and 6:2 (Wang et al., 2024).

6. Relation to other cross-layer attention mechanisms

Layer-wise criss-cross attention is not identical to every form of cross-layer information mixing. Depth-Attention, introduced for LLMs, performs a depth-wise attention step in which the query at layer $u$4 for token $u$5 attends to the keys of layers $u$6 at the same token position and mixes their values into a depth-mixed value $u$7, which then replaces the original value in the standard self-attention computation. It reuses the existing $u$8, $u$9, and $\ell$0 projections, adds no parameters, and introduces no persistent inference state beyond the standard key-value cache (Zeng et al., 3 Jun 2026).

The distinction from MAP is structural. MAP’s neighborhood

$\ell$1

combines same-layer cross-token aggregation with same-token cross-layer aggregation, and then broadcasts the resulting summary across the entire current layer (Li et al., 3 Aug 2025). Depth-Attention, by contrast, performs layer-wise selection only along depth at a fixed token position before ordinary sequence-wise attention (Zeng et al., 3 Jun 2026). This suggests that “layer-wise” alone is insufficient to characterize the mechanism: the decisive property is whether the operator jointly traverses two axes in a criss-cross topology.

A related misconception is that criss-cross attention is intrinsically tied to image grids. The surveyed formulations contradict that view. The same design pattern appears on decoder-layer semantic maps, spatial feature maps, EEG channel-time lattices, and entity-pair matrices. Across these settings, the operator serves a common role: it preserves structured inductive bias while avoiding the cost of unrestricted all-to-all attention.