Recurrent Criss-Cross Attention (RCCA)

• The paper demonstrates that recurrent criss-cross attention approximates dense non-local self-attention, achieving global connectivity with minimal computational overhead.
• It reduces complexity by using repeated criss-cross passes, lowering FLOPs by up to 85% and memory usage by 11× compared to full attention.
• RCCA is integrated as a plug‑in module in deep networks to enhance tasks like semantic segmentation and action recognition while maintaining state‑of‑the‑art accuracy.

Recurrent Criss-Cross Attention (RCCA) is an efficient context aggregation mechanism for dense prediction problems in computer vision. Designed to achieve global context modeling with minimal computational and memory overhead, RCCA recurrently applies a sparse attention mechanism—criss-cross attention—across spatial or spatiotemporal positions to approximate the effect of dense non-local self-attention. Through the repetition of attention along rows and columns (or their spatiotemporal generalizations), RCCA allows each position to aggregate features from all other positions in the input with a fraction of the cost of fully-connected attention. This approach has led to state-of-the-art performance in semantic segmentation, change detection, and action recognition, among other tasks (Huang et al., 2018, Chen et al., 2021, Cao et al., 2021).

1. Criss-Cross Attention: Sparse Pathwise Context

The criss-cross attention (CCA) mechanism computes attention not between all pairs of positions but only among those sharing the same row or column with the query. For a feature map $H \in \mathbb{R}^{C \times H \times W}$, each position $u = (i,j)$ defines its "criss-cross path," the set $\Omega(i,j)$ consisting of all spatial positions with $i$ fixed or $j$ fixed, i.e., the union of the $i$-th row and $j$-th column. For each CCA pass:

1. Three 1×1 convolutions generate $Q = W_Q * H$, $K = W_K * H$, $V = W_V * H$.
2. For each $u = (i,j)$0 and every $u = (i,j)$1, affinities are computed as

$u = (i,j)$2

1. Attention weights $u = (i,j)$3 are obtained by softmax normalization over $u = (i,j)$4:

$u = (i,j)$5

2. The output at each position is

$u = (i,j)$6

where the residual connection aids optimization (Huang et al., 2018).

2. Recurrence: Achieving Full-Image Connectivity

A single CCA pass restricts information flow to pixels that are co-linear; arbitrary pixel pairs not aligned spatially remain unconnected. RCCA addresses this limitation by applying the CCA module recurrently, typically $u = (i,j)$7 times for 2D images. With each recurrence, information propagates farther: after two passes, every position is linked to all others via a two-step path through shared rows or columns. Mathematically, the recurrent update is:

$u = (i,j)$8

and the final contextual feature is $u = (i,j)$9. This scheme uses shared weights and, aside from the within-CCA softmax and residual, does not involve additional gating (Huang et al., 2018, Chen et al., 2021).

Generalizations to higher dimensions replace the spatial "criss-cross" with joint attention along axes in spatio-temporal cubes, as in RCCA-3D. In this scenario, for a feature map $\Omega(i,j)$0, each query attends along its temporal, horizontal, and vertical axes, and $\Omega(i,j)$1 recurrences suffice to achieve full spatiotemporal context propagation (Cao et al., 2021).

3. Computational Complexity and Resource Efficiency

Compared to full non-local self-attention (which scales as $\Omega(i,j)$2 for $\Omega(i,j)$3 positions), RCCA achieves sub-quadratic complexity. The cost of one CCA pass is $\Omega(i,j)$4; for square feature maps, this is $\Omega(i,j)$5.

Empirical profiles (for $\Omega(i,j)$6):

• Non-local block: 108 GFLOPs, 1411 MB memory
• RCCA ($\Omega(i,j)$7): 16.5 GFLOPs, 127 MB memory

This yields an 85% reduction in FLOPs and an 11× reduction in memory usage (Huang et al., 2018). In specialized architectures such as EffCDNet, full-attention cost is $\Omega(i,j)$810.34 GFLOPs for $\Omega(i,j)$9 inputs, while RCCA at $i$0 requires only $i$10.95 GFLOPs (~9.2% of full attention) (Chen et al., 2021). Similar efficiency gains (25–30% reduction in parameter and FLOPs overhead) are reported in the spatiotemporal RCCA-3D module relative to video non-local blocks (Cao et al., 2021).

4. Integration into Deep Networks

RCCA has been implemented as a plug-in module—typically post-backbone, pre-classification—in encoder-decoder or fully-convolutional networks:

• CCNet (semantic segmentation): RCCA with $i$2 on ResNet-101 backbone. Outputs $i$3 are fused (concatenated) with backbone features, projected, normalized, and classified (Huang et al., 2018).
• EffCDNet (change detection): Two sequential RCCA blocks with $i$4 each in the decoder; group convolutions and channel shuffle further economize parameters (Chen et al., 2021).
• Action recognition architectures: RCCA-3D inserted before the last residual units in 3D CNN backbones (TSM, MF-Net), using shared convolution weights and accumulated via a quantized residual path (Cao et al., 2021).

Typical design involves dimension reduction for $i$5, $i$6 (e.g., $i$7 or $i$8) and residual fusion, with recurrent step count ($i$9 for 2D, $j$0 for 3D) tuned experimentally for maximal coverage and minimal cost.

5. Impact on Performance and Quantitative Outcomes

Across benchmarks, RCCA yields superior accuracy relative to non-local or local context baselines for a fraction of the computational cost. Representative results (Huang et al., 2018, Chen et al., 2021, Cao et al., 2021):

Task / Dataset	Baseline (mIoU / F1)	RCCA / RCCA-3D (mIoU / F1)	Overhead (%)
Cityscapes seg.	75.1% (FCN+ResNet)	81.9% (test, RCCA+CCLoss)	-85% FLOPs, -11x mem
ADE20K seg.	44.7% (prior best)	45.76% (RCCA+CCLoss)	see above
EffCDNet (SVCD, F1)	91.73%	94.30% (RCCA, R=2)	9.2% of full attention
TSM (UCF101, top-1)	85.01%	86.81% (RCCA-3D)	-30% FLOPs, -27% params
MF-Net (UCF101, top-1)	78.60%	80.76% (5×RCCA-3D)	see above

In these results, RCCA modules closely match or slightly exceed non-local performance, while incurring substantially lower resource costs. In change detection, RCCA in EffCDNet improved F1 score by up to 2.57% over the backbone and achieves 94.30%—just 0.09% below full-attention—at only 10% of the compute (Chen et al., 2021). In action recognition, RCCA-3D over TSM or MF-Net yields 1–2% accuracy gains over backbones and is comparable to (or outperforms) non-local at lower compute and parameter cost (Cao et al., 2021).

6. Extensions and Generalizations

The criss-cross attention principle has been extended from 2D spatial grids to higher dimensions:

• 3D Criss-Cross Attention (RCCA-3D): Factorizes global context in 3D cubes by attention along temporal, vertical, and horizontal axes. Three recurrent passes enable aggregation of full-volume context, while keeping the overhead sub-cubic (Cao et al., 2021).
• Application to video and medical volumes: In CamVid video segmentation, RCCA-3D yields 79.1% mIoU with ResNet-101 pretraining (Huang et al., 2018). Applications to volumetric medical data and point clouds are plausible by defining similar scan-line sets.
• Further efficiency strategies: Group convolution and channel shuffle compress parameter count without degrading context quality (Chen et al., 2021).
• Potential adaptive recurrences: Gated or learned recurrent schemes may allow per-location context depth adaptation, though the standard in reviewed work is fixed $j$1.

7. Discriminative Loss and Class Separation

To improve intra-class compactness and inter-class separability of pixel features, CCNet incorporates a category consistent (discriminative) loss. This loss combines terms penalizing within-class variance, maximizing class centroid distance, and regularizing centroid magnitude:

$j$2

where

• $j$3 encourages embeddings of the same class to be close,
• $j$4 penalizes class centroids that are too near,
• $j$5 controls centroid norm.

This loss is typically applied to a low-dimensional embedding (e.g., 16-channel) collapsed from the final feature. With this addition, RCCA modules yield more discriminative feature spaces, as reflected in t-SNE plots and feature distance maps (Huang et al., 2018, Chen et al., 2021).

---

RCCA provides a general, plug-and-play alternative to conventional non-local attention for full-context aggregation in deep networks. By leveraging criss-cross (and in 3D, axis-wise) recurrence, it achieves leading accuracy in context-sensitive vision tasks, with parameter and computational costs orders of magnitude lower than fully-connected attention modules (Huang et al., 2018, Chen et al., 2021, Cao et al., 2021).