---
title: Recurrent Criss-Cross Attention (RCCA)
url: https://www.emergentmind.com/topics/recurrent-criss-cross-attention-rcca
type: topic
---

# Recurrent Criss-Cross Attention (RCCA)

Recurrent Criss-Cross Attention (RCCA) is an efficient context aggregation mechanism for dense prediction problems in computer vision. Designed to achieve global context modeling with minimal computational and memory overhead, RCCA recurrently applies a sparse attention mechanism—criss-cross attention—across spatial or spatiotemporal positions to approximate the effect of dense non-local self-attention. Through the repetition of attention along rows and columns (or their spatiotemporal generalizations), RCCA allows each position to aggregate features from all other positions in the input with a fraction of the cost of fully-connected attention. This approach has led to state-of-the-art performance in semantic segmentation, change detection, and action recognition, among other tasks [1811.11721][2108.08157][2103.11190].

## 1. Criss-Cross Attention: Sparse Pathwise Context

The criss-cross attention (CCA) mechanism computes attention not between all pairs of positions but only among those sharing the same row or column with the query. For a feature map $H \in \mathbb{R}^{C \times H \times W}$, each position $u = (i,j)$ defines its "criss-cross path," the set $\Omega(i,j)$ consisting of all spatial positions with $i$ fixed or $j$ fixed, i.e., the union of the $i$-th row and $j$-th column. For each CCA pass:

1. Three 1×1 convolutions generate $Q = W_Q * H$, $K = W_K * H$, $V = W_V * H$.
2. For each $(i,j)$ and every $(p,q) \in \Omega(i,j)$, affinities are computed as
   $$
   d_{(i,j) \to (p,q)} = Q_{i,j} \cdot K_{p,q}
   $$
3. Attention weights $H_{i,j \to p,q}$ are obtained by softmax normalization over $\Omega(i,j)$:
   $$
   H_{i,j \to p,q} = \frac{\exp(Q_{i,j} \cdot K_{p,q})}{\sum_{(u,v) \in \Omega(i,j)} \exp(Q_{i,j} \cdot K_{u,v})}
   $$
4. The output at each position is
   $$
   O_{i,j} = \sum_{(p,q) \in \Omega(i,j)} H_{i,j \to p,q} \cdot V_{p,q} + H_{i,j} 
   $$
where the residual connection aids optimization [1811.11721].

## 2. Recurrence: Achieving Full-Image Connectivity

A single CCA pass restricts information flow to pixels that are co-linear; arbitrary pixel pairs not aligned spatially remain unconnected. RCCA addresses this limitation by applying the CCA module recurrently, typically $R=2$ times for 2D images. With each recurrence, information propagates farther: after two passes, every position is linked to all others via a two-step path through shared rows or columns. Mathematically, the recurrent update is:
$$
H^{0} = H, \quad H^{r} = \text{CCA}(H^{r-1}), \quad r=1,\dots,R
$$
and the final contextual feature is $H^{R}$. This scheme uses shared weights and, aside from the within-CCA softmax and residual, does not involve additional gating [1811.11721][2108.08157].

Generalizations to higher dimensions replace the spatial "criss-cross" with joint attention along axes in spatio-temporal cubes, as in RCCA-3D. In this scenario, for a feature map $X \in \mathbb{R}^{C \times T \times H \times W}$, each query attends along its temporal, horizontal, and vertical axes, and $R=3$ recurrences suffice to achieve full spatiotemporal context propagation [2103.11190].

## 3. Computational Complexity and Resource Efficiency

Compared to full non-local self-attention (which scales as $\mathcal{O}(N^2)$ for $N=H \cdot W$ positions), RCCA achieves sub-quadratic complexity. The cost of one CCA pass is $\mathcal{O}(N \cdot (H+W))$; for square feature maps, this is $\mathcal{O}(N^{3/2})$.

Empirical profiles (for $N \approx 10^4$):
- Non-local block: 108 GFLOPs, 1411 MB memory
- RCCA ($R=2$): 16.5 GFLOPs, 127 MB memory

This yields an 85% reduction in FLOPs and an 11× reduction in memory usage [1811.11721]. In specialized architectures such as EffCDNet, full-attention cost is $\sim$10.34 GFLOPs for $256\times256$ inputs, while RCCA at $R=2$ requires only $\sim$0.95 GFLOPs (~9.2% of full attention) [2108.08157]. Similar efficiency gains (25–30% reduction in parameter and FLOPs overhead) are reported in the spatiotemporal RCCA-3D module relative to video non-local blocks [2103.11190].

## 4. Integration into Deep Networks

RCCA has been implemented as a plug-in module—typically post-backbone, pre-classification—in encoder-decoder or fully-convolutional networks:

- **CCNet (semantic segmentation):** RCCA with $R=2$ on ResNet-101 backbone. Outputs $H^R$ are fused (concatenated) with backbone features, projected, normalized, and classified [1811.11721].
- **EffCDNet (change detection):** Two sequential RCCA blocks with $R=2$ each in the decoder; group convolutions and channel shuffle further economize parameters [2108.08157].
- **Action recognition architectures:** RCCA-3D inserted before the last residual units in 3D CNN backbones (TSM, MF-Net), using shared convolution weights and accumulated via a quantized residual path [2103.11190].

Typical design involves dimension reduction for $Q$, $K$ (e.g., $C'\approx C/2$ or $C/4$) and residual fusion, with recurrent step count ($R=2$ for 2D, $R=3$ for 3D) tuned experimentally for maximal coverage and minimal cost.

## 5. Impact on Performance and Quantitative Outcomes

Across benchmarks, RCCA yields superior accuracy relative to non-local or local context baselines for a fraction of the computational cost. Representative results [1811.11721][2108.08157][2103.11190]:

| Task / Dataset          | Baseline (mIoU / F1) | RCCA / RCCA-3D (mIoU / F1) | Overhead (%)        |
|------------------------|----------------------|----------------------------|---------------------|
| Cityscapes seg.        | 75.1% (FCN+ResNet)   | 81.9% (test, RCCA+CCLoss)  | -85% FLOPs, -11x mem|
| ADE20K seg.            | 44.7% (prior best)   | 45.76% (RCCA+CCLoss)       | see above           |
| EffCDNet (SVCD, F1)    | 91.73%               | 94.30% (RCCA, R=2)         | 9.2% of full attention|
| TSM (UCF101, top-1)    | 85.01%               | 86.81% (RCCA-3D)           | -30% FLOPs, -27% params|
| MF-Net (UCF101, top-1) | 78.60%               | 80.76% (5×RCCA-3D)         | see above           |

In these results, RCCA modules closely match or slightly exceed non-local performance, while incurring substantially lower resource costs. In change detection, RCCA in EffCDNet improved F1 score by up to 2.57% over the backbone and achieves 94.30%—just 0.09% below full-attention—at only 10% of the compute [2108.08157]. In action recognition, RCCA-3D over TSM or MF-Net yields 1–2% accuracy gains over backbones and is comparable to (or outperforms) non-local at lower compute and parameter cost [2103.11190].

## 6. Extensions and Generalizations

The criss-cross attention principle has been extended from 2D spatial grids to higher dimensions:

- **3D Criss-Cross Attention (RCCA-3D):** Factorizes global context in 3D cubes by attention along temporal, vertical, and horizontal axes. Three recurrent passes enable aggregation of full-volume context, while keeping the overhead sub-cubic [2103.11190].
- **Application to video and medical volumes:** In CamVid video segmentation, RCCA-3D yields 79.1% mIoU with ResNet-101 pretraining [1811.11721]. Applications to volumetric medical data and point clouds are plausible by defining similar scan-line sets.
- **Further efficiency strategies:** Group convolution and channel shuffle compress parameter count without degrading context quality [2108.08157].
- **Potential adaptive recurrences:** Gated or learned recurrent schemes may allow per-location context depth adaptation, though the standard in reviewed work is fixed $R$.

## 7. Discriminative Loss and Class Separation

To improve intra-class compactness and inter-class separability of pixel features, CCNet incorporates a category consistent (discriminative) loss. This loss combines terms penalizing within-class variance, maximizing class centroid distance, and regularizing centroid magnitude:
$$
\ell = \ell_{\text{seg}} + \alpha \ell_{\text{var}} + \beta \ell_{\text{dis}} + \gamma \ell_{\text{reg}}
$$
where
- $\ell_{\text{var}}$ encourages embeddings of the same class to be close,
- $\ell_{\text{dis}}$ penalizes class centroids that are too near,
- $\ell_{\text{reg}}$ controls centroid norm.

This loss is typically applied to a low-dimensional embedding (e.g., 16-channel) collapsed from the final feature. With this addition, RCCA modules yield more discriminative feature spaces, as reflected in t-SNE plots and feature distance maps [1811.11721][2108.08157].

---

RCCA provides a general, plug-and-play alternative to conventional non-local attention for full-context aggregation in deep networks. By leveraging criss-cross (and in 3D, axis-wise) recurrence, it achieves leading accuracy in context-sensitive vision tasks, with parameter and computational costs orders of magnitude lower than fully-connected attention modules [1811.11721][2108.08157][2103.11190].

Source: https://www.emergentmind.com/topics/recurrent-criss-cross-attention-rcca