Criss-Cross Network (CCNet)

• Criss-Cross Network (CCNet) is a deep learning model that uses a criss-cross attention mechanism to aggregate image context by focusing on rows and columns.
• It employs recurrent criss-cross attention and a category consistent loss to achieve state-of-the-art segmentation accuracy while reducing computational and memory costs.
• Empirical evaluations on benchmarks like Cityscapes and ADE20K show improved mIoU and efficiency compared to traditional non-local attention methods.

The Criss-Cross Network (CCNet) is a deep learning architecture for semantic segmentation and dense visual prediction that leverages a novel criss-cross attention (CCA) mechanism to efficiently capture full-image contextual dependencies. CCNet achieves this by restricting pixel-level attention to the criss-cross path of the reference pixel, substantially reducing the computational and memory overhead compared to traditional non-local self-attention, while maintaining or surpassing state-of-the-art segmentation performance (Huang et al., 2018).

1. Motivation and Architectural Overview

Fully convolutional networks (FCNs) are constrained by limited receptive fields, often aggregating only local context and failing to capture dependencies between distant pixels—a limitation detrimental to tasks like semantic segmentation. Traditional non-local/self-attention modules overcome this by computing pairwise similarities for all $N=H\times W$ pixels, but at prohibitive $\mathcal O(N^2)$ time and space complexity. CCNet introduces a more tractable criss-cross attention structure: each pixel attends solely to pixels in the same row and column, reducing the attention set size for each pixel to approximately $2\sqrt N$ for images of nearly equal height $H$ and width $W$.

The high-level CCNet pipeline is as follows:

1. A backbone CNN (e.g., ResNet-101 with dilated convolutions) processes the input, yielding a feature map $\mathbf X\in \mathbb R^{C\times H\times W}$.
2. A $1\times1$ convolution reduces channels to $C'$, producing $\mathbf H\in \mathbb R^{C'\times H\times W}$.
3. The recurrent criss-cross attention (RCCA) module, applied $R=2$ times, augments $\mathcal O(N^2)$0 with full-image context to produce $\mathcal O(N^2)$1.
4. The contextualized $\mathcal O(N^2)$2 is concatenated with the original $\mathcal O(N^2)$3, further fused using convolutional layers, and then fed to a pixel-wise classifier.

2. Criss-Cross Attention Mechanism

2.1. Linear Projections

Given input features $\mathcal O(N^2)$4, three $\mathcal O(N^2)$5 convolutional layers generate queries ($\mathcal O(N^2)$6), keys ($\mathcal O(N^2)$7), and values ($\mathcal O(N^2)$8):

$\mathcal O(N^2)$9

with $2\sqrt N$0, $2\sqrt N$1 and $2\sqrt N$2.

2.2. Affinity and Normalization

For each spatial position $2\sqrt N$3, attention operates over $2\sqrt N$4—the set of all pixels sharing the same row or column. The affinity between reference position $2\sqrt N$5 and a spatial position $2\sqrt N$6 is computed as:

$2\sqrt N$7

Normalization via softmax is applied over the criss-cross set:

$2\sqrt N$8

2.3. Context Aggregation

The output feature at position $2\sqrt N$9 is computed as:

$H$0

or, in tensor notation,

$H$1

3. Recurrent Application for Global Context

Because a single criss-cross attention module propagates information only within rows and columns, CCNet employs recurrence: two sequential passes of CCA with shared parameters. The update is:

$H$2

The final feature $H$3 thus incorporates information from the entire image, as pixels propagate context "across" and then "down" or vice versa, covering all spatial dependencies after two iterations.

4. Category Consistent Loss for Feature Discrimination

CCNet introduces a category consistent loss (CCL) to enhance class-discriminative feature clustering:

• For each class $H$4 with pixel count $H$5, feature vectors $H$6 (from a $H$7 conv) define cluster centers $H$8.
• The loss consists of:
  • Intra-class variance term:

  $H$9

  with

  $W$0 - Inter-class distance term:

  $W$1

  $W$2 - Regularization:

  $W$3

• The full loss is:

$W$4

Recommended weights: $W$5, $W$6, $W$7, $W$8.

5. Computational and Memory Efficiency

Comparing with non-local modules:

• Non-local attention: Each of $W$9 pixels attends to $\mathbf X\in \mathbb R^{C\times H\times W}$0 positions, yielding $\mathbf X\in \mathbb R^{C\times H\times W}$1 computational/memory cost.

• Criss-cross attention: Each pixel processes only $\mathbf X\in \mathbb R^{C\times H\times W}$2 positions for nearly square images, so total cost is $\mathbf X\in \mathbb R^{C\times H\times W}$3.

• In empirical evaluation:

  • Non-local block ($\mathbf X\in \mathbb R^{C\times H\times W}$4) on $\mathbf X\in \mathbb R^{C\times H\times W}$5 features: ~108 GFLOPs, +1411 MB GPU memory.
  • RCCA ($\mathbf X\in \mathbb R^{C\times H\times W}$6): ~16.5 GFLOPs, +127 MB GPU memory.
  • Thus, RCCA is about 11× more memory efficient and reduces FLOPs by ~85% compared to non-local blocks.

Module	Computational Cost	Memory Cost
Non-local	$\mathbf X\in \mathbb R^{C\times H\times W}$7	$\mathbf X\in \mathbb R^{C\times H\times W}$8
CCA (single)	$\mathbf X\in \mathbb R^{C\times H\times W}$9	$1\times1$0
RCCA ($1\times1$1)	$1\times1$2	$1\times1$3

This efficiency enables deployment on high-resolution images where non-local modules are infeasible.

6. Empirical Evaluation on Benchmark Datasets

CCNet's performance has been validated on multiple semantic and instance segmentation benchmarks:

• Cityscapes:
  • Val: 80.5% mIoU with ResNet-101 + $1\times1$4 + CCL.
  • Test: 81.9% mIoU, outperforming DenseASPP (80.6%).
• ADE20K (150 classes):
  • Val: 45.76% mIoU, an improvement of +1.1 percentage points over EncNet (44.65%).
• LIP (human parsing, 20 parts):
  • Val: 55.47% mIoU, +2.3 over previous best CE2P (53.10%).
• COCO (instance segmentation):
  • Mask R-CNN + RCCA (ResNet-50): Box AP increases from 38.2 to 39.3; Mask AP from 34.8 to 36.1.
  • With ResNet-101: Box AP 40.1→41.0; Mask AP 36.2→37.3.
• CamVid (video segmentation):
  • 3D-RCCA with 5-frame clips (ResNet-101, $1\times1$5, $1\times1$6): 79.1% mIoU, up from 77.9% ($1\times1$7), exceeding VideoGCRF (75.2%).

These results demonstrate that CCNet not only improves performance metrics, but does so with significantly reduced computation and memory requirements (Huang et al., 2018).

7. Summary and Significance

CCNet advances semantic segmentation by combining criss-cross attention with recurrent propagation and a category consistent loss, yielding dense contextual aggregation across images at a fraction of the computational cost incurred by non-local self-attention modules. Empirical results demonstrate its broad applicability and state-of-the-art segmentation accuracy on standard benchmarks. The source code is publicly available at https://github.com/speedinghzl/CCNet (Huang et al., 2018).