---
title: Dual-Attention Vision Transformer (DaViT)
url: https://www.emergentmind.com/topics/dual-attention-vision-transformer-davit
type: topic
---

# Dual-Attention Vision Transformer (DaViT)

Dual-Attention Vision Transformers (DaViT) constitute a vision transformer architecture designed to balance global context modeling and computational efficiency by leveraging two orthogonal types of self-attention: spatial window attention and channel group attention. This dual-attention scheme enables the architecture to alternate between fine-grained local feature refinement and global feature interactions, providing significant gains in image classification, object detection, and semantic segmentation benchmarks. DaViT achieves state-of-the-art trade-offs between accuracy, parameter count, and computational cost, while scaling linearly with both spatial resolution and channel dimension [2204.03645].

## 1. Dual-Attention Design and Mechanisms

DaViT alternates two distinct but complementary forms of self-attention within each block:

- **Spatial window attention:** Operates over “spatial tokens,” where each token corresponds to a spatial location (e.g., an image patch).
- **Channel group attention:** Operates over “channel tokens,” where each token corresponds to a channel (feature map dimension), enabling global spatial aggregation.

### 1.1 Spatial Tokens and Window Self-Attention

Given a feature map $X \in \mathbb{R}^{P \times C}$, with $P$ spatial positions and $C$ channels, each spatial token is a row $x_p \in \mathbb{R}^C$. The $P$ tokens are partitioned into $N_w$ non-overlapping windows of size $P_w$ such that $P = N_w \cdot P_w$. Within each window $i$, multi-head self-attention is computed independently:

For head $h$:
$$
Q_{i,h} = X_i W^Q_h, \quad K_{i,h} = X_i W^K_h, \quad V_{i,h} = X_i W^V_h,
$$
$$
\text{head}_{i,h} = \mathrm{softmax}\left(\frac{Q_{i,h} K_{i,h}^T}{\sqrt{C_h}}\right) V_{i,h}
$$
Aggregated output per window:
$$
A_{\text{window}}(X) = \mathrm{Concat}_{h=1\ldots N_h}(\text{head}_{i,h}) W^O
$$

The complexity per block is $O(2 P P_w C + 4 P C^2)$, linear in $P$ as $P_w$ is a fixed window size.

### 1.2 Channel Tokens and Grouped Self-Attention

Transposing $X$ yields $X^T \in \mathbb{R}^{C \times P}$, where each “channel token” is the $j$th row. Instead of global attention (quadratic in $C$), DaViT divides $C$ channels into $N_g$ groups of size $C_g$.

For each group $g$:
$$
X^{(g)} = X[:, (g-1)C_g+1 : gC_g] \in \mathbb{R}^{P \times C_g}
$$
Single-head global attention is then applied within each group:
$$
Q^{(g)} = X^{(g)} W^Q_g, \quad K^{(g)} = X^{(g)} W^K_g, \quad V^{(g)} = X^{(g)} W^V_g,
$$
$$
A_{\text{group}}(X^{(g)}) = \left(\mathrm{softmax}\left(\frac{Q^{(g)T} K^{(g)}}{\sqrt{C_g}}\right) V^{(g)T}\right)^T
$$
The outputs are concatenated:
$$
A_{\text{channel}}(X) = \mathrm{Concat}_{g=1..N_g} A_{\text{group}}(X^{(g)})
$$

Channel attention thus models global context by letting each token aggregate information from all spatial positions, while grouping reduces the computational complexity in $C$.

## 2. Token Grouping and Complexity Analysis

DaViT's efficiency arises from structured grouping in both attention types. Let $N=P$ (spatial length), $C$ (channel dim), $g = C_g$, $w^2 = P_w$.

| Attention Type              | Pre-Grouping Complexity     | Post-Grouping Complexity   |
|-----------------------------|----------------------------|----------------------------|
| Spatial attention           | $O(N^2 C)$                 | $O(N w^2 C)$               |
| Channel attention           | $O(N C^2)$                 | $O(N C g)$                 |

With fixed $w$ and $g$, both attention mechanisms scale linearly in $N$ and $C$, supporting efficient processing of high-resolution images and wide networks [2204.03645].

## 3. Architectural Variants and Network Staging

DaViT’s architecture is arranged in four stages, each comprising patch embedding followed by several alternating dual-attention blocks. The model’s four principal variants are:

| Variant      | C   | L (per stage)   | $N_h$/$N_g$ per stage         | Params (M) | FLOPs (G) | ImageNet Top-1 (%) |
|--------------|-----|-----------------|-------------------------------|------------|-----------|--------------------|
| DaViT-Tiny   | 96  | 1,1,3,1         | 3,6,12,24                     | 28.3       | 4.5       | 82.8               |
| DaViT-Small  | 96  | 1,1,9,1         | 3,6,12,24                     | 49.7       | 8.8       | 84.2               |
| DaViT-Base   | 128 | 1,1,9,1         | 4,8,16,32                     | 87.9       | 15.5      | 84.6               |
| DaViT-Giant  | 384 | 1,1,12,3        | 12,24,48,96                   | ~1,440     | 1,038     | 90.4\*             |

*\*pre-trained on 1.5B image-text pairs; Top-1 is on ImageNet-1K.*

The number of window heads $N_h$ and channel groups $N_g$ are matched per stage. DaViT-Giant leverages large-scale weakly supervised training for maximal performance.

## 4. Training Protocols and Hyperparameters

### 4.1 ImageNet-1K Image Classification

- 300 epochs, batch size 2048
- AdamW optimizer, weight decay 0.05, gradient norm clip 1.0
- Learning rate: triangular schedule with linear warmup and decay, peak $\approx 10^{-3}$
- Data augmentation and regularization mirror DeiT (excluding repeated augmentation and EMA)
- Stochastic depth: rates of 0.1 (Tiny), 0.2 (Small), 0.4 (Base)
- Training employs random crop to $224\times224$, evaluation uses center crop

### 4.2 COCO 2017 Object Detection

- DaViT backbones are integrated into RetinaNet and Mask R-CNN
- Schedules: 1× (12 epochs), 3× (36 epochs), multi-scale training with image side $\in[480,800]$
- AdamW optimizer, initial LR $= 10^{-4}$, weight decay 0.05, stochastic depth as above
- FLOPs reported at $800\times1280$ resolution

### 4.3 ADE20K Semantic Segmentation

- UPerNet framework, $512\times512$ input
- 160,000 iterations, batch size 16
- Other hyperparameters consistent with established segmentation approaches, weight decay 0.05

## 5. Empirical Evaluation and Comparisons

DaViT demonstrates state-of-the-art performance across image recognition, detection, and segmentation tasks, outperforming Swin Transformer models given matched model size and FLOPs.

### 5.1 ImageNet-1K Classification

| Model         | Params (M) | FLOPs (G) | Top-1 (%) |
|---------------|------------|-----------|-----------|
| DaViT-Tiny    | 28.3       | 4.5       | 82.8      |
| Swin-Tiny     | 28.3       | 4.5       | 81.2      |
| DaViT-Small   | 49.7       | 8.8       | 84.2      |
| Swin-Small    | 49.6       | 8.7       | 83.1      |
| DaViT-Base    | 87.9       | 15.5      | 84.6      |
| Swin-Base     | 87.8       | 15.4      | 83.4      |

Pretraining on ImageNet-22K: DaViT-Base achieves 86.9% (vs. Swin-Large 86.4%). DaViT-Huge (90.2%), DaViT-Giant (90.4%) obtain top-tier results when pretrained on large-scale weakly supervised data.

### 5.2 COCO 2017 Detection, ADE20K Segmentation

| Task        | Model         | AP$^b$ / mIoU | Improvement over Swin |
|-------------|---------------|--------------|----------------------|
| COCO RetinaNet 3× | DaViT-Tiny   | 46.5          | +1.5                |
|             | DaViT-Small  | 48.2          | +1.8                |
|             | DaViT-Base   | 48.7          | +2.9                |
| Mask R-CNN 3×     | DaViT-Tiny   | 47.4          | +1.4                |
|             | DaViT-Small  | 49.5          | +1.0                |
|             | DaViT-Base   | 49.9          | +1.4                |
| ADE20K (mIoU)     | DaViT-Tiny   | 46.3          | +1.8                |
|             | DaViT-Small  | 48.8          | +1.2                |
|             | DaViT-Base   | 49.4          | +1.3                |

DaViT consistently surpasses Swin Transformers in both accuracy and computational efficiency for all evaluated tasks [2204.03645].

## 6. Synthesis and Implications

DaViT’s combination of local spatial window attention and global channel group attention enables explicit modeling of both fine-grained local structures and holistic contextual dependencies. Because channel attention integrates spatial information globally within each token and spatial attention preserves local relationships, these mechanisms are complementary. The architectural strategy of alternating and grouping tokens ensures all attention remains linear in both spatial and channel dimensions, supporting scaling to larger images and models. This suggests extensibility to high-resolution vision tasks and large-scale pretrained regimes, as manifested in DaViT-Giant’s results on ImageNet-1K following weakly supervised training with 1.5B image-text pairs. 

DaViT demonstrates that a dual-attention transformer design, grounded in efficient grouping and alternation, can achieve superior trade-offs among computational cost, parameter count, and task accuracy across a range of modern vision benchmarks [2204.03645].

Source: https://www.emergentmind.com/topics/dual-attention-vision-transformer-davit