CoCAViT: Compact Vision Transformer with CoCA

• CoCAViT is a compact hybrid vision transformer that employs the Coordinator–Patch Cross Attention (CoCA) mechanism to restore global context in local window attention.
• Its architecture features a wider early CNN stage and deeper transformer stages with progressive FFN allocation, achieving up to 84.0% top-1 accuracy on ImageNet-1K while maintaining low latency.
• The model enhances out-of-distribution robustness and improves performance on tasks like object detection and semantic segmentation by efficiently aggregating global information.

CoCAViT is a compact, hybrid vision transformer backbone introduced in “CoCAViT: Compact Vision Transformer with Robust Global Coordination” (Wang et al., 7 Aug 2025). It is designed to close the robustness–efficiency gap of small models under distribution shift by restoring global coordination to windowed attention with minimal overhead. The model combines a reconfigured architecture—wider in the early CNN stage and deeper in the transformer stages, with progressive FFN capacity allocation—with a Coordinator–Patch Cross Attention (CoCA) mechanism that uses a small set of dynamic, domain-aware global tokens (“coordinators”) to aggregate and broadcast global context across local windows. At a resolution of 224×224, CoCAViT-28M achieves 84.0% top-1 accuracy on ImageNet-1K, 52.2 mAP on COCO object detection, and 51.3 mIoU on ADE20K semantic segmentation, while maintaining low latency (Wang et al., 7 Aug 2025).

1. Motivation and problem setting

CoCAViT is motivated by the observation that small, efficient visual backbones often match larger models on in-domain benchmarks like ImageNet-1K but degrade disproportionately on out-of-distribution data. Empirically, compact models suffer relative drops exceeding 45% on OOD benchmarks such as ImageNet-R, whereas larger backbones are more resilient (Wang et al., 7 Aug 2025). The model is therefore framed around a generalization deficit tied to architectural constraints rather than training alone.

Two causes are identified. First, aggressive parameter scaling in compact transformers—shallower depth and diminished head dimensions—reduces attention diversity and hierarchical abstraction, encouraging overfit to in-domain textures and styles. Second, heavy reliance on local attention, especially pure window-based multi-head self-attention, creates “information silos”: windows cannot exchange long-range context efficiently, and shifting windows only partially alleviates this in a rigid and incomplete way for complex cross-window semantics (Wang et al., 7 Aug 2025).

The design objective is correspondingly narrow and specific: retain global coordination and robust feature aggregation without incurring quadratic attention costs. In that sense, CoCAViT is not a generic efficiency-oriented ViT variant; it is a compact backbone explicitly engineered to improve OOD generalization while preserving real-time efficiency.

2. Architectural composition

The backbone is hybrid. Stage 1 is a CNN front-end, followed by hierarchical transformer stages with window attention augmented by global coordinators. The CNN stage uses convolutional patch embedding and MBConv blocks to downsample the input to $H/4 \times W/4$. This stage injects locality and translation equivariance, stabilizes low-level representation learning, and remains hardware-friendly (Wang et al., 7 Aug 2025).

After early downsampling at $1/8$ resolution, CoCAViT introduces a Global Semantic Token Generator. This module uses a parallel Squeeze-and-Excitation and spatial attention pipeline, followed by fusion through element-wise multiplication, global average pooling, and an MLP projection to produce $K$ domain-robust global coordinator tokens $G \in \mathbb{R}^{K \times D}$. The default is $K=16$, with $D$ matching the hidden dimension (Wang et al., 7 Aug 2025).

The transformer stages alternate Window Self-Attention and CoCA blocks. Each CoCA block contains Gated Global Cross Attention and Global-Coordinated Window Attention, and also includes a Convolutional GLU module for gated local mixing. Depth is increased while head dimension is reduced, but head count is maintained. The stated intent is to yield thinner and deeper transformers that improve long-range abstraction without quadratic cost. FFN capacity is allocated progressively across stages, with MLP ratios $[-, 5, 4, 3]$ for stages $[1..4]$, where stage 1 is MBConv (Wang et al., 7 Aug 2025).

At 224×224, the principal configurations are as follows.

Model	Params / FLOPs / Throughput	ImageNet-1K Top-1
CoCAViT-11M	11.4M / 2.2G / 1387 images/s	82.7%
CoCAViT-21M	20.6M / 4.1G / 902 images/s	83.6%
CoCAViT-28M	27.8M / 4.9G / 791 images/s	84.0%

For these variants, the window size is 7 in the transformer stages, and the interaction frequency is $[2, 3, -1]$, indicating how often GGCA+GCWA replaces WSA within a stage to balance compute and global coordination (Wang et al., 7 Aug 2025).

3. Coordinator–Patch Cross Attention

The central mechanism in CoCAViT is Coordinator–Patch Cross Attention. It operates on patch tokens $P \in \mathbb{R}^{N \times D}$ and a small set of global coordinators $1/8$0, with $1/8$1. CoCA proceeds in two sequential steps.

The first step is Gated Global Cross Attention. Coordinators query a joint key–value pool formed by coordinators and all patch tokens, thereby aggregating global information into the coordinators. The second step is Global-Coordinated Window Attention. Each local window performs windowed attention, but its keys and values are augmented by the updated coordinators, injecting global context back into local patches (Wang et al., 7 Aug 2025).

The attention primitive is the standard formulation

$1/8$2

For GGCA, the updated coordinators are

$1/8$3

where coordinators act as queries, keys, and values alongside patches (Wang et al., 7 Aug 2025).

For GCWA, if $1/8$4 denotes a local window, then

$1/8$5

Each window therefore mixes local interactions with coordinator-derived global cues via learned weights (Wang et al., 7 Aug 2025).

Algorithmically, a CoCA block receives patch tokens and current coordinators, updates coordinators through GGCA, partitions patch tokens into windows, augments each window’s keys and values with the updated coordinators, computes window attention, and then applies residual connections and FFN processing. Coordinators may be compressed across stages via attention-based token merging that selects the most semantically relevant coordinators through cross-attention (Wang et al., 7 Aug 2025).

A common simplification is to view CoCA as merely adding a few global tokens to window attention. The mechanism is more structured than that. The coordinators are explicitly “domain-aware” through the GS-Token generator, and are regularized by an anchor loss to maintain diversity and stability. This suggests that CoCAViT treats global tokens not as static positional carriers but as adaptive semantic anchors.

4. Global field restoration and computational profile

The paper characterizes CoCA as “restoring the global field” of pure window attention. In operational terms, this means enabling global information flow across otherwise isolated local windows without resorting to quadratic global attention. The upward path is GGCA, where coordinators compress global scene semantics and filter domain-specific noise. The downward path is GCWA, where every local window augments its keys and values with the updated coordinators, allowing patches to attend to both nearby neighbors and global anchors in a single attention operation (Wang et al., 7 Aug 2025).

The connectivity pattern is therefore “locally dense, globally sparse.” This breaks window silos while avoiding the cost of full global self-attention. The relevant complexity expressions are given explicitly. For a feature map of size $1/8$6, channels $1/8$7, window size $1/8$8, and $1/8$9 coordinators, the complexity of full multi-head self-attention is

$K$0

whereas window self-attention is

$K$1

For CoCA, the full complexity is

$K$2

Since $K$3, this is approximated as

$K$4

The added global coordination therefore scales linearly with resolution and remains far below full MSA (Wang et al., 7 Aug 2025).

This complexity profile is central to the model’s positioning. CoCAViT does not attempt to recover full global self-attention behavior exactly; it introduces a moderate overhead above WSA but avoids the quadratic term that would otherwise dominate at higher spatial resolution.

5. Training procedure and empirical results

For classification pretraining, CoCAViT is trained from scratch on ImageNet-1K at 224×224 for 300 epochs on 8 GPUs with automatic mixed precision, following the TinyViT recipe with the addition of anchor loss for coordinators (Wang et al., 7 Aug 2025). The anchor loss is

$K$5

with

$K$6

and

$K$7

where $K$8 is the batch-wise mean of coordinator tokens (Wang et al., 7 Aug 2025).

For downstream transfer, COCO 2017 object detection is performed with Cascade Mask R-CNN using a 3× schedule, and ADE20K semantic segmentation uses UperNet for 160k iterations with multi-scale testing for mIoU (Wang et al., 7 Aug 2025).

The reported ImageNet-1K, IN-Real, and IN-V2 results are:

Model	ImageNet-1K / IN-Real / IN-V2	Throughput
CoCAViT-11M	82.7 / 87.8 / 72.9	1387 img/s
CoCAViT-21M	83.6 / 88.3 / 73.7	902 img/s
CoCAViT-28M	84.0 / 88.4 / 74.0	791 img/s

On OOD benchmarks at 224×224, CoCAViT-28M reports 39.8 on ImageNet-A, 51.1 on ImageNet-R, and 40.2 on Sketch. The corresponding values for CoCAViT-21M are 38.9, 50.1, and 39.6; for CoCAViT-11M they are 34.1, 47.8, and 35.5 (Wang et al., 7 Aug 2025).

On COCO 2017 with Cascade Mask R-CNN, CoCAViT-28M attains $K$9 52.2 and $G \in \mathbb{R}^{K \times D}$0 45.2. On ADE20K with UperNet and multi-scale testing, CoCAViT-28M reaches 51.3 mIoU, while CoCAViT-21M reaches 50.8 (Wang et al., 7 Aug 2025). These results are used in the paper as evidence that the robustness-oriented design also improves dense prediction and localization-sensitive tasks.

6. Ablations, interpretation, and limitations

The paper presents an ablation path from Swin-Tiny to CoCAViT-21M. Starting from a Swin-Tiny baseline at 28.3M parameters and 81.2% top-1, replacing shifted windows with pure WSA only reduces performance to 80.1%. Adding an MBConv stage raises it to 80.6%. Adding the CoCA block raises it to 82.1%. Reconfiguring capacity through a wider convolutional stage and deeper transformer increases it further to 83.1%, and adding Convolutional GLU reaches 83.6% (Wang et al., 7 Aug 2025). In the paper’s own interpretation, CoCA contributes substantial gains by restoring global coordination, while capacity redistribution across stages enhances robustness.

Relative to compact baselines, CoCAViT-11M improves +1.2% over TinyViT-11M, +5.6% over EfficientViT-M5, and +4.0% over PVTv2-B1 on ImageNet-1K, with strong OOD gains. CoCAViT-28M matches TransNeXt-Tiny’s 84.0% while yielding nearly 2× throughput, 791 versus 421 img/s, and lower FLOPs, 4.9G versus 5.7G (Wang et al., 7 Aug 2025).

Several interpretive points follow directly from the reported design. First, CoCAViT’s robustness gains are not attributed to a single module. The wider early CNN stage increases low-level feature diversity; the deeper, thinner transformer stages strengthen hierarchical abstraction; and the progressive MLP ratios reduce late-stage redundancy. Second, CoCA’s coordinators act as shared dynamic anchors across windows, which the paper associates with cross-domain semantic consistency that window shifting cannot reliably supply (Wang et al., 7 Aug 2025).

The stated limitations are restrained. Memory footprint is not explicitly reported, though the paper notes that the linear overhead from $G \in \mathbb{R}^{K \times D}$1 coordinators and windowed operations suggests a modest memory increase relative to WSA-only designs. It also notes that CoCA’s efficacy depends on effective coordinator generation and tuning $G \in \mathbb{R}^{K \times D}$2, and that extreme resource limits may prefer fewer interactions or smaller $G \in \mathbb{R}^{K \times D}$3 (Wang et al., 7 Aug 2025). A plausible implication is that CoCAViT’s deployment envelope is broad among compact backbones, but not independent of interaction scheduling and coordinator budget.

In summary, CoCAViT advances compact ViTs by explicitly restoring global coordination to window attention through CoCA, a two-step GGCA+GCWA mechanism that aggregates global semantics into a small set of domain-robust coordinators and broadcasts them back to local windows. Its architectural corrections—wider early CNN processing, deeper and lighter transformers, and progressive MLP ratios—are presented as the structural basis for improved OOD robustness and favorable speed–accuracy trade-offs in compact vision backbones (Wang et al., 7 Aug 2025).