AttentionViG: Cross-Attention Vision GNN

• AttentionViG is a vision GNN backbone that integrates CNN-style local processing with cross-attention-based dynamic node–neighbor aggregation for improved semantic message passing.
• It employs a fixed sparse vision graph attention (SVGA) to define neighbor connectivity, achieving competitive SOTA performance with low computational costs on ImageNet-1K, COCO, and ADE20K.
• Empirical results and ablations show that its learned per-neighbor relevance weights mitigate limitations of crude graph construction, enhancing classification, object detection, and segmentation outcomes.

AttentionViG is a vision backbone that combines CNN-style local processing with graph neural network (GNN)–style non-local message passing, built around a cross-attention-based node–neighbor aggregation mechanism for Vision GNNs (ViGs). It is presented as a general, architecture-agnostic aggregation function for ViGs, and instantiated in a multi-stage CNN–GNN hybrid architecture that achieves SOTA performance on ImageNet‑1K, COCO, and ADE20K with competitive efficiency (Gedik et al., 29 Sep 2025).

1. Problem setting and design objective

AttentionViG is motivated by limitations in CNNs, ViTs, and earlier ViGs. Convolutions are strictly local operators; global context is captured only indirectly via depth and pooling. ViTs use global self-attention over image patches, which models long-range dependencies well, but has quadratic complexity in the number of patches and weak inductive bias. ViGs treat image patches as graph nodes and perform message passing, but their performance depends strongly on graph construction and on node–neighbor aggregation (Gedik et al., 29 Sep 2025).

The paper separates two design choices in ViGs. The first is graph construction: vanilla ViG uses k‑nearest neighbors in feature space; MobileViG uses a fixed sparse criss-cross pattern (SVGA); GreedyViG and related models refine static sparsity with heuristics. The second is aggregation: Max‑Relative, EdgeConv, GIN, and GraphSAGE typically use simple symmetric operations over neighbors and do not learn per-neighbor importances. If graph construction is imperfect, these schemes cannot down-weight or ignore irrelevant neighbors (Gedik et al., 29 Sep 2025).

The limitation is explicit in the Max‑Relative aggregation used in vanilla ViG and MobileViG: $$\mathbf{x}_i' = \mathbf{W} \left[ \mathbf{x}_i,\ \max_{j \in \mathcal{N}(i)} (\mathbf{x}_j - \mathbf{x}_i) \right]$$ where every neighbor contributes through the max. The design goal of AttentionViG is therefore to provide a general-purpose aggregation function that explicitly learns per-neighbor relevance weights, is compatible with either dynamic or static graph construction, can recover useful semantic structure even when the graph is crude, and maintains efficiency comparable to existing ViGs (Gedik et al., 29 Sep 2025).

2. Cross-attention-based dynamic neighbor aggregation

At GNN layer $l$, node features are

$$\{\mathbf{x}_0^l, \dots, \mathbf{x}_{N-1}^l\}, \quad \mathbf{x}_i^l \in \mathbb{R}^{d},$$

and for each node $i$, the neighbor set $\mathcal{N}(\mathbf{x}_i^l)$ has size $M$, with neighbor features $\mathbf{y}_{i,j}^l$. AttentionViG uses asymmetric cross-attention: the query comes from the central node, while keys and values come from its neighbors (Gedik et al., 29 Sep 2025).

The projections are

$$\mathbf{q}_i^l = \mathbf{Q}^l \mathbf{x}_i^l, \qquad \mathbf{k}_{i,j}^l = \mathbf{K}^l \mathbf{y}_{i,j}^l, \qquad \mathbf{v}_{i,j}^l = \mathbf{V}^l \mathbf{y}_{i,j}^l.$$

Relevance between node $i$ and neighbor $j$ is computed via cosine similarity: $l$0 This similarity is then mapped to an attention weight with an exponential affinity function: $l$1 Here $l$2 is learned per cross-attention layer, and the implementation optimizes $l$3 for stability (Gedik et al., 29 Sep 2025).

Aggregation is a weighted sum of values, fused with the original node feature: $l$4 with $l$5 as GeLU. This produces dynamic neighbor aggregation because $l$6 is recomputed for each node and layer, even if the underlying adjacency is static (Gedik et al., 29 Sep 2025).

Several distinctions are central. Relative to Transformer self-attention, AttentionViG does not perform full self-attention over all tokens, and it does not normalize over all neighbors with a softmax. Relative to GAT, it uses separate $l$7, cosine similarity, and an exponential affinity without neighbor-wise competition. A common misconception is to equate “dynamic neighbor aggregation” with dynamic graph construction. In AttentionViG, the neighbor set can remain fixed, while the contribution of each neighbor is dynamic through $l$8 (Gedik et al., 29 Sep 2025).

3. Network architecture and message-passing pathway

AttentionViG is a multi-scale, hybrid CNN–GNN architecture comprising a convolutional stem, Inverted Residual Blocks (IRB) for local processing, Grapher layers for non-local message passing over a sparse graph, downsampling blocks, and a classification head with global pooling and MLP (Gedik et al., 29 Sep 2025).

The stem consists of two $l$9 convolutions with stride 2, each followed by BatchNorm and GeLU. This reduces spatial resolution with overall stride 4 and outputs a feature map whose spatial locations are treated as graph nodes. A graph is then built using Sparse Vision Graph Attention (SVGA): for each node, neighbors are chosen in a fixed criss-cross pattern along horizontal and vertical directions with stride 2. No dynamic kNN search is used (Gedik et al., 29 Sep 2025).

The Grapher is the fundamental graph module. It applies Conditional Positional Encoding (CPE), then the cross-attention aggregation, then a Feed-Forward Network (FFN) with expansion ratio 4 and GeLU: $$\{\mathbf{x}_0^l, \dots, \mathbf{x}_{N-1}^l\}, \quad \mathbf{x}_i^l \in \mathbb{R}^{d},$$0 IRBs are MobileNetV2-style blocks with $$\{\mathbf{x}_0^l, \dots, \mathbf{x}_{N-1}^l\}, \quad \mathbf{x}_i^l \in \mathbb{R}^{d},$$1 expansion, depthwise $$\{\mathbf{x}_0^l, \dots, \mathbf{x}_{N-1}^l\}, \quad \mathbf{x}_i^l \in \mathbb{R}^{d},$$2, $$\{\mathbf{x}_0^l, \dots, \mathbf{x}_{N-1}^l\}, \quad \mathbf{x}_i^l \in \mathbb{R}^{d},$$3 projection, residual connection, expansion ratio 4, and GeLU. Downsampling between stages uses $$\{\mathbf{x}_0^l, \dots, \mathbf{x}_{N-1}^l\}, \quad \mathbf{x}_i^l \in \mathbb{R}^{d},$$4 convolution with stride 2 and BatchNorm (Gedik et al., 29 Sep 2025).

There are four stages. For AttentionViG-S, the configuration is: Stage 1 IRB×2, Grapher×2, $$\{\mathbf{x}_0^l, \dots, \mathbf{x}_{N-1}^l\}, \quad \mathbf{x}_i^l \in \mathbb{R}^{d},$$5; Stage 2 IRB×2, Grapher×2, $$\{\mathbf{x}_0^l, \dots, \mathbf{x}_{N-1}^l\}, \quad \mathbf{x}_i^l \in \mathbb{R}^{d},$$6; Stage 3 IRB×6, Grapher×2, $$\{\mathbf{x}_0^l, \dots, \mathbf{x}_{N-1}^l\}, \quad \mathbf{x}_i^l \in \mathbb{R}^{d},$$7; Stage 4 IRB×2, Grapher×2, $$\{\mathbf{x}_0^l, \dots, \mathbf{x}_{N-1}^l\}, \quad \mathbf{x}_i^l \in \mathbb{R}^{d},$$8. For AttentionViG-M: IRB×4, Grapher×2, $$\{\mathbf{x}_0^l, \dots, \mathbf{x}_{N-1}^l\}, \quad \mathbf{x}_i^l \in \mathbb{R}^{d},$$9; IRB×4, Grapher×2, $i$0; IRB×12, Grapher×2, $i$1; IRB×4, Grapher×2, $i$2. For AttentionViG-B: IRB×5, Grapher×2, $i$3; IRB×5, Grapher×2, $i$4; IRB×15, Grapher×2, $i$5; IRB×5, Grapher×2, $i$6 (Gedik et al., 29 Sep 2025).

The number of Grapher layers is fixed at two per stage, based on SVGA’s property that two hops suffice for global connectivity. In a single Grapher, each node sees its SVGA neighbors; after two Grapher layers, information from many distant nodes can flow to each node. The architecture therefore replaces quadratic global self-attention with sparse graph attention while preserving non-local context through staged message passing (Gedik et al., 29 Sep 2025).

4. Complexity, efficiency, and scaling behavior

Let $i$7 denote the number of nodes, $i$8 the number of neighbors per node, and $i$9 the feature dimension. Query computation costs $\mathcal{N}(\mathbf{x}_i^l)$0; keys and values cost $\mathcal{N}(\mathbf{x}_i^l)$1; cosine similarities, exponentials, and weighted sums cost $\mathcal{N}(\mathbf{x}_i^l)$2. Because $\mathcal{N}(\mathbf{x}_i^l)$3 is a fixed constant determined by SVGA and $\mathcal{N}(\mathbf{x}_i^l)$4, the overall complexity of the Grapher is linear in the number of patches and hence linear in input resolution (Gedik et al., 29 Sep 2025).

This complexity profile is the central efficiency claim. AttentionViG uses sparse cross-attention over SVGA neighbors rather than quadratic token-to-token attention. In ImageNet classification, AttentionViG-S uses 12.3M parameters and 1.6 GFLOPs for 81.3% top‑1; AttentionViG-M uses 22.2M parameters and 3.2 GFLOPs for 83.1% top‑1; AttentionViG-B uses 32.3M parameters and 4.8 GFLOPs for 83.9% top‑1 (Gedik et al., 29 Sep 2025).

The same section of results positions the model against prior efficient backbones. PViG-M reports 51.7M parameters, 9.0G FLOPs, and 83.1%; PViHGNN-M reports 52.4M parameters, 11.6G FLOPs, and 83.4%; EfficientFormerV2-L reports 26.1M parameters, 2.6G FLOPs, and 83.3%; ConvNeXt-S reports 50.0M parameters, 12.9G FLOPs, and 83.1%. AttentionViG-B reaches 83.9% with 32.3M parameters and 4.8 GFLOPs, which the paper presents as competitive efficiency with strong accuracy (Gedik et al., 29 Sep 2025).

A common misunderstanding is to read AttentionViG as a dense attention model implemented on a graph. The architecture instead uses a sparse neighborhood defined by SVGA and learns dynamic edge weights only within that neighborhood. Its efficiency therefore comes from the combination of fixed sparse connectivity and learned aggregation, not from replacing sparsity with full attention (Gedik et al., 29 Sep 2025).

5. Training, transfer, and empirical results

For ImageNet‑1K classification, the model is trained on 16× NVIDIA A100 for 300 epochs with batch size 2048, AdamW, initial learning rate $\mathcal{N}(\mathbf{x}_i^l)$5, cosine annealing schedule, input size $\mathcal{N}(\mathbf{x}_i^l)$6, and hard distillation from a RegNetY‑16GF teacher following DeiT. For COCO detection and instance segmentation, the backbone is pretrained on ImageNet and used in Mask R‑CNN with a 12-epoch $\mathcal{N}(\mathbf{x}_i^l)$7 schedule, batch size 16, AdamW, initial learning rate $\mathcal{N}(\mathbf{x}_i^l)$8, weight decay 0.05, learning-rate decay by $\mathcal{N}(\mathbf{x}_i^l)$9 at epochs 8 and 11, and input size $M$0. For ADE20K semantic segmentation, the pretrained backbone is used in Semantic FPN for 40,000 iterations with batch size 32, AdamW, initial learning rate $M$1, weight decay $M$2, polynomial decay with power 0.9, and input size $M$3 (Gedik et al., 29 Sep 2025).

On ImageNet‑1K, AttentionViG-S at 12.3M parameters and 1.6G FLOPs achieves 81.3% top‑1, which is reported as +2.5% over PViG‑Ti at 78.2%, +1.8% over PViHGNN‑Ti at 78.9%, and +0.2% over GreedyViG‑S at 81.1%. AttentionViG-M at 22.2M parameters and 3.2G FLOPs reaches 83.1% top‑1. AttentionViG-B at 32.3M parameters and 4.8G FLOPs reaches 83.9% top‑1, on par with GreedyViG‑B at 83.9% while using fewer FLOPs (Gedik et al., 29 Sep 2025).

On COCO 2017, AttentionViG-S obtains $M$4 and $M$5. AttentionViG-B obtains $M$6 and $M$7. On ADE20K, AttentionViG-S reports mIoU = 43.8 and AttentionViG-B reports mIoU = 47.8. The paper states that these results show the learned cross-attention aggregation transfers well to dense prediction tasks (Gedik et al., 29 Sep 2025).

A notable implementation detail is that the $M$8 parameters are frozen during downstream fine-tuning to avoid catastrophic forgetting of pretraining statistics. When $M$9 is not frozen, AP drops by about 0.5–0.9 points on COCO, and mIoU drops by about 0.8 on ADE20K in the ablation reported for AttentionViG-S and AttentionViG-B (Gedik et al., 29 Sep 2025).

6. Ablations, interpretation, and relation to adjacent attention-based ViGs

The aggregation-function ablation is one of the clearest arguments for the method. Within vanilla ViG on ImageNet‑1K, GIN reports 72.8% top‑1 at 1.3G FLOPs, Max‑Relative 73.9% at 1.3G, GraphSAGE 74.0% at 1.6G, EdgeConv 74.3% at 2.4G, and Cross-Attention 74.3% at 1.6G. Cross-attention therefore matches EdgeConv’s accuracy with only about 66% of its FLOPs, while outperforming GIN, Max‑Relative, and GraphSAGE (Gedik et al., 29 Sep 2025).

The normalization ablation further distinguishes AttentionViG from standard attention design. Softmax yields 80.8% top‑1 in AttentionViG-S; exponential affinity normalized by $\mathbf{y}_{i,j}^l$0 yields 80.7%; exponential affinity without normalization yields 81.3%. The paper interprets neighbor-count normalization as oversmoothing and presents the non-softmax formulation as more expressive because each edge weight is determined independently rather than through competition (Gedik et al., 29 Sep 2025).

Head configuration and learned temperature analysis are similarly specific. A single head with full dimension yields 81.0%; fixed head dimension 6 or 8 yields 81.1%; 8 heads yields 81.3%. Learned $\mathbf{y}_{i,j}^l$1 values after ImageNet training for AttentionViG-S are typically in the range 4–30. Early and mid-level layers often have sharper affinities, the very first cross-attention is relatively soft, and deeper layers are also softer (Gedik et al., 29 Sep 2025).

Heatmap visualization of query–key cosine similarity shows high similarity concentrated on semantically related regions, even when those regions are spatially distant. This is the empirical basis for the claim that cross-attention mitigates crude graph construction by learning semantic neighbor relevance (Gedik et al., 29 Sep 2025).

Within adjacent literature, MobileViG introduced SVGA as a fixed, structured sparse graph with row/column connections and no kNN search, using roll operations and max-relative graph convolution for mobile deployment (Munir et al., 2023). A different line of work, also named ViG, replaced softmax self-attention with Gated Linear Attention and bidirectional gating locality injection to obtain linear-complexity visual sequence learning (Liao et al., 2024). In the broader literature represented here, graph-structured or attention-guided visual modeling also includes “Attention Graph” for semantic scanpaths and saliency (Yang et al., 11 Mar 2025), instruction-aligned visual attention for LVLM decoding (Li et al., 24 Mar 2025), and attention-guided CAM for ViT explanation (Leem et al., 2024). This suggests that, in the present usage, “AttentionViG” denotes a specific Vision GNN architecture centered on cross-attention-based dynamic neighbor aggregation rather than a generic label for all attention-based visual graph methods.