GG-Transformer: Glance for Efficient Global Attention

• Glance is a key component of the GG-Transformer that performs self-attention on adaptively-dilated partitions to efficiently capture global context.
• It reduces quadratic computation by processing fixed-size groups, enabling linear complexity in high-resolution dense prediction tasks.
• Combined with the Gaze branch, which applies depth-wise convolution for local detail, Glance supports a balanced mix of global dependency and local continuity.

The Glance-and-Gaze Vision Transformer, or GG-Transformer, is a vision Transformer designed to preserve Transformer-style long-range dependency modeling while reducing the quadratic cost of standard self-attention on high-resolution feature maps. It is motivated by the observation that dense prediction tasks make the sequence length $N=HW$ very large, so the $2N^2C$ term in standard multi-head self-attention becomes prohibitive. GG-Transformer addresses this by combining two parallel branches: Glance, which performs self-attention on adaptively-dilated partitions of the input, and Gaze, which applies a depth-wise convolutional layer to compensate local image context. The result is a block that aims to retain a global receptive field with linear complexity in the number of tokens when partition size is fixed, while also restoring local continuity and fine neighborhood cues (Yu et al., 2021).

1. Motivation and problem formulation

The paper starts from the standard multi-head self-attention formulation

$$\mathrm{MSA}(X)=\mathrm{Softmax}\!\left(\frac{QK^T}{\sqrt{C}}\right)V,$$

with $Q,K,V\in\mathbb{R}^{N\times C}$, and gives its computational complexity as

$\Omega(\mathrm{MSA})=4NC^2+2N^2C.$

The problematic term is the quadratic $2N^2C$, which becomes especially costly for dense prediction because feature maps are high-resolution and $N=HW$ can be very large (Yu et al., 2021).

The paper explicitly argues that two common efficiency strategies are insufficient in different ways. Spatial reduction lowers cost but loses details, whereas local window attention is efficient but weakens global dependency modeling. The architectural objective of GG-Transformer is therefore stated in terms of three properties: global receptive field, linear complexity in the number of tokens, and no need to rely on stacked shifted-window blocks to approximate global context (Yu et al., 2021).

The design is framed through a human visual analogy. People first take in the overall scene by a glance, then inspect fine details by a gaze. GG-Transformer transfers that idea into a two-branch attention module in which global modeling and local context modeling are performed in parallel.

2. Glance: self-attention on adaptively-dilated partitions

The Glance component is the paper’s core efficiency mechanism. Rather than applying self-attention over the full dense token set, it performs attention over adaptively-dilated partitions of the feature map (Yu et al., 2021).

The token sequence is first reshaped into a 2D grid,

$$\mathbf{z}_{\ell-1}=[\mathbf{z}^{1,1}_{\ell-1}, \mathbf{z}^{1,2}_{\ell-1}, \dots, \mathbf{z}^{h,w}_{\ell-1}],$$

where $\mathbf{z}^{i,j}_{\ell-1}$ denotes the token at spatial location $(i,j)$ and $2N^2C$0. The Glance branch applies

$2N^2C$1

Partition construction is governed by a chosen partition size $2N^2C$2. The feature map is divided into $2N^2C$3 partitions, and each partition is formed by sampling tokens with dilation rate

$2N^2C$4

As a result, each partition contains only $2N^2C$5 tokens, but those tokens are spatially spread across the entire feature map rather than confined to a local window (Yu et al., 2021).

The paper emphasizes four properties of this splitting procedure. Each partition has only $2N^2C$6 tokens; tokens within a partition are globally scattered; the operation is invertible via merging; and the partition shape adapts to the current feature resolution. This allows the mechanism to split and merge features without changing the overall feature-map size.

After splitting, ordinary self-attention is applied independently inside each partition: $2N^2C$7 The outputs are then reassembled by

$2N^2C$8

followed by the usual Transformer feed-forward stage,

$2N^2C$9

The paper describes this as a modified MSA module, denoted G-MSA, which changes the token grouping rather than the attention formula itself (Yu et al., 2021).

3. Complexity reduction and global receptive field

The complexity of the Glance attention module is given as

$$\mathrm{MSA}(X)=\mathrm{Softmax}\!\left(\frac{QK^T}{\sqrt{C}}\right)V,$$0

Since $$\mathrm{MSA}(X)=\mathrm{Softmax}\!\left(\frac{QK^T}{\sqrt{C}}\right)V,$$1 is fixed and much smaller than $$\mathrm{MSA}(X)=\mathrm{Softmax}\!\left(\frac{QK^T}{\sqrt{C}}\right)V,$$2, the attention cost becomes effectively linear in $$\mathrm{MSA}(X)=\mathrm{Softmax}\!\left(\frac{QK^T}{\sqrt{C}}\right)V,$$3 rather than quadratic (Yu et al., 2021).

This is the paper’s main theoretical claim. The $$\mathrm{MSA}(X)=\mathrm{Softmax}\!\left(\frac{QK^T}{\sqrt{C}}\right)V,$$4 term remains the usual projection cost, but the attention matrix term changes from $$\mathrm{MSA}(X)=\mathrm{Softmax}\!\left(\frac{QK^T}{\sqrt{C}}\right)V,$$5 to $$\mathrm{MSA}(X)=\mathrm{Softmax}\!\left(\frac{QK^T}{\sqrt{C}}\right)V,$$6. In the intended regime, the model therefore keeps self-attention’s long-range modeling while avoiding dense token-to-token interaction over the full image.

The paper also stresses that global receptive field should not be confused with full dense attention. In Glance, each attention group is small, but each group is dilated across the whole map. A partition therefore includes positions distributed over top-left, center, bottom-right, and other distant regions. This is why the mechanism is described as having a global spatial footprint even though each partition contains only $$\mathrm{MSA}(X)=\mathrm{Softmax}\!\left(\frac{QK^T}{\sqrt{C}}\right)V,$$7 tokens (Yu et al., 2021).

A central distinction from Swin-style window attention follows directly. In local window attention, each group sees only a contiguous local neighborhood. In Glance, each group is sparse but globally distributed. This suggests that GG-Transformer is meant as a more direct substitute for exact global attention than purely local-window approximations.

4. Gaze: local context compensation and branch fusion

The paper treats Glance and Gaze as complementary rather than interchangeable. Glance captures long-range dependencies through self-attention on adaptively-dilated partitions, but this sparse grouping can miss local continuity and fine neighborhood cues. The Gaze branch is therefore introduced to compensate local image context (Yu et al., 2021).

Gaze is implemented as a depth-wise convolutional layer, defined as

$$\mathrm{MSA}(X)=\mathrm{Softmax}\!\left(\frac{QK^T}{\sqrt{C}}\right)V,$$8

where $$\mathrm{MSA}(X)=\mathrm{Softmax}\!\left(\frac{QK^T}{\sqrt{C}}\right)V,$$9 is the value tensor from Glance attention. Its role is lightweight local aggregation rather than a second global reasoning path.

When the two branches are combined, the paper gives the complexity

$Q,K,V\in\mathbb{R}^{N\times C}$0

where $Q,K,V\in\mathbb{R}^{N\times C}$1 is the convolution kernel size. The additional $Q,K,V\in\mathbb{R}^{N\times C}$2 term is presented as small relative to dense attention, so Gaze adds local inductive bias at negligible extra cost (Yu et al., 2021).

The branches operate in parallel inside the GG-Transformer block. Their functional division is stated explicitly: Glance extracts global relationships, while Gaze extracts local details. The paper further reports that the Glance branch alone is not enough, the Gaze branch alone is also not enough, and only their combination gives the best result. That empirical point is important because it rejects the interpretation that GG-Transformer is merely a sparse-attention approximation; it is instead a coupled global-local module.

The paper also compares two variants of Gaze. Fixed Gazing uses a constant kernel such as $Q,K,V\in\mathbb{R}^{N\times C}$3, whereas Adaptive Gazing uses a kernel size matched to the dilation rate $Q,K,V\in\mathbb{R}^{N\times C}$4. Adaptive gazing is reported as slightly better and is adopted in the final design because it gives a more complete view (Yu et al., 2021).

5. Architectural instantiation and empirical results

For fair comparison, the authors instantiate hierarchical GG-Transformers analogously to Swin-Transformer: the models use the same depth/width, the same hierarchical stages, the same partition size $Q,K,V\in\mathbb{R}^{N\times C}$5, and the same model scale as Swin-T and Swin-S (Yu et al., 2021). The architectural difference is concentrated in the attention block: Swin uses window attention and shifted-window attention, whereas GG-Transformer uses Glance + Gaze in a single block.

The ablation study isolates the contribution of the attention mechanism. The reported comparison is as follows.

Mechanism	Reported score
Swin window + shifted window MSA	78.50%
full MSA	79.79%
Glance only	77.21%
Gaze only	76.76%
Glance + Gaze (attention version)	79.07%
Glance + Gaze (conv version, final GG-T)	80.28%

These numbers support three claims made in the paper. First, Glance alone is better than Gaze alone, but both are insufficient by themselves. Second, combining them improves performance substantially. Third, the final Glance + Gaze design can outperform the baseline full MSA result while remaining efficient (Yu et al., 2021).

The paper also reports that replacing Swin’s attention in deeper stages with full MSA improves accuracy, which is presented as evidence of a performance gap between exact global attention and efficient local alternatives. The Glance-and-Gaze design is then interpreted as narrowing or exceeding that gap while keeping cost low.

The method is also applied beyond the Swin-style hierarchy. When GG-MSA is inserted into DeiT, the reported results improve from 72.2% to 73.8% for DeiT-T and from 79.9% to 80.5% for DeiT-S. The paper uses this result to argue that the mechanism is not tied to the Swin-style hierarchy and generalizes to other ViT backbones (Yu et al., 2021).

6. Conceptual significance and common points of confusion

The most important conceptual point is that Glance is not local window attention. It computes standard self-attention inside groups that are small but globally scattered, not contiguous. This is why the paper attributes to it both efficient global modeling and global receptive field (Yu et al., 2021).

A second frequent confusion concerns whether GG-Transformer simply replaces attention with convolution. The paper does not make that claim. The global branch remains a self-attention branch; the local branch is a depth-wise convolution added specifically because Glance can under-represent local image context. In that sense, GG-Transformer is a hybrid module in which self-attention remains the primary long-range operator.

A third point concerns the meaning of “linear complexity.” The paper’s statement is conditional: complexity becomes

$Q,K,V\in\mathbb{R}^{N\times C}$6

so it is effectively linear in $Q,K,V\in\mathbb{R}^{N\times C}$7 when $Q,K,V\in\mathbb{R}^{N\times C}$8 is fixed and much smaller than $Q,K,V\in\mathbb{R}^{N\times C}$9. This is not the same as eliminating all dependence on attention-group size; it is a replacement of quadratic dense-token interaction by partition-wise attention with fixed partition cardinality (Yu et al., 2021).

In summary, GG-Transformer is best understood as a sparse but globally distributed self-attention mechanism coupled with a lightweight local-context branch. Its architectural claim is that efficient vision Transformers for dense prediction should not choose between global context and local detail; they should separate those functions explicitly, then fuse them in a single block.