---
title: 'GG-Transformer: Glance for Efficient Global Attention'
url: https://www.emergentmind.com/topics/glance
type: topic
---

# GG-Transformer: Glance for Efficient Global Attention

The **Glance-and-Gaze Vision Transformer**, or **GG-Transformer**, is a vision Transformer designed to preserve Transformer-style long-range dependency modeling while reducing the quadratic cost of standard self-attention on high-resolution feature maps. It is motivated by the observation that dense prediction tasks make the sequence length \(N=HW\) very large, so the \(2N^2C\) term in standard multi-head self-attention becomes prohibitive. GG-Transformer addresses this by combining two parallel branches: **Glance**, which performs self-attention on **adaptively-dilated partitions** of the input, and **Gaze**, which applies a **depth-wise convolutional layer** to compensate local image context. The result is a block that aims to retain a **global receptive field** with **linear complexity in the number of tokens** when partition size is fixed, while also restoring local continuity and fine neighborhood cues [2106.02277].

## 1. Motivation and problem formulation

The paper starts from the standard multi-head self-attention formulation
\[
\mathrm{MSA}(X)=\mathrm{Softmax}\!\left(\frac{QK^T}{\sqrt{C}}\right)V,
\]
with \(Q,K,V\in\mathbb{R}^{N\times C}\), and gives its computational complexity as
\[
\Omega(\mathrm{MSA})=4NC^2+2N^2C.
\]
The problematic term is the quadratic \(2N^2C\), which becomes especially costly for dense prediction because feature maps are high-resolution and \(N=HW\) can be very large [2106.02277].

The paper explicitly argues that two common efficiency strategies are insufficient in different ways. **Spatial reduction** lowers cost but loses details, whereas **local window attention** is efficient but weakens global dependency modeling. The architectural objective of GG-Transformer is therefore stated in terms of three properties: **global receptive field**, **linear complexity in the number of tokens**, and **no need to rely on stacked shifted-window blocks to approximate global context** [2106.02277].

The design is framed through a human visual analogy. People first take in the overall scene by a **glance**, then inspect fine details by a **gaze**. GG-Transformer transfers that idea into a two-branch attention module in which global modeling and local context modeling are performed in parallel.

## 2. Glance: self-attention on adaptively-dilated partitions

The **Glance** component is the paper’s core efficiency mechanism. Rather than applying self-attention over the full dense token set, it performs attention over **adaptively-dilated partitions** of the feature map [2106.02277].

The token sequence is first reshaped into a 2D grid,
\[
\mathbf{z}_{\ell-1}=[\mathbf{z}^{1,1}_{\ell-1}, \mathbf{z}^{1,2}_{\ell-1}, \dots, \mathbf{z}^{h,w}_{\ell-1}],
\]
where \(\mathbf{z}^{i,j}_{\ell-1}\) denotes the token at spatial location \((i,j)\) and \(hw=N\). The Glance branch applies
\[
\mathbf{z}^{\dagger}_{\ell-1}=\mathrm{AdaptivelyDilatedSplitting}(\mathbf{z}_{\ell-1}).
\]

Partition construction is governed by a chosen partition size \(M\times M\). The feature map is divided into \(\frac{h}{M}\times\frac{w}{M}\) partitions, and each partition is formed by sampling tokens with dilation rate
\[
\left(\frac{h}{M},\frac{w}{M}\right).
\]
As a result, each partition contains only \(M^2\) tokens, but those tokens are **spatially spread across the entire feature map** rather than confined to a local window [2106.02277].

The paper emphasizes four properties of this splitting procedure. **Each partition has only \(M^2\) tokens**; **tokens within a partition are globally scattered**; **the operation is invertible via merging**; and **the partition shape adapts to the current feature resolution**. This allows the mechanism to split and merge features without changing the overall feature-map size.

After splitting, ordinary self-attention is applied independently inside each partition:
\[
\mathbf{z}^{\prime,i,j}_{\ell}= \mathrm{MSA}\big(\mathrm{LN}(\mathbf{z}^{\dagger,i,j}_{\ell-1})\big) + \mathbf{z}^{\dagger,i,j}_{\ell-1}.
\]
The outputs are then reassembled by
\[
\mathbf{z}^{\prime}_{\ell} = \mathrm{Merging}(\mathbf{z}^{\prime,1,1}_{\ell},\dots,\mathbf{z}^{\prime,\frac{h}{M},\frac{w}{M}}_{\ell}),
\]
followed by the usual Transformer feed-forward stage,
\[
\mathbf{z}_{\ell} = \mathrm{MLP}\big(\mathrm{LN}(\mathbf{z}^{\prime}_{\ell})\big) + \mathbf{z}^{\prime}_{\ell}.
\]

The paper describes this as a modified MSA module, denoted **G-MSA**, which changes the **token grouping** rather than the attention formula itself [2106.02277].

## 3. Complexity reduction and global receptive field

The complexity of the Glance attention module is given as
\[
\Omega(\mathrm{G\text{-}MSA})=4hwC^2+2M^2hwC =4NC^2+2M^2NC.
\]
Since \(M\) is fixed and much smaller than \(N\), the attention cost becomes effectively **linear in \(N\)** rather than quadratic [2106.02277].

This is the paper’s main theoretical claim. The \(4NC^2\) term remains the usual projection cost, but the attention matrix term changes from \(2N^2C\) to \(2M^2NC\). In the intended regime, the model therefore keeps self-attention’s long-range modeling while avoiding dense token-to-token interaction over the full image.

The paper also stresses that **global receptive field** should not be confused with full dense attention. In Glance, each attention group is small, but each group is **dilated across the whole map**. A partition therefore includes positions distributed over top-left, center, bottom-right, and other distant regions. This is why the mechanism is described as having a global spatial footprint even though each partition contains only \(M^2\) tokens [2106.02277].

A central distinction from Swin-style window attention follows directly. In local window attention, each group sees only a contiguous local neighborhood. In Glance, each group is sparse but globally distributed. This suggests that GG-Transformer is meant as a more direct substitute for exact global attention than purely local-window approximations.

## 4. Gaze: local context compensation and branch fusion

The paper treats **Glance** and **Gaze** as complementary rather than interchangeable. Glance captures **long-range dependencies** through self-attention on adaptively-dilated partitions, but this sparse grouping can miss **local continuity and fine neighborhood cues**. The **Gaze** branch is therefore introduced to compensate local image context [2106.02277].

Gaze is implemented as a **depth-wise convolutional layer**, defined as
\[
\mathrm{Gaze}(X)=\mathrm{DepthwiseConv2d}(\mathrm{Merging}(V)),
\]
where \(V\) is the value tensor from Glance attention. Its role is lightweight local aggregation rather than a second global reasoning path.

When the two branches are combined, the paper gives the complexity
\[
\Omega(\mathrm{GG\text{-}MSA})=4NC^2+2M^2NC+k^2NC,
\]
where \(k\) is the convolution kernel size. The additional \(k^2NC\) term is presented as small relative to dense attention, so Gaze adds local inductive bias at negligible extra cost [2106.02277].

The branches operate **in parallel** inside the GG-Transformer block. Their functional division is stated explicitly: **Glance** extracts global relationships, while **Gaze** extracts local details. The paper further reports that the **Glance branch alone is not enough**, the **Gaze branch alone is also not enough**, and **only their combination gives the best result**. That empirical point is important because it rejects the interpretation that GG-Transformer is merely a sparse-attention approximation; it is instead a coupled global-local module.

The paper also compares two variants of Gaze. **Fixed Gazing** uses a constant kernel such as \(3\times3\), whereas **Adaptive Gazing** uses a kernel size matched to the dilation rate \((h/M,w/M)\). Adaptive gazing is reported as **slightly better** and is adopted in the final design because it gives a more complete view [2106.02277].

## 5. Architectural instantiation and empirical results

For fair comparison, the authors instantiate hierarchical GG-Transformers analogously to Swin-Transformer: the models use the **same depth/width**, the **same hierarchical stages**, the **same partition size \(M=7\)**, and the **same model scale as Swin-T and Swin-S** [2106.02277]. The architectural difference is concentrated in the attention block: Swin uses window attention and shifted-window attention, whereas GG-Transformer uses **Glance + Gaze in a single block**.

The ablation study isolates the contribution of the attention mechanism. The reported comparison is as follows.

| Mechanism | Reported score |
|---|---:|
| Swin window + shifted window MSA | 78.50% |
| full MSA | 79.79% |
| Glance only | 77.21% |
| Gaze only | 76.76% |
| Glance + Gaze (attention version) | 79.07% |
| Glance + Gaze (conv version, final GG-T) | 80.28% |

These numbers support three claims made in the paper. First, **Glance alone is better than Gaze alone**, but both are insufficient by themselves. Second, combining them improves performance substantially. Third, the final **Glance + Gaze** design can outperform the baseline full MSA result while remaining efficient [2106.02277].

The paper also reports that replacing Swin’s attention in deeper stages with full MSA improves accuracy, which is presented as evidence of a performance gap between exact global attention and efficient local alternatives. The Glance-and-Gaze design is then interpreted as narrowing or exceeding that gap while keeping cost low.

The method is also applied beyond the Swin-style hierarchy. When GG-MSA is inserted into DeiT, the reported results improve from **72.2% to 73.8%** for **DeiT-T** and from **79.9% to 80.5%** for **DeiT-S**. The paper uses this result to argue that the mechanism is **not tied to the Swin-style hierarchy** and generalizes to other ViT backbones [2106.02277].

## 6. Conceptual significance and common points of confusion

The most important conceptual point is that **Glance is not local window attention**. It computes standard self-attention inside groups that are **small but globally scattered**, not contiguous. This is why the paper attributes to it both **efficient global modeling** and **global receptive field** [2106.02277].

A second frequent confusion concerns whether GG-Transformer simply replaces attention with convolution. The paper does not make that claim. The global branch remains a self-attention branch; the local branch is a depth-wise convolution added specifically because Glance can under-represent local image context. In that sense, GG-Transformer is a hybrid module in which self-attention remains the primary long-range operator.

A third point concerns the meaning of “linear complexity.” The paper’s statement is conditional: complexity becomes
\[
4NC^2+2M^2NC,
\]
so it is effectively linear in \(N\) **when \(M\) is fixed** and much smaller than \(N\). This is not the same as eliminating all dependence on attention-group size; it is a replacement of quadratic dense-token interaction by partition-wise attention with fixed partition cardinality [2106.02277].

In summary, GG-Transformer is best understood as a **sparse but globally distributed self-attention mechanism** coupled with a **lightweight local-context branch**. Its architectural claim is that efficient vision Transformers for dense prediction should not choose between global context and local detail; they should separate those functions explicitly, then fuse them in a single block.

Source: https://www.emergentmind.com/topics/glance