---
title: Hierarchical Local-Global Transformers (HLG)
url: https://www.emergentmind.com/topics/hierarchical-local-global-transformers-hlg
type: topic
---

# Hierarchical Local-Global Transformers (HLG)

Hierarchical Local-Global Transformers (HLG) are a family of neural architectures designed to efficiently and effectively combine local and global contextual modeling via a principled alternation or fusion of localized content aggregation and global dependency reasoning, organized over explicit structural hierarchies. Initially introduced for dense visual prediction with the HLG architecture in vision [2207.09339], the local-global hierarchical paradigm has subsequently influenced sequence modeling for text [2406.02657, 2604.14442], video-language grounding [2208.14882], code representation [2303.07826], and hybrid CNN-transformer visual systems [2506.12982, 2407.13920]. The core architectural idea is to interleave or stage operations that focus on capturing fine-grained local cues (within spatial windows, temporal segments, graph neighborhoods, or syntactic/semantic substructures) with global modules that propagate information across the hierarchy or the entire domain, allowing the model to combine scalable context with rich high-capacity reasoning.

## 1. Architectural Principles and General Motivation

The primary motivation for HLG-style architectures stems from the conflict between the need for strong global context and the computational/inductive limitations of vanilla transformers and convolutional networks. Standard Vision Transformers (ViTs) compute full self-attention at every layer, which is $O(N^2)$ in the number of tokens, making them intractable for high-resolution images. CNNs, in contrast, have localized receptive fields and only capture global context at deep layers. HLGs seek a middle path by structuring the network into stages or hierarchies where local attention within small neighborhoods is alternated or fused with global attention that aggregates across the entire or large regions of the domain, thus ensuring both scalability and context retention [2207.09339, 2407.13920].

This paradigm applies broadly:

- In vision, stages are aligned with spatial scales and patches [2207.09339, 2506.12982, 2407.13920].
- In dense segmentation, local aggregation forms "superpixels" and global aggregation forms "group" (object) tokens [2409.01353].
- In text, tokens are grouped into blocks for coarse-grained global attention followed by local refinement within blocks [2406.02657].
- In recurrence-based transformers, fast modules compute local refinement and slow modules synthesize global summaries [2604.14442].
- In code modeling, hierarchical embeddings encode both local (token path) and global (statement-level) structure [2303.07826].

## 2. Canonical HLG Architectures: Vision and Language

### Vision Transformers

HLG for vision (see [2207.09339]) adopts a four-stage pyramidal backbone:

| Stage | Spatial Size          | Channel Dim ($C_i$) | #Layers ($L_i$) | Heads ($H_i$) | Window Size ($R_i$) | Dilation ($D_i$) |
|-------|----------------------|---------------------|-----------------|---------------|---------------------|------------------|
| 1     | $H/4 \times W/4$     | 96                  | 2               | 3             | 7                   | 8                |
| 2     | $H/8 \times W/8$     | 192                 | 6               | 3             | 7                   | 4                |
| 3     | $H/16 \times W/16$   | 384                 | 6               | 4             | 7                   | 2                |
| 4     | $H/32 \times W/32$   | 768                 | 2               | 4             | 7                   | 1                |

Within each stage, blocks alternate:
- Local attention within $R\times R$ non-overlapping windows (optionally with different dilation rates).
- Global attention is performed across pooled window tokens, enabling each window to access global context at a fraction of quadratic cost.

Patch-embedding between stages typically uses strided depth-wise convolution. Relative position biases facilitate spatial information [2207.09339].

### Block Transformers for Language Modeling

HLG methods in autoregressive transformers, such as Block Transformer [2406.02657], partition a sequence into blocks of length $L_B$, process blocks with global attention to form block-level embeddings, then refine within-block predictions via local attention applied to subword tokens. This structure enables dramatic reductions in KV-cache and inference latency.

The generic process is:
1. Embed input sequence into blocks: $[x_0, ..., x_{L-1}] \to \{\mathrm{Block}_i\}_{i=0}^{B-1}$, $B=L/L_B$.
2. Block encoder: full self-attention over $B$ block embeddings (global).
3. Token decoder: local self-attention within each block, leveraging block context embedding $c_i$ for each block.
4. Generation alternates global (blockwise, amortized over $L_B$ tokens) and local (per-token) computation, yielding $10-20\times$ throughput increases [2406.02657].

### Recurrence-based HLG (Two-Speed Hierarchy)

In the HRM-LM of [2604.14442], a "fast" module $f_L$ does local refinement at every iteration, while a "slow" module $f_H$ is applied every $T$ steps for global compression. Shared weights are used across time. A fusion module at the end combines the final states. This two-speed process prevents the representational collapse seen in flat shared-weight transformers.

## 3. Mathematical Formulations of Local and Global Attention

The mathematical underpinnings follow the alternation between localized and global modules.

**Local Attention** (within vision/win):
\[
Z^{(m)}_W \to Q^{(m)}, K^{(m)}, V^{(m)} \in \mathbb{R}^{R^2 \times d}
\]
\[
\mathrm{SA}_{\mathrm{local}}(Z^{(m)}_W) = Z^{(m)}_W + \mathrm{softmax}\left( \frac{Q^{(m)} K^{(m)\top}}{\sqrt{d}} + B_{\mathrm{local}} \right) V^{(m)}
\]

**Global Attention** (across windows):
\[
\text{Pool within windows: } Z_L \to Z_G \in \mathbb{R}^{M \times C}
\]
\[
Q_L = Z_L W_Q^{(G)},\quad K_G = Z_G W_K^{(G)},\quad V_G = Z_G W_V^{(G)}
\]
\[
\mathrm{SA}_{\mathrm{global}}(Z_L, Z_G) = Z_L + \mathrm{softmax}\left( \frac{Q_L K_G^\top}{\sqrt{d}} + B_{\mathrm{global}} \right) V_G
\]
[2207.09339]

**Language (Block Transformer):**
- Block-level attention over $B$ block tokens: $O(B^2)$ cost.
- Token-level attention within each block: $O(L_B^2)$ cost per block.
[2406.02657]

## 4. Variants Across Domains

### Dense Segmentation: Part and Object Hierarchies

In [2409.01353], hierarchical local-global transformers bridge pixel-to-superpixel-to-group representation, with:

- Superpixel Context Aggregation (SCA): local self-attention to merge pixels into part-level tokens.
- Group Context Aggregation (GCA): cross-attention to merge superpixels into group (object) tokens.

Alternating SCA (local) and GCA (global) stages enables both fine and holistic scene parsing.

### Hybrid CNN-Transformer Designs

DuoFormer [2506.12982, 2407.13920] illustrates an HLG structure incorporating CNN backbones for hierarchical, multiscale feature extraction, followed by dedicated local (patch-wise) and global (scale-wise) attention modules:

1. CNN stages produce feature maps at four scales.
2. Each is projected and patch-tokenized to fixed $N$ per scale.
3. Patch-attention: local MSA within each scale group.
4. Scale-attention: cross-scale MSA (often via a "scale token").
5. Experimental results in medical imaging demonstrate superior data efficiency and performance over CNN, ViT, and Swin backbones.

### Hierarchical Structure in Source Code

The Hierarchy Transformer (HiT) [2303.07826] leverages hierarchy from abstract syntax trees (CSTs): global embeddings (statement path) and local embeddings (token path) are concatenated for each input token and fed to a vanilla transformer. This supplies powerful structure-aware context, yielding clear improvements in classification, clone detection, and generation.

### Video-Language Grounding

In [2208.14882], HLG combines local (clip/phrase) and global (full video/query) representations via specialized transformers for each granularity, then fuses both within a global-local transformer and parallel cross-modal decoder.

## 5. Computational Complexity and Empirical Findings

Hierarchical local-global architectures reduce the $O(N^2)$ cost of full attention to a more scalable $O(NP + N^2/P)$ (where $P$ is window size), or, in block-based language models, convert quadratic inference-time memory and compute to nearly linear [2207.09339, 2406.02657]. For instance, fixing $R = 7$ achieves substantial complexity reduction at large $N$.

Extensive empirical evaluation underlines the utility of HLG:

- In ImageNet-1K, HLG-Transformers achieve up to 84.1% top-1 accuracy with lower FLOPs compared to full ViTs [2207.09339].
- For segmentation (ADE20K), HLG variants reach up to 49.8% mIoU; for Cityscapes, 82.9% mIoU.
- Object and instance segmentation on COCO with HLG-Backbones outperform vanilla ViT and CNN backbones.
- In Block Transformer, throughput increases by $10$–$20\times$ while maintaining comparable perplexity [2406.02657].
- In code representation, HiT yields up to $+25\%$ gains over vanilla transformer on challenging benchmarks [2303.07826].
- In video-language temporal grounding, HLG achieves state-of-the-art R@1 at IoU=0.5 on ActivityNet, Charades-STA, and TACoS [2208.14882].

Ablations consistently show that removal of either global or local modules substantially degrades performance, confirming that both are necessary [2207.09339, 2506.12982].

## 6. Implementation Strategies, Insights, and Variations

Key design choices in HLG include:

- Consistent window sizes (e.g., $R=7$) and staged dilation to connect non-neighboring regions [2207.09339].
- Pooling operations to exchange information between granularities (downsampled representations, scale tokens, or pooling across axes).
- Plug-and-play nature (especially in DuoFormer), allowing arbitrary CNN backbones to be combined with local-global transformer blocks [2506.12982, 2407.13920].
- Relative positional encoding and parameter sharing between local/global attention modules for parameter efficiency.
- Cross-modal cycle-consistency losses in video-text [2208.14882], and specialized fusion schemes (e.g., attention over local/global outputs in code) [2303.07826].
- The split of parameter budget between local and global modules is often U-shaped in quality: balanced allocation is optimal [2406.02657].
- For recurrence-based HLGs, two-speed updates and gated updates prevent representational collapse and improve stability [2604.14442].

## 7. Limitations, Open Problems, and Future Directions

Despite strong empirical results, current HLG frameworks face several open challenges:

- Scalability to very deep trees or long sequences remains a concern due to the quadratic growth of some submodules.
- Most vision HLGs operate on fixed-scale pyramids; dynamic or recursive hierarchies could improve flexibility.
- Cross-task and multi-modal adaptation (e.g., vision-language, graph-structured data) remains nascent in HLG architectures, though the generality of the local-global template suggests fertile ground.
- Limitations in parameter scaling, memory savings, and training stability in very large or data-scarce regimes are only partially understood [2604.14442].
- For code, reliance on external parsers and lack of pretraining integration (e.g., with CodeBERT) mark further work [2303.07826].
- The full theoretical underpinnings of why two-speed or local-global alternation maintains representation diversity yet avoids redundancy merit further study.

A plausible implication is that continued advances in HLG design will be central to efficient, context-rich modeling for multimodal, high-resolution, or long-context tasks.

---

**References**:  
- [2207.09339], [2407.13920], [2506.12982], [2409.01353], [2406.02657], [2604.14442], [2303.07826], [2208.14882], [2107.04735]

Source: https://www.emergentmind.com/topics/hierarchical-local-global-transformers-hlg