---
title: Long Short Distance Attention (LSDA)
url: https://www.emergentmind.com/topics/long-short-distance-attention-lsda
type: topic
---

# Long Short Distance Attention (LSDA)

Long Short Distance Attention (LSDA) encompasses a family of Transformer attention mechanisms that explicitly decompose interactions into short-range (local, neighborhood) and long-range (global, cross-block) components. This architectural paradigm aims to address the prohibitive $O(n^2)$ cost of full self-attention while maintaining the ability to model both fine-grained dependencies and global context, in both sequence and grid-based modalities. LSDA has emerged independently in natural language processing (NLP) and vision (CV) settings, with instantiations including LSG Attention [2210.15497], CrossFormer/CrossFormer++ LSDA [2108.00154, 2303.06908], Long Short-attention (LS-attention) [2505.15548], and Long-Short Range Attention (LSRA) [2004.11886]. Common to all is the structural separation—or alternation—of dense local context modeling from sparser global routing, implemented with different algorithmic schemes depending on modality and use case.

## 1. Mathematical Foundations and Variants

Several principal formalizations of LSDA exist, distinguished by the granularity of the decomposition and the mechanism for mixing local and global signals.

**Block-based Local/Sparse/Global (LSG):**  
The LSG architecture [2210.15497] operates over a sequence of length $n$ by partitioning tokens into $n_b = n / b_t$ non-overlapping blocks of size $b_t$. LSG merges three components per attention head:

- **Local:** Each token in block $i$ attends densely to all tokens in blocks $i-1, i, i+1$, comprising its immediate neighborhood. The local context size per query is at most $3 b_t$.
- **Sparse:** Two additional windows per block sample $b_t / f$ tokens each, using head-specific strategies (strided, block-strided, average-pooling, max-norm, or one-round LSH clustering).
- **Global:** $g$ global tokens, learned and prepended, attend to and are attended by all $n$ sequence tokens.

Combined, the query in block $i$ attends to up to $3b_t + 2(b_t / f) + g$ keys—a count constant in $n$. Complexity per layer is $O(n d_h)$.

**Head-based Local/Global (LS-attention, LSRA):**  
Long Short-attention (LS-attention) [2505.15548] and Long-Short Range Attention (LSRA) [2004.11886] decompose self-attention across heads:
- $H_s$ *local heads* attend to a window of $p \ll n$ neighbors ($|i-j| \leq p$), using banded or convolutional operations.
- $H_\ell$ *global heads* attend unrestrictedly (full or causal mask).
- Each head uses standard query/key/value projections, with local masks ensuring each only computes attention over its targeted region.

**Grid-based Alternating Local/Long (Vision LSDA):**  
In vision, input tokens from spatial grids ($S \times S$) are grouped:
- **SDA (Short-Distance Attention):** Each group consists of contiguous $G \times G$ spatial neighbors, with attention internally per group ($O(S^2 G^2)$ cost).
- **LDA (Long-Distance Attention):** Groups are formed by strided sampling with interval $I$; tokens that are spatially distant become group neighbors. Each group undergoes standard attention, then outputs are scattered back to native positions.
- In CrossFormer/CrossFormer++ [2108.00154, 2303.06908], SDA and LDA alternate across blocks, leveraging cross-scale embedding (CEL) and dynamic position bias (DPB).

## 2. Architectural Integration

LSDA mechanisms are incorporated at the multi-head attention (MHA) layer or block level, with the following strategies:

- **LSG [2210.15497]:** All heads use the local+sparse+global mask for their attention calculation; the number of keys is bounded for each query. Transformers can be converted post hoc by replacing full self-attention with LSG attention and expanding the positional embedding table.
- **LS-attention [2505.15548]:** The heads are statically divided (e.g., $H_s=5$ local, $H_\ell=1$ global). Per-head projections are computed, attention is run with distinct masks per head, outputs concatenated across heads, then linearly projected.
- **LSRA [2004.11886]:** Inputs are split along channels, executing global (MHSA) on half and local (light conv or dynamic conv) on the other, fusing outputs and following with feed-forward layers.
- **Vision LSDA [2108.00154, 2303.06908]:** Each Transformer block alternates between SDA and LDA; group sizes ($G$) and strides ($I$) are stage-dependent and can be made progressive (PGS), growing receptive field deeper in the network.

## 3. Computational Complexity and Efficiency

LSDA instantiates linear or near-linear attention cost in both sequence and image domains, replacing the quadratic $O(n^2)$ or $O(S^4)$ cost with $O(n)$ or $O(S^2 G^2)$:

| Variant                   | Complexity per Layer          | Local Context   | Global Context         |
|---------------------------|------------------------------|-----------------|-----------------------|
| LSG [2210.15497]          | $O(n d_h)$                   | $3b_t$          | $g$ tokens, sparse    |
| LS-attention [2505.15548] | $O(H_s n p + H_\ell n^2)$    | $p$             | Quadratic in $H_\ell$ |
| Vision LSDA [2108.00154]  | $O(S^2 G^2 d)$               | $G \times G$    | Strided $G \times G$  |
| LSRA [2004.11886]         | $O(N^2 d/2) + O(N k d/2)$    | Conv window $k$ | Global (full)         |

Empirically, LSG attention achieves training step time $\approx 1.5$ s/step and memory $\approx 32$ GB for 4096 tokens, outperforming Longformer and BigBird in speed with similar or less memory. CrossFormer-S (with LSDA) achieves attention cost reductions up to $1/64$ compared to global attention at high spatial resolutions [2210.15497, 2108.00154, 2303.06908].

## 4. Practical Adaptation and Implementation

LSDA is compatible with a wide range of Transformer architectures and can frequently be integrated with minimal changes:

- **Conversion Tools:** Scripts exist to convert HuggingFace BERT, RoBERTa, DistilBERT, and BART checkpoints to LSG attention, simply by replacing attention modules and expanding positional embeddings [2210.15497].
- **Hyperparameters:** Key tunables include block size $b_t$, sparsity factor $f$, number of global tokens $g$, group sizes $G$, stride $I$, and the local window $p$. For head-based variants, a typical split is $H_\ell=1$ global, $H_s=H-1$ local heads [2505.15548].
- **Mixed-precision and Acceleration:** Fast attention kernels (e.g., FlashAttention-2/3) are applicable for both local and global heads. LS-attention and LSRA can reduce inference latency by up to 36% at sequence length 8192 [2505.15548].
- **Progressive Design:** In vision, group size is adapted per stage, shallow layers prioritizing smaller groups (local attention), deep layers growing global context (PGS) [2303.06908].

## 5. Empirical Evaluation and Comparative Performance

LSDA-based models consistently outperform or match standard full-attention and windowed/sparse baselines across domains:

- **NLP (LSG, LS-attention, LSRA):**  
  - LSG (block=128, $f$=4) matches or exceeds Longformer/BigBird in classification and summarization on long documents. For BERT-class models adapted with LSG, masked LM accuracy degrades minimally—unlike catastrophic collapse for vanilla models extrapolated to longer sequences [2210.15497].
  - LS-attention reduces LM perplexity to about $2/5$ of QK-norm stabilizers and matches FlashAttention at $1/20$ the GPU-hours [2505.15548].
  - Lite Transformer with LSRA exceeds baselines by 1–2 BLEU for translation and 1.8 PPL for language modeling under strict compute constraints [2004.11886].
- **Vision (LSDA, CrossFormer/++):**
  - LSDA improves Top-1 ImageNet-1K accuracy by 0.6–1.2% over Swin/PVT-style baselines; removal of either SDA or LDA degrades accuracy by over 1%. On COCO, CrossFormer++-S (LSDA backbone) attains +0.7 AP over CrossFormer-S [2303.06908, 2108.00154].
  - Ablation replacing LSDA with alternative sparse attentions in CrossFormer++ confirms the unique gain (+0.4–1.4% Top-1) of the local/global alternation [2303.06908].

## 6. Theoretical and Empirical Rationale

LSDA schemes are motivated by the inadequacies of global MHSA alone, especially for:

- **Short-range dependency binding:** Standard global MHSA is rank-deficient for expressing dense local banded dependencies when $n \gg d$, leading to logit explosion and instability [2505.15548].
- **Computation-accuracy tradeoff:** Local-only attention (e.g. windows, convolution) sacrifices global context, while global-only is inefficient and numerically brittle. By decomposing, each submodule operates in its optimal regime.
- **Adaptability:** LSDA variants improve robustness to input length/distribution shift (e.g., LSG adaptation allows direct extrapolation to longer sequences without pretraining, provided positional embeddings are extended appropriately [2210.15497]).

## 7. Limitations and Prospects

LSDA approaches are not without tradeoffs:

- **Hyperparameter dependence:** Optimal block/group/window sizes and head splits are task- and distribution-specific; there is no universally superior setting. Performance can be sensitive to these parameters.
- **Scalability:** While asymptotic cost is reduced, absolute compute may remain significant at extreme sequence/grid sizes due to constants (e.g., $b_t$, $G$).
- **Heterogeneous architectures:** Adapting conversion tools and LSDA modules across nonstandard Transformer architectures may require per-variant engineering effort [2210.15497].
- **Dynamic allocation:** Static assignment (fixed local/global splits) can be suboptimal. Future directions propose learnable or mixture-of-expert-based splits, continuous routing, or block-wise relative position embedding [2210.15497, 2303.06908].

Extensions being investigated include mixing block-local/sparse/global scaffolding with dynamic convolution or kernel attention methods, integrating relative positional encoding within blocks, and dynamic or learned allocation of global attention capacity.

---

The LSDA paradigm—across LSG attention, head-based local/global decomposition, and vision block alternation—demonstrates consistent benefits in efficiency, stability, and accuracy for both language and vision transformers, establishing itself as a universal principle for scalable attention architectures [2210.15497, 2108.00154, 2303.06908, 2505.15548, 2004.11886].

Source: https://www.emergentmind.com/topics/long-short-distance-attention-lsda