Semi-Local Attention Mechanism

• Semi-Local Attention (SLA) is a hybrid attention mechanism integrating fixed sliding-window and residual linear streams to capture both local and global contexts.
• It efficiently scales with sequence length by reducing computational and memory costs compared to full-attention models.
• Kernel optimizations enable SLA to run effectively on modern accelerators, making it practical for large-scale transformer architectures.

Semi-Local Attention (SLA) is a hybrid attention mechanism that integrates a fixed-size sliding-window (local) attention with a residual linear-attention stream designed to capture contextual information from tokens outside the window. Introduced in the context of the RATTENTION model, SLA addresses intrinsic limitations of local-global attention architectures by enabling efficient context aggregation that scales favorably with sequence length, while maintaining or exceeding the accuracy of full-attention models at drastically reduced computational and memory costs (Wang et al., 18 Jun 2025).

1. Mathematical Structure of Semi-Local Attention

SLA fuses two computations: standard sliding-window attention (“SWA”) and a residual linear attention (“RLA”) stream.

1.1 Sliding-Window Attention

Given $Q, K, V \in \mathbb{R}^{n \times d}$ (queries, keys, and values), for each position $i \in [1, n]$, compute attention coefficients $\alpha_{i,j}$ for the window $j \in [\max(1,i-w+1), i]$: $$\alpha_{i,j} = \frac{\exp(Q_i K_j^\top/\sqrt d)} {\sum_{k=\max(1,i-w+1)}^{i} \exp(Q_i K_k^\top/\sqrt d)}$$ The local attention output: $$y_i^{\rm swa} = \sum_{j=\max(1,i-w+1)}^i \alpha_{i,j} V_j$$

1.2 Residual Linear Attention

A feature map $\phi: \mathbb{R}^d \rightarrow \mathbb{R}^{d'}$ satisfying $$\langle \phi(x), \phi(x') \rangle \approx \exp(x^\top x'/\sqrt d)$$ is employed. In practice, $\phi(x) = \exp(x)$ (elementwise), with RMSNorm. The recurrent state $S_t \in \mathbb{R}^{d' \times d}$ is updated sequentially: $S_t = S_{t-1} + \phi(K_t)^\top V_t,\quad S_0 = 0$ At position $i$, the RLA stream reads from $S_{i-w}$: $y_i^{\rm rla} = \phi(Q_i) S_{i-w}$ For $i \le w$, $S_{i-w}=0$.

1.3 Fusion

Both streams are fused: $$\tilde y_i = \mathrm{RMSNorm}(y_i^{\rm swa}) + \mathrm{RMSNorm}(y_i^{\rm rla}) \in \mathbb{R}^d$$ $\tilde y_i$ is projected via $W_O \in \mathbb{R}^{d \times d}$ before the standard Transformer residual and feed-forward blocks.

2. Layerwise Algorithmic Workflow

A single Transformer layer with SLA operates as follows:

1. Linear Projection: $Q = XW_Q$, $K = XW_K$, $V = XW_V$ for layer input $X \in \mathbb{R}^{n \times d}$.
2. Iteration (across $i=1, \ldots, n$):
   • Compute sliding-window attention over $[i-w+1, \ldots, i]$.
   • Update $S \gets S + \phi(K_i)^\top V_i$.
   • Read residual stream $y_i^{\rm rla} = \phi(Q_i) S_{i-w}$ ($0$ if $i\leq w$).
   • Fuse: $\tilde y_i$ as above.
3. Output: $Y = \tilde Y W_O + X$.
4. Feed-forward: $Z = \mathrm{SwiGLU}(\mathrm{LN}(Y)) W_{FF} + Y$.

Architecturally, RATTENTION interleaves three SLA layers with one full-attention layer.

3. Computational Complexity Analysis

The table below summarizes comparative asymptotic costs, where $n=$ sequence length, $d=$ hidden size, $w=$ window size, $d'=d$:

Mechanism	Time Complexity	Memory Complexity
Full Attention	$O(n^2d)$	$O(n^2)$
Sliding-Window (SWA)	$O(nwd)$	$O(nw)$
SLA (SWA + RLA)	$O(nwd + nd^2)$	$O(nw + d^2 + Cd)$

• The local part matches SWA cost; the linear part adds $O(nd^2)$ extra time and $O(d^2)$ memory.
• SLA provides a strict efficiency gain when $w \ll n$ and $d \ll n$; $S$ is a constant-size recurrent state.
• SLA sits between local and global attention, offering a trade-off unattainable by either independently.

4. Kernel and Implementation Considerations

RATTENTION employs two kernel optimizations to ensure SLA’s practicality on accelerators (e.g., TPU, GPU):