---
title: Semi-Local Attention Mechanism
url: https://www.emergentmind.com/topics/semi-local-attention-sla
type: topic
---

# Semi-Local Attention Mechanism

Semi-Local Attention (SLA) is a hybrid attention mechanism that integrates a fixed-size sliding-window (local) attention with a residual linear-attention stream designed to capture contextual information from tokens outside the window. Introduced in the context of the RATTENTION model, SLA addresses intrinsic limitations of local-global attention architectures by enabling efficient context aggregation that scales favorably with sequence length, while maintaining or exceeding the accuracy of full-attention models at drastically reduced computational and memory costs [2506.15545].

## 1. Mathematical Structure of Semi-Local Attention

SLA fuses two computations: standard sliding-window attention (“SWA”) and a residual linear attention (“RLA”) stream.

### 1.1 Sliding-Window Attention

Given $Q, K, V \in \mathbb{R}^{n \times d}$ (queries, keys, and values), for each position $i \in [1, n]$, compute attention coefficients $\alpha_{i,j}$ for the window $j \in [\max(1,i-w+1), i]$:
\[
\alpha_{i,j} = \frac{\exp(Q_i K_j^\top/\sqrt d)} 
{\sum_{k=\max(1,i-w+1)}^{i} \exp(Q_i K_k^\top/\sqrt d)}
\]
The local attention output:
\[
y_i^{\rm swa} = \sum_{j=\max(1,i-w+1)}^i \alpha_{i,j} V_j
\]

### 1.2 Residual Linear Attention

A feature map $\phi: \mathbb{R}^d \rightarrow \mathbb{R}^{d'}$ satisfying $\langle \phi(x), \phi(x') \rangle \approx \exp(x^\top x'/\sqrt d)$ is employed. In practice, $\phi(x) = \exp(x)$ (elementwise), with RMSNorm. The recurrent state $S_t \in \mathbb{R}^{d' \times d}$ is updated sequentially:
\[
S_t = S_{t-1} + \phi(K_t)^\top V_t,\quad S_0 = 0
\]
At position $i$, the RLA stream reads from $S_{i-w}$:
\[
y_i^{\rm rla} = \phi(Q_i) S_{i-w}
\]
For $i \le w$, $S_{i-w}=0$.

### 1.3 Fusion

Both streams are fused:
\[
\tilde y_i = \mathrm{RMSNorm}(y_i^{\rm swa}) + \mathrm{RMSNorm}(y_i^{\rm rla}) \in \mathbb{R}^d
\]
$\tilde y_i$ is projected via $W_O \in \mathbb{R}^{d \times d}$ before the standard Transformer residual and feed-forward blocks.

## 2. Layerwise Algorithmic Workflow

A single Transformer layer with SLA operates as follows:

1. **Linear Projection:** $Q = XW_Q$, $K = XW_K$, $V = XW_V$ for layer input $X \in \mathbb{R}^{n \times d}$.
2. **Iteration (across $i=1, \ldots, n$):**
    - Compute sliding-window attention over $[i-w+1, \ldots, i]$.
    - Update $S \gets S + \phi(K_i)^\top V_i$.
    - Read residual stream $y_i^{\rm rla} = \phi(Q_i) S_{i-w}$ ($0$ if $i\leq w$).
    - Fuse: $\tilde y_i$ as above.
3. **Output:** $Y = \tilde Y W_O + X$.
4. **Feed-forward:** $Z = \mathrm{SwiGLU}(\mathrm{LN}(Y)) W_{FF} + Y$.

Architecturally, RATTENTION interleaves three SLA layers with one full-attention layer.

## 3. Computational Complexity Analysis

The table below summarizes comparative asymptotic costs, where $n=$ sequence length, $d=$ hidden size, $w=$ window size, $d'=d$:

| Mechanism             | Time Complexity            | Memory Complexity        |
|-----------------------|---------------------------|-------------------------|
| Full Attention        | $O(n^2d)$                 | $O(n^2)$                |
| Sliding-Window (SWA)  | $O(nwd)$                  | $O(nw)$                 |
| SLA (SWA + RLA)       | $O(nwd + nd^2)$           | $O(nw + d^2 + Cd)$      |

- The local part matches SWA cost; the linear part adds $O(nd^2)$ extra time and $O(d^2)$ memory.
- SLA provides a strict efficiency gain when $w \ll n$ and $d \ll n$; $S$ is a constant-size recurrent state.
- SLA sits between local and global attention, offering a trade-off unattainable by either independently.

## 4. Kernel and Implementation Considerations

RATTENTION employs two kernel optimizations to ensure SLA’s practicality on accelerators (e.g., TPU, GPU):

Source: https://www.emergentmind.com/topics/semi-local-attention-sla