---
title: Document-Level Sparse Attention
url: https://www.emergentmind.com/topics/document-level-sparse-attention
type: topic
---

# Document-Level Sparse Attention

Searching arXiv for relevant papers on document-level sparse attention.
Document-level sparse attention denotes a family of attention mechanisms for long-sequence transformers in which the full all-to-all interaction pattern is replaced, augmented, or guided by selective connectivity at document scale. The motivation is consistent across retrieval, summarization, translation, document understanding, information extraction, and long-context generation: standard dense self-attention scales quadratically with sequence length and becomes a practical bottleneck as inputs move from passage length to full documents. In the literature, document-level sparsity is realized through several distinct designs, including local sliding windows, asymmetric query-document coupling, blockwise masks, hierarchical sentence selection, structure-derived global tokens, query-adaptive token retrieval, and evidence-guided weighting [2312.17649]. These mechanisms differ in whether they primarily target computational efficiency, inductive bias, or semantic selectivity, but they share the underlying premise that long documents rarely require every token to interact with every other token.

## 1. Quadratic attention and the document-length regime

A standard transformer computes attention from query, key, and value matrices through scaled dot-product attention. In the formulation used for cross-encoder re-ranking, this is written as
\[
O \;=\; \Attention(Q, K, V) \;=\; \softmax\left( \dfrac{QK^T}{\sqrt{h}} \right) V\,.
\]
With total sequence length \(s\), dense self-attention computes an \(s\times s\) attention matrix in every layer, so both memory and compute scale quadratically in sequence length [2312.17649]. This scaling is manageable for passage-length inputs but substantially more problematic for full documents. In one document re-ranking setup, passages are truncated to **512 tokens**, whereas documents are processed up to **4096 tokens**, and the methodological consequence is explicit: at passage length, dense cross-encoders are expensive but still feasible; at document length, prior work resorted to cropping, splitting, MaxP, or sparse architectures [2312.17649].

The same bottleneck appears in sequence-to-sequence settings. For summarization, encoder-decoder attention has complexity
\[
O(MN)=O(MN_1N_2),
\]
with \(M\) target length, \(N\) source length, \(N_1\) source sentences, and \(N_2\) average words per sentence [2109.03888]. For document-level machine translation, standard attention is given as
\[
\operatorname{attn}\left(q_t,\{k_i\},\{v_i\}\right)=\sum_{i=1}^{N} \frac{\exp\left(q_t \cdot k_i \right)} {\sum_{j=1}^{N}\exp\left(q_t \cdot k_{j} \right)}v_i^\top,
\]
and its cost is quadratic in \(N\) [2210.08431]. A related document-MT study likewise states that, because the architecture uses self-attention and cross-attention, “the memory usage is \(O(L^2)\) with \(L\) being the sequence length,” and ties long-context degradation partly to attention becoming less focused on the current sentence [2306.05183].

This body of work establishes document-level sparse attention as a response to a specific regime shift: once sequence lengths reach document scale, the question is no longer merely whether full attention is expressive, but whether the same token-interaction budget is necessary, affordable, or even desirable.

## 2. Canonical sparse patterns at document scale

The most common document-level sparse pattern is local sliding-window attention. In the cross-encoder formulation, if the window size is \(w\), each token attends to \(2w+1\) positions: \(w\) tokens on the left, itself, and \(w\) tokens on the right. The paper formalizes this as
\[
O \;=\; \Attention_w(Q, K, V) \;=\; \softmax\left( \dfrac{Q \boxdot_w K^T}{\sqrt{h} \right) \odot_w V\,,
\]
with zero-padding when windows run outside sequence bounds. The theoretical complexity drops from \(\mathcal{O}(s^2)\) space and \(\mathcal{O}(s^2h)\) compute to \(\mathcal{O}(s(2w+1))\) space and \(\mathcal{O}(s(2w+1)h)\) compute [2312.17649]. Longformer-style architectures use this local pattern plus a small number of global-attention tokens, and LED inherits this design for sequence-to-sequence summarization, where ordinary encoder tokens attend locally while global tokens create dense rows and columns in the sparse attention matrix [2410.08971].

A second major pattern is fixed structured sparsity. BlockBERT partitions the input into contiguous equal-length blocks and replaces dense token-token attention with a blockwise sparse mask. Its masked attention is
\[
\text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}, \mathbf{M}) = \text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d} \odot \mathbf{M}\right)\mathbf{V},
\]
where masked-out entries receive \(-\infty\) before softmax [1911.02972]. If the sequence is partitioned into \(n\) blocks and each query block attends to only one key block per head, the dense \(O(N^2)\) attention term is reduced to \(O(N^2/n)\) [1911.02972]. This is a blockwise rather than local-token formulation, and its principal design lesson is that hardware-friendly sparsity can matter as much as abstract sparsity rate.

A third pattern is hierarchical or sentence-structured sparsity. In summarization, one line of work argues that encoder-decoder attention is sparse at the sentence level: for a given decoding step, most useful source attention mass is concentrated on a small subset of source sentences [2109.03888]. The proposed mechanism first predicts sentence saliency,
\[
a_{m,i} \approx \tilde a_{m,i} = \text{softmax}\!\left(f_1(q_m)\cdot f_2(k_{i,1},\ldots,k_{i,J_i})\right),
\]
then restricts token-level cross-attention to words inside the top-\(r\) selected sentences [2109.03888]. In document-level NMT, a related but distinct hierarchical design performs sparse sentence-level selection with
\[
\alpha_s = sparsemax({Q_s{K_s}^T}/{\sqrt{d_k}),
\]
followed by word-level attention within each sentence and multiplicative reweighting
\[
\alpha_{hier}^j = \alpha_s(j)\alpha_{w}^j
\]
to produce a document context vector \(\alpha_{hier}V_w\) [1903.08788]. Here sparsity is semantic and differentiable, induced by sparsemax rather than by a hard positional mask.

A fourth pattern is alignment- or position-constrained attention. In long-context document MT, one paper replaces all self-attention and cross-attention modules with a window-attention mask centered on an alignment position \(b_i\), so a query vector \(q_i\) is allowed to attend only to key vectors \(k_j\) within a narrow window around \(b_i\) [2306.05183]. This reduces memory from \(O(L^2)\) to \(O(L \cdot w)\) and is explicitly motivated as both an efficiency mechanism and a way to prevent attention diffusion over long concatenated documents [2306.05183].

These patterns already indicate that “document-level sparse attention” is not a single architecture class. It includes local token sparsity, block sparsity, hierarchical sentence selection, and alignment-aware masking, each emphasizing a different approximation to document structure.

## 3. Query-conditioned sparsity in long-document retrieval

Long-document retrieval provides some of the clearest evidence that document-level attention can be aggressively structured around task asymmetry. QDS-Transformer defines a sparse attention graph for ranking as
\[
A_\text{QDS} = A_\text{local} \cup A_\text{sent} \cup A_\text{query} \cup A_\text{[CLS]}.
\]
The components encode local contextualization, sentence hierarchy through **[SOS]** sentence markers, query-global visibility, and global **[CLS]** aggregation [2010.12683]. The self-attention term is thereby reduced from
\[
\mathcal{O}(n^2 \text{dim})
\]
to
\[
\mathcal{O}(n \cdot \text{dim} \cdot (w + |q| + |s|)),
\]
where \(w\) is local window size, \(|q|\) query length, and \(|s|\) the number of sentences [2010.12683]. Empirically, QDS-Transformer achieves **NDCG@10 = 0.667**, **MAP = 0.278**, and **MRR@10 = 0.360** on TREC 2019 Deep Learning Track document ranking, outperforming several sparse baselines in that study [2010.12683]. The ablation is especially revealing: local + query-global attention is markedly stronger than local + sentence-global alone, and the full union of local, query, sentence, and **[CLS]** connectivity performs best [2010.12683].

A later cross-encoder study extends this logic by directly examining which cross-sequence interactions are actually necessary at document length. It introduces a sparse asymmetric cross-encoder in which the sequence is partitioned into **[\(\text{CLS}\)]**, query tokens, and document tokens, and attention is defined as
\[
O_c \;=\; \Attention_{(\infty, \infty, \infty)}(Q_c, (K_c, K_q, K_d), (V_c, V_q, V_d)),
\]
\[
O_q \;=\; \Attention_{(\infty)}(Q_q, (K_q), (V_q)),
\]
\[
O_d \;=\; \Attention_{(\infty, \infty, w)}(Q_d, (K_c, K_q, K_d), (V_c, V_q, V_d)).
\]
Under this pattern, query tokens attend only to query tokens; they do not attend to **[\(\text{CLS}\)]** or document tokens, while document tokens attend to **[\(\text{CLS}\)]**, all query tokens, and a local document window [2312.17649]. On document re-ranking, average nDCG@10 is **0.590** for sparse CE \(w=64\), **0.594** for \(w=16\), **0.577** for \(w=4\), **0.589** for \(w=1\), and **0.561** for \(w=0\), compared with **0.582** for the Longformer \(w=64\) document reference [2312.17649]. The paper concludes that “independent query contextualization only marginally affects re-ranking effectiveness,” and specifically that query tokens need not attend to document tokens for effective document re-ranking [2312.17649].

This suggests a directional interpretation of relevance modeling in cross-encoder rankers. Useful interactions arise primarily from document-side contextualization conditioned on the query, while the query itself can remain independently contextualized. The implication is architectural rather than merely empirical: relevance estimation in long-document ranking may be inherently asymmetric.

## 4. Global tokens, document structure, and content-aware routing

A separate line of work modifies sparse attention not by changing local windows, but by choosing global tokens more intelligently. In LED-style summarization, standard usage gives global attention only to the first token, described as a special token indicating the summarization task [2410.08971]. One proposed extension prefixes the input with additional keywords selected from the document and assigns those prefixed keyword tokens global attention. The procedure is concrete: run keyword detection, prefix the selected keywords, and mark them as global-attention tokens in the LED encoder [2410.08971]. For automated experiments, the method uses **TF-IDF** and drops unknown vocabulary from keyword selection [2410.08971].

The empirical picture is mixed. On the arXiv summarization dataset in the **10-example few-shot** regime, the paper’s LED baseline obtains ROUGE-1/2/L of **31.9 / 11.6 / 16.9**, while adding **10 keywords** gives **34.5 / 12.7 / 17.9** and **20 keywords** gives **33.9 / 12.9 / 17.9** [2410.08971]. On **AMI**, the method is more consistently positive, whereas on **ICSI** it mostly hurts, and the authors interpret ICSI as more **multi-topic**, so globally attended keywords may over-focus the model on a subset of themes [2410.08971]. A small ablation shows that content matters: on AMI, **10 TF-IDF keywords** outperform **10 random keywords sampled from the document** and **10 gibberish words**, indicating that gains do not come merely from adding more global slots [2410.08971]. This is evidence that which tokens receive global attention at document scale matters, and that content-aware global designation can serve as a lightweight communication scaffold.

StructFormer pushes the same idea earlier in the pipeline, into pre-training. It uses Longformer’s local + global sparse attention, but marks tokens belonging to titles and headings extracted from arXiv LaTeX markup as global tokens during MLM pre-training [2411.16618]. With local window size **256**, the sparse mask can be reconstructed as
\[
M_{ij}=0 \quad \text{iff} \quad \big(|i-j|\le w/2\big)\ \lor\ g_i=1\ \lor\ g_j=1,
\]
where \(g_j\) indicates whether token \(j\) is a structure token marked for global attention [2411.16618]. StructFormer is pre-trained on **100,000** filtered arXiv documents selected from **1,129,787 LaTeX documents** spanning 2000–2018 [2411.16618]. Structure-aware pre-training yields **2.2136 BPC** versus **2.3051** for default pre-training, and attention analysis shows an increase of more than **20%** in attention between keywords and header tokens relative to the vanilla model [2411.16618]. On SciREX, salient cluster F1 improves from **0.3182** for vanilla Longformer to **0.3419** for StructFormer, with smaller but consistently favorable improvements on binary and 4-ary relation extraction [2411.16618].

These papers collectively reposition global attention. Rather than treating global tokens as fixed architectural conveniences, they treat global-token designation as a document-level modeling decision: global tokens can be chosen by keyword salience, by explicit document hierarchy, or by both.

## 5. Adaptive and training-free sparsity for inference

Beyond fixed sparse masks, some recent work makes sparsity query-adaptive at inference time. Adamas is a training-free sparse attention mechanism for long-context autoregressive inference that retrieves a token-level top-\(k\) subset of keys for each query using compact Hadamard-transformed, bucketized, 2-bit-compressed representations [2510.18413]. Its retrieval proxy is
\[
\mathrm{sim}(\widehat{\mathbf{H}_Q}, \widehat{\mathbf{H}_K}) \approx - \lVert \widehat{\mathbf{H}_Q} - \widehat{\mathbf{H}_K} \rVert_1,
\]
and the actual sparse attention over the retained subset is then computed on the original, uncompressed keys and values [2510.18413]. The paper’s headline claim is that Adamas “matches the accuracy of full attention with only a 64-token budget” and is “near-lossless at 128,” while delivering up to **4.4x** self-attention and **1.5x** end-to-end speedups on 32K-length sequences [2510.18413]. On LongBench at budget 64, the reported performance gap over static windows and coarse page-based retrieval is especially large on long-document tasks such as GovReport and Qasper [2510.18413].

A different content-adaptive design appears in document-level NMT. Lasformer introduces a lightweight attention selector with reduced hidden dimension \(d_s\) that predicts which key tokens are worth attending to, then restricts the main full-dimensional attention to the top-\(k\) subset [2309.14174]. The selector is supervised to imitate original dense attention through
\[
L_s = \text{kl\_div}(A_s, A),
\]
and the total loss is
\[
Loss = L_{nmt} + \alpha * L_s
\]
with \(\alpha = 0.01\) [2309.14174]. Using \(d_s=64\), adaptive threshold \(t=0.95\), and layer sharing, the paper reports up to **95% sparsity**, **93% reduction in attention-module cost**, and **20% end-to-end inference speedup**, while keeping BLEU comparable to the Transformer baseline across TED, News, Europarl, and PDC [2309.14174]. The ablation is decisive: removing attention supervision collapses PDC BLEU from **28.04** to **12.94** [2309.14174].

Although the mechanisms differ substantially, both Adamas and Lasformer treat document-level sparsity as a retrieval problem: not every token is structurally irrelevant, but only a small subset is relevant to a given query. This is a different design philosophy from local-window models, which encode relevance mainly through positional proximity.

## 6. Task-specific document sparsity beyond retrieval

Summarization research has shown that sparse attention at document scale can occur not only in encoder self-attention but also in decoder cross-attention. One study demonstrates that, on **CNN/DailyMail**, restricting each decoding step to only **5 sentences** causes only a tiny drop in ROUGE if those are the right sentences: full vanilla attention gives **44.03 / 20.92 / 40.99** for R1/R2/RL, whereas ideal top-5 sentence restriction yields **43.94 / 20.82 / 40.81** [2109.03888]. A learned selector approaches full-attention performance, with the best sparse learned configuration reaching **43.72 / 20.40 / 40.70** [2109.03888]. The estimated \(r^\*\) where oracle sparse selection reaches the full-attention plateau is about **5** on CNNDM, **10** on XSum, and **30** on Podcast and arXiv [2109.03888]. This establishes sentence-structured sparsity as a property of summarization cross-attention, particularly under autoregressive decoding.

Document-level machine translation contributes a different perspective: sparsity can help quality by enforcing focus. In one long-context MT study, the percentage of cross-attention mass assigned to the current source sentence drops from **100.0%** in the sentence-level model to **76.0%** with one preceding sentence and **46.6%** with 1000-token context, while BLEU on NEWS falls from **32.8** to **33.1** to **29.5** [2306.05183]. Replacing dense attention with alignment-centered window-attention reduces memory from \(O(L^2)\) to \(O(Lw)\) and yields **33.1 BLEU / 48.1 TER** on NEWS compared with **29.5 / 53.7** for full concatenation at 1000 tokens [2306.05183]. The same study reports that longer context is especially helpful for style consistency, whereas pronoun disambiguation benefits mostly from local context [2306.05183]. This suggests that document-level sparse attention may serve not only as an approximation to dense context, but as a regularizer against indiscriminate long-range interactions.

Document-level relation extraction offers a contrasting case. GEGA is not computationally sparse in the efficiency sense, but it implements semantic selectivity through evidence-guided weighting and graph-derived attention concentration [2407.21384]. It uses dense softmax-derived adjacency matrices, sentence-level evidence scoring, and KL supervision toward evidence sentences, improving DocRED test **Evi-F1** from **55.43** in DREEAM to **55.89** in GEGA-single (student), and improving test F1 to **66.31** in GEGA-fusion (student) [2407.21384]. In this setting, “sparsity” is better understood as pair-specific relevance concentration than as a subquadratic attention operator.

These cases indicate that document-level sparse attention is task-specific in both mechanism and interpretation. For ranking, asymmetry and local windows dominate. For summarization, sentence-level cross-attention sparsity is central. For translation, alignment-local focus can improve both efficiency and discourse behavior. For relation extraction, evidence-guided attention may be selective without being computationally sparse.

## 7. Scope, trade-offs, and recurring design lessons

Several design lessons recur across the literature. First, local document context is often sufficient at surprising scales. In document re-ranking, Longformer variants with \(w=16\), \(w=4\), and \(w=1\) remain very competitive relative to a \(w=64\) reference, and even \(w=0\) is not catastrophic [2312.17649]. In the same study, for a query length of 10 and document length of 4086, latency and memory are **14 ms / 160 MB** for Longformer \(w=64\), **12 ms / 111 MB** for sparse CE \(w=64\), and **8 ms / 66 MB** for sparse CE \(w=4\), making sparse CE \(w=4\) **43% faster** and **59% lower** in memory than Longformer \(w=64\) [2312.17649]. BlockBERT likewise shows that mild structured sparsity can yield large practical savings: compared with RoBERTa-1seq at \(N=1024\), BlockBERT \(n=2\) reduces training time from **9.66** to **7.51** days and memory from **13.39 GB** to **9.73 GB**, while remaining competitive on long-paragraph QA [1911.02972].

Second, global access is valuable, but indiscriminate globality is not. Query tokens as global anchors are important in QDS-Transformer [2010.12683], content-bearing keywords can help as LED global tokens in some summarization regimes [2410.08971], and headings as global tokens during pre-training measurably alter downstream document understanding [2411.16618]. By contrast, adding sentence-level global tokens in QDS-Transformer does not by itself outperform query-global attention, and in some summarization settings too many global keyword tokens degrade performance [2010.12683; 2410.08971].

Third, directionality matters. The asymmetric cross-encoder result that document-to-query attention can remain while query-to-document attention is removed is one of the strongest mechanistic findings in the retrieval literature [2312.17649]. This challenges a common assumption that cross-sequence modeling must remain symmetric to preserve quality.

Fourth, semantic sparsity and computational sparsity are not identical. GEGA shows that a model can be selective and evidence-focused while still using dense softmax attention [2407.21384]. Conversely, block or local sparse masks may be efficient without explicitly modeling document semantics [1911.02972]. Many recent systems combine both aims only partially.

Finally, the scope of most claims is narrow. The asymmetric document re-ranking results are explicitly for **cross-encoder re-rankers** trained and evaluated on MS MARCO/TREC-style IR tasks, not for arbitrary long-document tasks [2312.17649]. Keyword-global LED results are dataset-dependent and sometimes negative [2410.08971]. StructFormer assumes access to reliable document structure such as arXiv LaTeX headings [2411.16618]. Adamas is evaluated up to 100K context for retrieval and 32K for most efficiency measurements, but not million-token scale [2510.18413]. Several diffusion-based long-document models report promising sparse-attention results, but some of their sparse-mask and absorbing-state details remain under-specified [2512.20724; 2512.20604].

Taken together, these works define document-level sparse attention as a research area centered on selective long-range interaction rather than a single architectural recipe. The field has moved from generic local-window approximations toward task-aware sparsity, document-structure-aware global routing, and query-adaptive retrieval. A plausible implication is that future progress will come less from universally denser long-context models than from better criteria for deciding which document interactions matter, for which task, and at which stage of training or inference.

Source: https://www.emergentmind.com/topics/document-level-sparse-attention