Document-Level Sparse Attention

• Document-Level Sparse Attention is a family of mechanisms that replace dense all-to-all interactions with selective connectivity to improve efficiency in processing long documents.
• It employs various designs such as local sliding windows, blockwise masks, hierarchical sentence selection, and global token routing to balance computational cost with semantic focus.
• These methods enable models to handle tasks like summarization, translation, and ranking by dynamically attending to the most relevant document content.

Searching arXiv for relevant papers on document-level sparse attention. Document-level sparse attention denotes a family of attention mechanisms for long-sequence transformers in which the full all-to-all interaction pattern is replaced, augmented, or guided by selective connectivity at document scale. The motivation is consistent across retrieval, summarization, translation, document understanding, information extraction, and long-context generation: standard dense self-attention scales quadratically with sequence length and becomes a practical bottleneck as inputs move from passage length to full documents. In the literature, document-level sparsity is realized through several distinct designs, including local sliding windows, asymmetric query-document coupling, blockwise masks, hierarchical sentence selection, structure-derived global tokens, query-adaptive token retrieval, and evidence-guided weighting (Schlatt et al., 2023). These mechanisms differ in whether they primarily target computational efficiency, inductive bias, or semantic selectivity, but they share the underlying premise that long documents rarely require every token to interact with every other token.

1. Quadratic attention and the document-length regime

A standard transformer computes attention from query, key, and value matrices through scaled dot-product attention. In the formulation used for cross-encoder re-ranking, this is written as

$O \;=\; \Attention(Q, K, V) \;=\; \softmax\left( \dfrac{QK^T}{\sqrt{h}} \right) V\,.$

With total sequence length $s$, dense self-attention computes an $s\times s$ attention matrix in every layer, so both memory and compute scale quadratically in sequence length (Schlatt et al., 2023). This scaling is manageable for passage-length inputs but substantially more problematic for full documents. In one document re-ranking setup, passages are truncated to 512 tokens, whereas documents are processed up to 4096 tokens, and the methodological consequence is explicit: at passage length, dense cross-encoders are expensive but still feasible; at document length, prior work resorted to cropping, splitting, MaxP, or sparse architectures (Schlatt et al., 2023).

The same bottleneck appears in sequence-to-sequence settings. For summarization, encoder-decoder attention has complexity

$O(MN)=O(MN_1N_2),$

with $M$ target length, $N$ source length, $N_1$ source sentences, and $N_2$ average words per sentence (Manakul et al., 2021). For document-level machine translation, standard attention is given as

$$\operatorname{attn}\left(q_t,\{k_i\},\{v_i\}\right)=\sum_{i=1}^{N} \frac{\exp\left(q_t \cdot k_i \right)} {\sum_{j=1}^{N}\exp\left(q_t \cdot k_{j} \right)}v_i^\top,$$

and its cost is quadratic in $N$ (Wu et al., 2022). A related document-MT study likewise states that, because the architecture uses self-attention and cross-attention, “the memory usage is $s$0 with $s$1 being the sequence length,” and ties long-context degradation partly to attention becoming less focused on the current sentence (Herold et al., 2023).

This body of work establishes document-level sparse attention as a response to a specific regime shift: once sequence lengths reach document scale, the question is no longer merely whether full attention is expressive, but whether the same token-interaction budget is necessary, affordable, or even desirable.

2. Canonical sparse patterns at document scale

The most common document-level sparse pattern is local sliding-window attention. In the cross-encoder formulation, if the window size is $s$2, each token attends to $s$3 positions: $s$4 tokens on the left, itself, and $s$5 tokens on the right. The paper formalizes this as

$s$6

with zero-padding when windows run outside sequence bounds. The theoretical complexity drops from $s$7 space and $s$8 compute to $s$9 space and $s\times s$0 compute (Schlatt et al., 2023). Longformer-style architectures use this local pattern plus a small number of global-attention tokens, and LED inherits this design for sequence-to-sequence summarization, where ordinary encoder tokens attend locally while global tokens create dense rows and columns in the sparse attention matrix (Lucas et al., 2024).

A second major pattern is fixed structured sparsity. BlockBERT partitions the input into contiguous equal-length blocks and replaces dense token-token attention with a blockwise sparse mask. Its masked attention is

$s\times s$1

where masked-out entries receive $s\times s$2 before softmax (Qiu et al., 2019). If the sequence is partitioned into $s\times s$3 blocks and each query block attends to only one key block per head, the dense $s\times s$4 attention term is reduced to $s\times s$5 (Qiu et al., 2019). This is a blockwise rather than local-token formulation, and its principal design lesson is that hardware-friendly sparsity can matter as much as abstract sparsity rate.

A third pattern is hierarchical or sentence-structured sparsity. In summarization, one line of work argues that encoder-decoder attention is sparse at the sentence level: for a given decoding step, most useful source attention mass is concentrated on a small subset of source sentences (Manakul et al., 2021). The proposed mechanism first predicts sentence saliency,

$s\times s$6

then restricts token-level cross-attention to words inside the top-$s\times s$7 selected sentences (Manakul et al., 2021). In document-level NMT, a related but distinct hierarchical design performs sparse sentence-level selection with

$s\times s$8

followed by word-level attention within each sentence and multiplicative reweighting

$s\times s$9

to produce a document context vector $O(MN)=O(MN_1N_2),$0 (Maruf et al., 2019). Here sparsity is semantic and differentiable, induced by sparsemax rather than by a hard positional mask.

A fourth pattern is alignment- or position-constrained attention. In long-context document MT, one paper replaces all self-attention and cross-attention modules with a window-attention mask centered on an alignment position $O(MN)=O(MN_1N_2),$1, so a query vector $O(MN)=O(MN_1N_2),$2 is allowed to attend only to key vectors $O(MN)=O(MN_1N_2),$3 within a narrow window around $O(MN)=O(MN_1N_2),$4 (Herold et al., 2023). This reduces memory from $O(MN)=O(MN_1N_2),$5 to $O(MN)=O(MN_1N_2),$6 and is explicitly motivated as both an efficiency mechanism and a way to prevent attention diffusion over long concatenated documents (Herold et al., 2023).

These patterns already indicate that “document-level sparse attention” is not a single architecture class. It includes local token sparsity, block sparsity, hierarchical sentence selection, and alignment-aware masking, each emphasizing a different approximation to document structure.

3. Query-conditioned sparsity in long-document retrieval

Long-document retrieval provides some of the clearest evidence that document-level attention can be aggressively structured around task asymmetry. QDS-Transformer defines a sparse attention graph for ranking as

$O(MN)=O(MN_1N_2),$7

The components encode local contextualization, sentence hierarchy through [SOS] sentence markers, query-global visibility, and global [CLS] aggregation (Jiang et al., 2020). The self-attention term is thereby reduced from

$O(MN)=O(MN_1N_2),$8

to

$O(MN)=O(MN_1N_2),$9

where $M$0 is local window size, $M$1 query length, and $M$2 the number of sentences (Jiang et al., 2020). Empirically, QDS-Transformer achieves NDCG@10 = 0.667, MAP = 0.278, and MRR@10 = 0.360 on TREC 2019 Deep Learning Track document ranking, outperforming several sparse baselines in that study (Jiang et al., 2020). The ablation is especially revealing: local + query-global attention is markedly stronger than local + sentence-global alone, and the full union of local, query, sentence, and [CLS] connectivity performs best (Jiang et al., 2020).

A later cross-encoder study extends this logic by directly examining which cross-sequence interactions are actually necessary at document length. It introduces a sparse asymmetric cross-encoder in which the sequence is partitioned into [$M$3], query tokens, and document tokens, and attention is defined as

$M$4

$M$5

$M$6

Under this pattern, query tokens attend only to query tokens; they do not attend to [$M$7] or document tokens, while document tokens attend to [$M$8], all query tokens, and a local document window (Schlatt et al., 2023). On document re-ranking, average nDCG@10 is 0.590 for sparse CE $M$9, 0.594 for $N$0, 0.577 for $N$1, 0.589 for $N$2, and 0.561 for $N$3, compared with 0.582 for the Longformer $N$4 document reference (Schlatt et al., 2023). The paper concludes that “independent query contextualization only marginally affects re-ranking effectiveness,” and specifically that query tokens need not attend to document tokens for effective document re-ranking (Schlatt et al., 2023).

This suggests a directional interpretation of relevance modeling in cross-encoder rankers. Useful interactions arise primarily from document-side contextualization conditioned on the query, while the query itself can remain independently contextualized. The implication is architectural rather than merely empirical: relevance estimation in long-document ranking may be inherently asymmetric.

4. Global tokens, document structure, and content-aware routing

A separate line of work modifies sparse attention not by changing local windows, but by choosing global tokens more intelligently. In LED-style summarization, standard usage gives global attention only to the first token, described as a special token indicating the summarization task (Lucas et al., 2024). One proposed extension prefixes the input with additional keywords selected from the document and assigns those prefixed keyword tokens global attention. The procedure is concrete: run keyword detection, prefix the selected keywords, and mark them as global-attention tokens in the LED encoder (Lucas et al., 2024). For automated experiments, the method uses TF-IDF and drops unknown vocabulary from keyword selection (Lucas et al., 2024).

The empirical picture is mixed. On the arXiv summarization dataset in the 10-example few-shot regime, the paper’s LED baseline obtains ROUGE-1/2/L of 31.9 / 11.6 / 16.9, while adding 10 keywords gives 34.5 / 12.7 / 17.9 and 20 keywords gives 33.9 / 12.9 / 17.9 (Lucas et al., 2024). On AMI, the method is more consistently positive, whereas on ICSI it mostly hurts, and the authors interpret ICSI as more multi-topic, so globally attended keywords may over-focus the model on a subset of themes (Lucas et al., 2024). A small ablation shows that content matters: on AMI, 10 TF-IDF keywords outperform 10 random keywords sampled from the document and 10 gibberish words, indicating that gains do not come merely from adding more global slots (Lucas et al., 2024). This is evidence that which tokens receive global attention at document scale matters, and that content-aware global designation can serve as a lightweight communication scaffold.

StructFormer pushes the same idea earlier in the pipeline, into pre-training. It uses Longformer’s local + global sparse attention, but marks tokens belonging to titles and headings extracted from arXiv LaTeX markup as global tokens during MLM pre-training (Ponkshe et al., 2024). With local window size 256, the sparse mask can be reconstructed as

$N$5

where $N$6 indicates whether token $N$7 is a structure token marked for global attention (Ponkshe et al., 2024). StructFormer is pre-trained on 100,000 filtered arXiv documents selected from 1,129,787 LaTeX documents spanning 2000–2018 (Ponkshe et al., 2024). Structure-aware pre-training yields 2.2136 BPC versus 2.3051 for default pre-training, and attention analysis shows an increase of more than 20% in attention between keywords and header tokens relative to the vanilla model (Ponkshe et al., 2024). On SciREX, salient cluster F1 improves from 0.3182 for vanilla Longformer to 0.3419 for StructFormer, with smaller but consistently favorable improvements on binary and 4-ary relation extraction (Ponkshe et al., 2024).

These papers collectively reposition global attention. Rather than treating global tokens as fixed architectural conveniences, they treat global-token designation as a document-level modeling decision: global tokens can be chosen by keyword salience, by explicit document hierarchy, or by both.

5. Adaptive and training-free sparsity for inference

Beyond fixed sparse masks, some recent work makes sparsity query-adaptive at inference time. Adamas is a training-free sparse attention mechanism for long-context autoregressive inference that retrieves a token-level top-$N$8 subset of keys for each query using compact Hadamard-transformed, bucketized, 2-bit-compressed representations (Yan et al., 21 Oct 2025). Its retrieval proxy is

$N$9

and the actual sparse attention over the retained subset is then computed on the original, uncompressed keys and values (Yan et al., 21 Oct 2025). The paper’s headline claim is that Adamas “matches the accuracy of full attention with only a 64-token budget” and is “near-lossless at 128,” while delivering up to 4.4x self-attention and 1.5x end-to-end speedups on 32K-length sequences (Yan et al., 21 Oct 2025). On LongBench at budget 64, the reported performance gap over static windows and coarse page-based retrieval is especially large on long-document tasks such as GovReport and Qasper (Yan et al., 21 Oct 2025).

A different content-adaptive design appears in document-level NMT. Lasformer introduces a lightweight attention selector with reduced hidden dimension $N_1$0 that predicts which key tokens are worth attending to, then restricts the main full-dimensional attention to the top-$N_1$1 subset (Liu et al., 2023). The selector is supervised to imitate original dense attention through

$N_1$2

and the total loss is

$N_1$3

with $N_1$4 (Liu et al., 2023). Using $N_1$5, adaptive threshold $N_1$6, and layer sharing, the paper reports up to 95% sparsity, 93% reduction in attention-module cost, and 20% end-to-end inference speedup, while keeping BLEU comparable to the Transformer baseline across TED, News, Europarl, and PDC (Liu et al., 2023). The ablation is decisive: removing attention supervision collapses PDC BLEU from 28.04 to 12.94 (Liu et al., 2023).

Although the mechanisms differ substantially, both Adamas and Lasformer treat document-level sparsity as a retrieval problem: not every token is structurally irrelevant, but only a small subset is relevant to a given query. This is a different design philosophy from local-window models, which encode relevance mainly through positional proximity.

6. Task-specific document sparsity beyond retrieval

Summarization research has shown that sparse attention at document scale can occur not only in encoder self-attention but also in decoder cross-attention. One study demonstrates that, on CNN/DailyMail, restricting each decoding step to only 5 sentences causes only a tiny drop in ROUGE if those are the right sentences: full vanilla attention gives 44.03 / 20.92 / 40.99 for R1/R2/RL, whereas ideal top-5 sentence restriction yields 43.94 / 20.82 / 40.81 (Manakul et al., 2021). A learned selector approaches full-attention performance, with the best sparse learned configuration reaching 43.72 / 20.40 / 40.70 (Manakul et al., 2021). The estimated $N_1$7 where oracle sparse selection reaches the full-attention plateau is about 5 on CNNDM, 10 on XSum, and 30 on Podcast and arXiv (Manakul et al., 2021). This establishes sentence-structured sparsity as a property of summarization cross-attention, particularly under autoregressive decoding.

Document-level machine translation contributes a different perspective: sparsity can help quality by enforcing focus. In one long-context MT study, the percentage of cross-attention mass assigned to the current source sentence drops from 100.0% in the sentence-level model to 76.0% with one preceding sentence and 46.6% with 1000-token context, while BLEU on NEWS falls from 32.8 to 33.1 to 29.5 (Herold et al., 2023). Replacing dense attention with alignment-centered window-attention reduces memory from $N_1$8 to $N_1$9 and yields 33.1 BLEU / 48.1 TER on NEWS compared with 29.5 / 53.7 for full concatenation at 1000 tokens (Herold et al., 2023). The same study reports that longer context is especially helpful for style consistency, whereas pronoun disambiguation benefits mostly from local context (Herold et al., 2023). This suggests that document-level sparse attention may serve not only as an approximation to dense context, but as a regularizer against indiscriminate long-range interactions.

Document-level relation extraction offers a contrasting case. GEGA is not computationally sparse in the efficiency sense, but it implements semantic selectivity through evidence-guided weighting and graph-derived attention concentration (Mao et al., 2024). It uses dense softmax-derived adjacency matrices, sentence-level evidence scoring, and KL supervision toward evidence sentences, improving DocRED test Evi-F1 from 55.43 in DREEAM to 55.89 in GEGA-single (student), and improving test F1 to 66.31 in GEGA-fusion (student) (Mao et al., 2024). In this setting, “sparsity” is better understood as pair-specific relevance concentration than as a subquadratic attention operator.

These cases indicate that document-level sparse attention is task-specific in both mechanism and interpretation. For ranking, asymmetry and local windows dominate. For summarization, sentence-level cross-attention sparsity is central. For translation, alignment-local focus can improve both efficiency and discourse behavior. For relation extraction, evidence-guided attention may be selective without being computationally sparse.

7. Scope, trade-offs, and recurring design lessons

Several design lessons recur across the literature. First, local document context is often sufficient at surprising scales. In document re-ranking, Longformer variants with $N_2$0, $N_2$1, and $N_2$2 remain very competitive relative to a $N_2$3 reference, and even $N_2$4 is not catastrophic (Schlatt et al., 2023). In the same study, for a query length of 10 and document length of 4086, latency and memory are 14 ms / 160 MB for Longformer $N_2$5, 12 ms / 111 MB for sparse CE $N_2$6, and 8 ms / 66 MB for sparse CE $N_2$7, making sparse CE $N_2$8 43% faster and 59% lower in memory than Longformer $N_2$9 (Schlatt et al., 2023). BlockBERT likewise shows that mild structured sparsity can yield large practical savings: compared with RoBERTa-1seq at $$\operatorname{attn}\left(q_t,\{k_i\},\{v_i\}\right)=\sum_{i=1}^{N} \frac{\exp\left(q_t \cdot k_i \right)} {\sum_{j=1}^{N}\exp\left(q_t \cdot k_{j} \right)}v_i^\top,$$0, BlockBERT $$\operatorname{attn}\left(q_t,\{k_i\},\{v_i\}\right)=\sum_{i=1}^{N} \frac{\exp\left(q_t \cdot k_i \right)} {\sum_{j=1}^{N}\exp\left(q_t \cdot k_{j} \right)}v_i^\top,$$1 reduces training time from 9.66 to 7.51 days and memory from 13.39 GB to 9.73 GB, while remaining competitive on long-paragraph QA (Qiu et al., 2019).

Second, global access is valuable, but indiscriminate globality is not. Query tokens as global anchors are important in QDS-Transformer (Jiang et al., 2020), content-bearing keywords can help as LED global tokens in some summarization regimes (Lucas et al., 2024), and headings as global tokens during pre-training measurably alter downstream document understanding (Ponkshe et al., 2024). By contrast, adding sentence-level global tokens in QDS-Transformer does not by itself outperform query-global attention, and in some summarization settings too many global keyword tokens degrade performance (Jiang et al., 2020, Lucas et al., 2024).

Third, directionality matters. The asymmetric cross-encoder result that document-to-query attention can remain while query-to-document attention is removed is one of the strongest mechanistic findings in the retrieval literature (Schlatt et al., 2023). This challenges a common assumption that cross-sequence modeling must remain symmetric to preserve quality.

Fourth, semantic sparsity and computational sparsity are not identical. GEGA shows that a model can be selective and evidence-focused while still using dense softmax attention (Mao et al., 2024). Conversely, block or local sparse masks may be efficient without explicitly modeling document semantics (Qiu et al., 2019). Many recent systems combine both aims only partially.

Finally, the scope of most claims is narrow. The asymmetric document re-ranking results are explicitly for cross-encoder re-rankers trained and evaluated on MS MARCO/TREC-style IR tasks, not for arbitrary long-document tasks (Schlatt et al., 2023). Keyword-global LED results are dataset-dependent and sometimes negative (Lucas et al., 2024). StructFormer assumes access to reliable document structure such as arXiv LaTeX headings (Ponkshe et al., 2024). Adamas is evaluated up to 100K context for retrieval and 32K for most efficiency measurements, but not million-token scale (Yan et al., 21 Oct 2025). Several diffusion-based long-document models report promising sparse-attention results, but some of their sparse-mask and absorbing-state details remain under-specified (Christoforos et al., 23 Dec 2025, Christoforos et al., 23 Dec 2025).

Taken together, these works define document-level sparse attention as a research area centered on selective long-range interaction rather than a single architectural recipe. The field has moved from generic local-window approximations toward task-aware sparsity, document-structure-aware global routing, and query-adaptive retrieval. A plausible implication is that future progress will come less from universally denser long-context models than from better criteria for deciding which document interactions matter, for which task, and at which stage of training or inference.