FS Blocks in Block-Sparse FlashAttention

• FS Blocks are selected key/value blocks computed after exact query-key score evaluation in BSFA to enable efficient long-context inference.
• They retain the top‑k most relevant blocks by applying a calibrated threshold, thus skipping nearly 50% of value computations.
• This strategy balances computational savings and accuracy, achieving moderate speedups while preserving over 99% of baseline performance.

FS Blocks are the key/value blocks retained for attention computation in Block-Sparse FlashAttention (BSFA), a training-free, drop-in replacement for FlashAttention that targets long-context inference. In the terminology used for BSFA, an FS Block refers to a key/value block $j$ that is selected for attention computation with a given query block $i$ on the basis of exact query-key similarity scores rather than a precomputed importance predictor. The mechanism is designed to reduce the quadratic bottleneck of dense attention by computing exact query-key scores, retaining the top-$k$ most important value blocks for each query, and skipping approximately $50\%$ of the computation and memory transfers for pruned blocks while maintaining above $99\%$ baseline accuracy in reported settings (Ohayon et al., 7 Dec 2025).

1. Definition within block-sparse attention

FS Blocks arise from the blocked execution model used by BSFA. The method preserves the tiled structure of FlashAttention, but changes the treatment of key/value blocks after score computation. Instead of processing every block densely, BSFA computes exact query-key similarities and then decides, block by block, whether the corresponding value block should participate in the attention update. In this setting, an FS Block is not an independent architectural component; it is a selected key/value block within the ordinary blocked attention layout (Ohayon et al., 7 Dec 2025).

A central point is that BSFA does not predict importance before computing scores. The method explicitly differs from approaches that infer sparsity from compressed surrogates or fixed patterns. Its pruning decision is made only after the score submatrix for a block has been computed exactly. This makes FS Blocks a form of late, score-driven sparsification rather than approximate preselection (Ohayon et al., 7 Dec 2025).

The immediate significance of this definition is operational. FS Blocks are the units through which BSFA turns exact score evaluation into selective value processing. A plausible implication is that the method tries to preserve rare but important dependencies that could be missed by earlier, approximate pruning schemes.

2. Block partitioning and the score criterion

BSFA partitions the sequence into query blocks of size $B_M$ and key/value blocks of size $B_N$. For block indices $i$ and $j$,

• $\mathbf{Q}_i \in \mathbb{R}^{B_M \times d}$,
• $i$0,
• $i$1.

For each tile $i$2, the score submatrix is

$i$3

The block-importance statistic is the maximum entry in that submatrix,

$i$4

This single value serves as the selection proxy for whether the key/value block $i$5 should become an FS Block for query block $i$6 (Ohayon et al., 7 Dec 2025).

Object	Definition	Role
Query block	$i$7	Current block of queries
Key/value block	$i$8	Candidate source block
Score block	$i$9	Exact similarity matrix
Block score	$k$0	Importance proxy
Threshold	$k$1	Retention criterion

The use of a block maximum rather than a finer-grained statistic is technically important. It reduces the decision to one scalar per block while preserving exact score computation within the block. This suggests a compromise between hardware-efficient tiling and content-adaptive sparsity.

3. Retention rule and threshold calibration

For each non-diagonal block in causal attention, with $k$2, BSFA compares the block score $k$3 against a calibrated threshold $k$4, where $k$5 is the layer, $k$6 is the head, $k$7 is the query-block index, and $k$8 is the desired number of off-diagonal blocks retained per query block. The gating rule is:

$k$9

The diagonal block $50\%$0 is always retained for causal consistency and local attention (Ohayon et al., 7 Dec 2025).

Thresholds are obtained by a one-time calibration on a small dataset. In the reported experiments, $50\%$1 samples suffice. For each layer, head, and query-block position, the method collects the distribution of block max-scores across off-diagonal blocks, orders them, selects the $50\%$2-th largest score as the threshold, and then aggregates these thresholds across calibration samples. The result is a stored tensor

$50\%$3

where $50\%$4 is the number of supported sparsity levels, $50\%$5 the number of layers, $50\%$6 the number of heads, and $50\%$7 the maximum sequence length. At inference time, BSFA uses the slice corresponding to the chosen $50\%$8 (Ohayon et al., 7 Dec 2025).

The calibration procedure is presented as task- and data-agnostic, and thresholds calibrated on RULER were reported to transfer to LongBench, including language-shifted settings. This suggests that the score distributions driving FS Block selection are relatively stable properties of the model rather than narrowly dataset-specific artifacts (Ohayon et al., 7 Dec 2025).

4. Kernel integration and skipped work

FS Block selection is integrated inline into the FlashAttention kernel. For a given query block, the algorithm loads $50\%$9, computes $99\%$0 exactly, extracts $99\%$1, and applies the threshold test. If the block is retained, $99\%$2 is loaded and the block is processed as in standard FlashAttention with stable softmax updates. If it is not retained, the kernel does not load $99\%$3, does not compute the corresponding probability-value multiplication, and does not update running accumulators or normalizers for that block (Ohayon et al., 7 Dec 2025).

The computational consequence is specific. BSFA always pays for the query-key pass, since the scores must be computed exactly, but it can skip the value-side work for pruned blocks. The summary reports that the query-key and probability-value computations are roughly equal in FLOPs per block, namely $99\%$4 each. Accordingly, pruning around half the blocks can eliminate up to $99\%$5 of attention computation and HBM traffic for those blocks, although measured end-to-end speedups are smaller because the score pass and other overheads remain (Ohayon et al., 7 Dec 2025).

The top-$99\%$6 structure also produces a regular workload. Because each query block retains a fixed number of off-diagonal blocks, threads process comparable amounts of work, apart from boundary effects. This contrasts with variable-sparsity schemes that can produce stragglers when some threads retain many more blocks than others.

5. Empirical behavior and performance regime

On Llama-3.1-8B, BSFA is reported to achieve up to $99\%$7 speedup on real-world reasoning benchmarks and up to $99\%$8 on needle-in-a-haystack retrieval tasks while maintaining above $99\%$9 baseline accuracy. The abstract further notes that some configurations improve accuracy by focusing attention on the most relevant content (Ohayon et al., 7 Dec 2025).

The detailed results expose the sparsity-accuracy-speed trade-off more explicitly. For a $B_M$0K input length, the summary gives the following examples: BSFA with $B_M$1 has sparsity $B_M$2, accuracy $B_M$3, and speedup $B_M$4; with $B_M$5, sparsity $B_M$6, accuracy $B_M$7, and speedup $B_M$8; with $B_M$9, sparsity $B_N$0, accuracy $B_N$1, and speedup $B_N$2. In the same table, FlashAttention-2 is listed at $B_N$3 accuracy and $B_N$4 speedup (Ohayon et al., 7 Dec 2025).

Task dependence is pronounced. Needle-in-a-haystack retrieval tolerates aggressive pruning, with the summary stating that extreme sparsity such as $B_N$5 can suffice because only a few blocks contain the needed information. Reasoning and aggregation tasks require more retained blocks. Measured densities also reportedly track the densities predicted by calibration closely, which supports the practical reliability of the threshold mechanism (Ohayon et al., 7 Dec 2025).

A plausible implication is that FS Blocks encode a task-sensitive notion of informational concentration: retrieval workloads often concentrate useful evidence into a small set of blocks, whereas multi-step reasoning distributes dependencies more broadly across context.

6. Relation to dense and approximate sparse attention

Relative to dense FlashAttention, FS Blocks are the unit of sparsity. Standard FlashAttention computes both score and value updates for all blocks, optimizing memory access and tiling but not reducing the set of processed blocks. BSFA retains the exact score computation but sparsifies the value path through FS Block selection (Ohayon et al., 7 Dec 2025).

Relative to approximate sparse-attention methods, BSFA is characterized by late pruning on real scores. The summary contrasts it with methods that predict block importance before seeing exact attention scores and with SpargeAttention, which uses approximate masks and variable thread workloads. BSFA, by comparison, uses actual attention scores, introduces only a conditional check after each query-key block, requires no extra projections or preprocessing, and remains training-free (Ohayon et al., 7 Dec 2025).

This distinction addresses a common misconception. FS Blocks are not heuristic candidates produced by a separate importance model; they are blocks retained after exact score computation under a calibrated top-$B_N$6 policy. Their role is therefore closer to selective execution within an exact tiled attention pipeline than to approximate structural sparsity.

7. Interpretive significance

Within BSFA, FS Blocks provide the mechanism by which long-context attention becomes selectively executable without abandoning exact query-key evaluation. They define the retained support of the value-side computation, layer by layer and head by head, under thresholds learned from calibration data. In that sense, they are the practical interface between block tiling, content-adaptive sparsity, and GPU-efficient inference (Ohayon et al., 7 Dec 2025).

The broader importance of FS Blocks lies in the combination of three properties that are usually in tension: exact score computation, training-free deployment, and hardware-friendly blockwise sparsity. The reported results suggest that this combination is sufficient to obtain moderate but consistent inference acceleration while preserving quality on long-context tasks. This suggests that FS Blocks may be understood not as a new attention formalism, but as a precise execution policy for deciding which blocked interactions merit full value-side processing.