---
title: Bidirectional Sparse Attention (BSA) Overview
url: https://www.emergentmind.com/topics/bidirectional-sparse-attention-bsa
type: topic
---

# Bidirectional Sparse Attention (BSA) Overview

Bidirectional Sparse Attention (BSA) denotes a family of attention-reduction strategies that preserve broad contextual modeling while avoiding the quadratic cost of dense attention. In the recent literature, the term does not have a single canonical meaning. In video diffusion Transformers, BSA can denote the simultaneous dynamic sparsification of Queries and Key–Value pairs within 3D full attention [2509.01085]. In training-free video generation, it can denote online bidirectional co-clustering that jointly partitions queries and keys before block selection [2603.18636]. In long-context language modeling, closely related formulations use bidirectional alignment between sparse and full attention streams, even when the paper does not explicitly adopt the name BSA [2511.20102]. A common thread across these works is the attempt to reduce the effective attention domain without discarding the content-adaptive structure that dense attention would otherwise model.

## 1. Terminology and conceptual scope

A useful way to read the BSA literature is to separate the **object being made bidirectional** from the **mechanism used to induce sparsity**. The term is therefore best understood as polysemous rather than fully standardized.

| Paper | What “bidirectional” denotes | Domain |
|---|---|---|
| [2509.01085] | Joint reduction of active queries and retained Key–Value blocks/tokens | Video diffusion training |
| [2603.18636] | Joint query–key partitioning via bidirectional co-clustering | Training-free video generation |
| [2511.20102] | Symmetric alignment between sparse-attention and full-attention outputs | Long-context language modeling |

This terminological variation matters technically. In "Bidirectional Sparse Attention for Faster Video Diffusion Training" [2509.01085], bidirectionality is not about left-to-right versus right-to-left context; it refers to **Q-side dynamic sparsification** and **K/V-side dynamic sparsification** performed together within 3D full attention. In SVOO, the BSA core is the coupling of query and key block assignments, so that block partitioning is not performed independently on the two sides [2603.18636]. In SSA, the authors explicitly use the phrase **bidirectional alignment** rather than BSA, and the provided terminology mapping presents SSA as a close analogue rather than a paper that formally names the method BSA [2511.20102].

Related work broadens the historical context. "Combiner: Full Attention Transformer with Sparse Computation Cost" treats self-attention as a conditional expectation and preserves full attention capability with sub-quadratic cost, including in bidirectional MLM settings where $\Omega_i = [L]$ [2107.05768]. "Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention" positions block-wise, content-adaptive sparse attention as especially relevant for bidirectional diffusion language models, although the accompanying description frames its operator as a design blueprint grounded in a normalization perturbation lemma rather than a fully specified empirical account [2605.19726].

## 2. Computational motivation

The motivating bottleneck is dense attention’s quadratic scaling. For a single head with sequence length $L$ and head dimension $d$, the standard formulation is
$$
S = QK^\top / \sqrt{d}, \qquad A = \text{softmax}(S), \qquad O = AV.
$$
In the video DiT setting, latent tensors of shape $(T, H, W)$ are flattened to a 1D sequence of length $L = T \times H \times W$, and the dominant per-head FLOP cost is approximately $4L^2 d$; with $h$ heads, it is approximately $4hL^2 d$ [2509.01085]. Memory is similarly dominated by the $O(L^2)$ score or attention matrix. The same basic bottleneck appears in training-free video generation, where per-head attention is described as $O(N^2 d)$ in 3D token spaces, with $N = T \cdot H \cdot W$ [2603.18636].

In video diffusion Transformers, this scaling is especially severe because both resolution and duration enlarge the token lattice. The BSA paper states that, in DiTs, attention often dominates more than $90\%$ of training cost, making dense 3D attention the principal bottleneck [2509.01085]. SVOO makes the same structural diagnosis on the inference side: attention is repeatedly applied across diffusion steps and layers, so even modest per-layer savings compound across generation [2603.18636].

This suggests that the central design problem is not merely to reduce the number of computed interactions, but to do so **adaptively**. Fixed sparse patterns such as local windows, striding, or static top-$k$ rules are described as suboptimal because attention distributions vary across time, space, heads, layers, samples, and training steps [2509.01085]. The modern BSA formulations therefore emphasize content-aware selection rather than static geometry.

## 3. Dynamic bidirectional sparsification in video diffusion Transformers

The formulation in "Bidirectional Sparse Attention for Faster Video Diffusion Training" is a trainable, hardware-aligned sparse attention mechanism for Video Diffusion Transformers that simultaneously sparsifies the Query side and the Key–Value side in 3D full attention [2509.01085]. Its two components are complementary.

On the query side, the video latent is partitioned into 3D blocks of size $(C_t, C_h, C_w)$, with block size $B = C_t \cdot C_h \cdot C_w$. Within each block, a representative center query is chosen, and token selection is driven by semantic similarity to that center. The similarity metric instantiated in the paper is cosine similarity,
$$
\mathrm{sim}(q_c^{(b)}, q_i) = \frac{q_c^{(b)} \cdot q_i}{\|q_c^{(b)}\| \cdot \|q_i\|},
$$
and the retained sparse query set is written as
$$
Q^{s} = \bigcup_{b=1}^{N} \left\{ q_i \in Q_c^{(b)} \mid \mathrm{rank}_b\!\left(1 - \cos(q_c^{(b)}, q_i)\right) \le \left\lceil r \cdot |Q_c^{(b)}| \right\rceil \right\}.
$$
The paper also introduces a window-based refinement in which each block is subdivided into windows of size $(w_t, w_h, w_w)$ and local centers are used instead of a single block center; this is reported to preserve fine-grained semantics better at the same sparsity [2509.01085].

On the Key–Value side, tensors are grouped into aligned spatiotemporal blocks, and inter-block saliency scores $S_b = \{s_1,\dots,s_n\}$ are computed before full attention materialization. The dynamic threshold is
$$
p = \mathrm{mean}(S_b) + \mathrm{std}(S_b) \cdot U(1-k/n),
$$
where $U(\cdot)$ is the quantile function and $k$ is derived from the sparsity schedule. For a query block $i$, the retained KV set is the minimal set satisfying a cumulative probability target,
$$
S_i = \arg\min_{S} \left\{ |S| : \sum_{j \in S} \pi_{i,j} \ge p \right\}, \qquad
\pi_{i,j} = \frac{\exp(Q_i K_j^\top)}{\sum_{j'} \exp(Q_i K_{j'}^\top)}.
$$
The retained tokens are then used in sparse attention
$$
S^s = Q^s K_S^\top / \sqrt{d}, \qquad A^s = \text{softmax}(S^s), \qquad O^s = A^s V_S.
$$

The resulting complexity depends on the retained query fraction $r_Q = N_Q/L$ and KV retention fraction $r_K = N_{KV}/L$. The paper states that FLOPs scale as $r_Q r_K$ relative to full attention, so the speedup is approximately $1/(r_Q r_K)$. The empirical rule of thumb given is $r_Q \approx 0.5$ and $r_K \approx 0.1$, yielding $r_Q r_K \approx 0.05$, or roughly $20\times$ FLOP reduction [2509.01085].

Several implementation choices are part of the method rather than incidental engineering. Hard binary masks are used for queries, gradients flow only through retained queries, outputs are scattered back to the original token layout, and mask computation overhead is measured as less than $0.1\%$ FLOPs. The sparsity schedule is annealed: training begins with full attention, then every 30 steps sparsity increases by 0.03 until approximately 0.9. Triton custom kernels and block-partitioned masks are used so that GPU SM tiles process or skip whole blocks, and the design is explicitly aligned with FlashAttention-style IO-aware tiling [2509.01085].

## 4. Training-free BSA via offline profiling and online bidirectional co-clustering

SVOO realizes a different BSA paradigm for inference-time acceleration in video generation [2603.18636]. The method is explicitly training-free and organized into two stages: **offline layer-wise sparsity profiling** and **online bidirectional co-clustering**.

The offline stage estimates intrinsic sparsity for each layer and head. For calibration input $x^{(k)}$, layer $\ell$, and head $h$, the post-softmax attention matrix is $A_{\ell,h}^{(k)} \in \mathbb{R}^{n \times n}$. For each query row, the smallest index set covering a recall threshold $\tau = 0.95$ is found, and the attention density is defined as
$$
d_{\ell,h}^{(k)} = \frac{1}{n} \sum_{i=1}^{n} \frac{|S_{\ell,h}^{(k)}(i)|}{n}.
$$
These densities are modeled as Gaussian across calibration samples, $d_{\ell,h}^{(k)} \sim \mathcal{N}(\mu_{\ell,h}, \sigma^2_{\ell,h})$, with conservative estimate
$$
\hat d_{\ell,h} = \mu_{\ell,h} + z_\alpha \sigma_{\ell,h}, \qquad \alpha = 0.95,
$$
and sparsity schedule
$$
s_{\ell,h} = 1 - \hat d_{\ell,h}.
$$
The paper argues that this generalizes across inputs because attention sparsity is an intrinsic property of each layer, with minor effects across different inputs, and it supports this claim through a stability bound on a pre-softmax logit variance proxy $V(X)$ [2603.18636].

The online stage performs **affinity-driven, alternating bidirectional co-clustering**. Query and key tokens are partitioned into $K_q$ and $K_k$ blocks, with the default experimental choice $K_q = 256$ and $K_k = 1024$. Current query anchors induce key-side affinity patterns,
$$
P_k = K (C_q^{(i-1)})^\top,
$$
which are normalized and used to assign each key to the nearest key cluster. Updated key anchors then induce query-side affinity patterns,
$$
P_q = Q (C_k^{(i)})^\top,
$$
which are used to assign queries to clusters. The paper uses $I_{\max} = 2$ co-clustering iterations per recompute. Block-pair saliency is then approximated by centroid dot products
$$
\bar A = C_q C_k^\top,
$$
and the fraction of active block pairs is chosen by a rule that balances intrinsic sparsity schedule and recall target:
$$
\rho_{\ell,h} =
\begin{cases}
\min(\mathrm{Recall}(\bar A, \tau), s_{\ell,h}) & \text{if } s_{\ell,h} > \theta,\\
\max(\mathrm{Recall}(\bar A, \tau), s_{\ell,h}) & \text{otherwise},
\end{cases}
$$
with $\theta = 0.1$.

The resulting attention complexity is written as $O(\alpha N^2 d)$ with $\alpha = R/(K_q K_k) < 1$, where $R$ is the number of active block pairs [2603.18636]. Integration is again hardware-conscious: Triton kernels are used for co-clustering, dynamic block-size FlashInfer kernels are used for block-sparse attention, and clustering assignments are reused every $N = 20$ diffusion steps because partitions are empirically stable across steps. First-layer dense attention and warm-up dense diffusion steps are retained: $20\%$ for Wan-series and $10\%$ for HunyuanVideo-series [2603.18636].

## 5. Bidirectional alignment and other generalizations

A broader strand of work uses **bidirectionality in supervision rather than in token selection**. "SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space" does not use the term BSA, but the provided terminology mapping presents it as a close functional instance of the concept [2511.20102]. SSA maintains two attention paths per layer: a native sparse-attention path and a native full-attention path. The main path is sampled per iteration with probability $0.5$, and the auxiliary path is the opposite mode. The layerwise alignment losses are
$$
L_{\text{sparsity}} = \|a_{\text{full}} - \mathrm{sg}[a_{\text{sparse}}]\|, \qquad
L_{\text{commitment}} = \|a_{\text{sparse}} - \mathrm{sg}[a_{\text{full}}]\|,
$$
with
$$
L_{\text{alignment}} = L_{\text{sparsity}} + L_{\text{commitment}},
$$
and total objective
$$
L = \mathbb{E}_{\text{mode} \sim \{\text{full}, \text{sparse}\}}[L_{\text{mode}}] + \alpha L_{\text{alignment}}.
$$
The paper’s central diagnosis is **gradient update deficiency**: low-ranked key–value pairs excluded during sparse training receive neither forward contribution nor backward gradients, so they never learn proper suppression. Alternating full and sparse streams, while aligning their outputs symmetrically, is the proposed remedy [2511.20102].

Other related formulations situate BSA-like ideas in broader bidirectional modeling. BA-Att is presented for diffusion language models, which require globally coherent, bidirectional, and controllable text generation. Its abstract states that the method identifies informative regions in a compact downsampled space, avoids fixed positional priors, achieves up to $6.95\times$ acceleration over FlashAttention in attention computation, and maintains near full-attention performance at $50\%$ sparsity across language models, multimodal language models, and video generation models [2605.19726]. The accompanying description, however, presents the operator as a theoretically motivated blueprint rather than a fully grounded empirical account, with the normalization perturbation lemma serving as the main formal anchor.

An earlier precursor is Combiner, which is not a sparse masking method in the usual sense but is directly relevant to bidirectional sparse-attention discussions because it preserves full attention capability with sub-quadratic cost [2107.05768]. Combiner factors the conditional attention distribution through region abstractions,
$$
p(j|i) = p(j \mid \Omega_i^{r_j})\, p(\Omega_i^{r_j} \mid i),
$$
and thereby retains full support in bidirectional MLM settings while achieving $O(L\log L)$ or $O(L\sqrt{L})$ complexity depending on the partition scheme. This suggests that some lines of research adjacent to BSA aim not to sparsify the support irreversibly, but to reparameterize or approximate full attention through structured intermediates [2107.05768].

## 6. Empirical behavior, trade-offs, and limitations

The empirical profile of BSA depends strongly on the formulation.

For trainable video diffusion BSA, experiments are reported with a Wan2.1-1.3B backbone, 300k videos from Vchitect T2V DataVerse, preprocessing including shot segmentation, 5-second truncation, and captions from Tarsier2, over 30,000 training steps on NVIDIA H100 GPUs [2509.01085]. At approximately 23K tokens, full attention yields Text Consistency $32.71\%$, BG Consistency $95.12\%$, Image Quality $64.33\%$, Subject Consistency $92.34\%$, and FLOPs approximately $1.51 \times 10^{12}$; BSA yields Text Consistency $32.79\%$, BG Consistency $95.22\%$, Image Quality $64.29\%$, Subject Consistency $92.39\%$, and FLOPs approximately $1.05 \times 10^{11}$, corresponding to $12.85\times$ speedup. At approximately 153K tokens, full attention yields Text $34.76\%$, BG $93.26\%$, Image $65.91\%$, Subject $93.79\%$, and FLOPs approximately $6.99 \times 10^{13}$; BSA yields Text $34.93\%$, BG $93.41\%$, Image $66.03\%$, Subject $94.13\%$, and FLOPs approximately $3.49 \times 10^{12}$, corresponding to $17.79\times$ speedup [2509.01085]. Inference latency on H100 is reduced from 31s to 5s, approximately $6.2\times$, with no perceptible quality degradation. The ablations are structurally informative: query-sparse with $r=0.5$ gives approximately $1.96\times$ speedup; KV-sparse with fixed threshold gives approximately $6.05\times$; KV-sparse with statistical dynamic threshold gives approximately $6.12\times$; combined query+KV sparsity gives approximately $12.85\times$ at sparsity approximately $0.93$ [2509.01085].

For SVOO, the reported quality–speed trade-off is more modest in raw speedup but broad across seven video generation models [2603.18636]. On Wan2.1-T2V-1.3B at 720p and 81 frames, SVOO reports PSNR $29.986$ dB, SSIM $0.898$, LPIPS $0.125$, ImageQual $66.57\%$, AesQual $64.45\%$, SubjectConsistency $96.62\%$, BackgroundConsistency $97.19\%$, latency 216 s, and speedup $1.93\times$ versus dense attention. HunyuanVideo-T2V reports latency 821 s and speedup $2.17\times$, which is the best speedup across the T2V experiments. Ablations show that removing offline profiling reduces efficiency for similar quality, while removing bidirectional co-clustering degrades PSNR and SSIM and raises LPIPS for similar speed [2603.18636].

For bidirectional alignment in SSA, the strongest evidence concerns sparsity fidelity rather than video generation throughput [2511.20102]. SSA reports the smallest KL divergence between sparse and full modes, at $0.0656$, and the highest attention sparsity, with AttnSparsity $0.658$ in sparse mode and $0.711$ in full mode. Under full-attention inference, SSA attains commonsense average $60.22$ and WikiText perplexity $15.19$; under sparse attention inference with receptive field 256, it attains average $59.87$ and perplexity $15.88$; with receptive field 1024, it attains average $60.27$ and perplexity $15.39$ [2511.20102]. Long-context extrapolation is also reported as unusually strong: in Needle-in-a-Haystack under full-attention inference, SSA maintains $100\%$ at 4k and 8k, $58.8\%$ at 16k, and $31.6\%$ at 32k.

The limitations are correspondingly specific. In trainable video BSA, failure modes include missing salient tokens if centers are poorly chosen or if $r$ is too low, and KV under-selection if $p$ is too high under unusual score distributions; validation loss remains stable up to approximately $0.93$ sparsity and degrades only beyond approximately $0.95$ [2509.01085]. In SVOO, extremely long sequences with low redundancy may require larger $K_q/K_k$ and higher $\alpha$, unusual inputs may benefit from input-aware schedule adjustments, and high-frequency textures or abrupt motion may need more active blocks [2603.18636]. In SSA, one-sided alignment is reported as unstable, and eliminating either stream harms performance, indicating that the “bidirectional” coupling is not merely auxiliary but structurally necessary [2511.20102].

Taken together, these results show that BSA is less a single algorithm than a design principle: sparsity should be imposed in a way that respects the bidirectional structure of the attention problem being solved. In video diffusion training, that structure lies in the simultaneous redundancy of queries and Key–Value pairs [2509.01085]. In training-free video generation, it lies in the coupling between query and key partitions [2603.18636]. In dual-stream long-context training, it lies in reciprocal supervision between sparse and dense pathways [2511.20102].

Source: https://www.emergentmind.com/topics/bidirectional-sparse-attention-bsa