Query-Broadcast Attention

• Query-broadcast attention is a mechanism that contracts token representations into a compact set and broadcasts critical information back to the full sequence for efficiency.
• The CBSA method employs a two-step block-coordinate process—contraction and broadcast—to achieve linear computational complexity and improved interpretability.
• In diffusion models, Pyramid Attention Broadcast reuses attention outputs across timesteps, significantly reducing redundant computation in video generation.

Query-broadcast attention encompasses a family of attention mechanisms that compress or cache attention computations—typically by contracting token representations into a compact set and then broadcasting information back to the full sequence—to improve efficiency and interpretability. Prominent methods include Contract-and-Broadcast Self-Attention (CBSA) for efficient representation learning in transformers and Pyramid Attention Broadcast (PAB) for minimizing redundant attention computation in diffusion-based video generation. Both approaches exemplify query-broadcast dynamics: a small set of queries (or representatives) are computed or identified, attention is computed on these, and their outputs are broadcast back, reducing the need for frequent or full attention computation.

1. Contract-and-Broadcast Self-Attention (CBSA): Theoretical Underpinnings

CBSA is derived from a unified optimization perspective, starting with a maximal coding-rate reduction (MCR²) objective that seeks to compress token sets into a compact yet informative representation. This approach explicitly formalizes the goals of interpretability and efficiency within self-attention layers.

The MCR² objective for tokens $Z \in \mathbb{R}^{d \times N}$ is given as: $$\max_{Z}\;\Delta R(Z) = R(Z)\;-\;\sum_{k=1}^K R(U_k^\top Z),$$ where $U_k \in O(d, p)$ spans $K$ incoherent $p$-dimensional subspaces and $R(\cdot)$ is a coding-rate function based on log-determinant regularization.

CBSA introduces a small set of representatives $Q_k \in \mathbb{R}^{d \times m}$ per subspace, enforcing that compressing these provides equivalent information as contracting all tokens in that subspace. This yields the regularized objective: $$\min_{Z}\; \sum_{k=1}^K R(Q_k) \quad\text{s.t.}\quad |R(Q_k)-R(U_k^\top Z)|\le\tau, \quad Q_k=Z A_k,$$ with $A_k\in\mathbb R^{N\times m}$ the contraction coefficients (Wen et al., 21 Sep 2025).

2. Algorithmic Realization: Contract–and–Broadcast Flow

CBSA unrolls a two-step block-coordinate optimization—contract and broadcast—into a purely feedforward operator:

• Contraction: Project each token block into its $p$-dimensional head subspace, extract $$\max_{Z}\;\Delta R(Z) = R(Z)\;-\;\sum_{k=1}^K R(U_k^\top Z),$$0 representatives via cross-attention, then perform self-attention-based contraction over them. The contraction per head $$\max_{Z}\;\Delta R(Z) = R(Z)\;-\;\sum_{k=1}^K R(U_k^\top Z),$$1 is:

$$\max_{Z}\;\Delta R(Z) = R(Z)\;-\;\sum_{k=1}^K R(U_k^\top Z),$$2

• Broadcast: Broadcast the contracted information across all tokens using the attention coefficient transpose:

$$\max_{Z}\;\Delta R(Z) = R(Z)\;-\;\sum_{k=1}^K R(U_k^\top Z),$$3

The final CBSA transformation is: $$\max_{Z}\;\Delta R(Z) = R(Z)\;-\;\sum_{k=1}^K R(U_k^\top Z),$$4 where contraction and broadcast are computed per attention head (Wen et al., 21 Sep 2025).

3. Complexity and Generalization

CBSA achieves linear complexity in the number of tokens $$\max_{Z}\;\Delta R(Z) = R(Z)\;-\;\sum_{k=1}^K R(U_k^\top Z),$$5 for practical parameter regimes $$\max_{Z}\;\Delta R(Z) = R(Z)\;-\;\sum_{k=1}^K R(U_k^\top Z),$$6, as compared to quadratic complexity in softmax self-attention. Specifically:

• CBSA per-layer complexity: $$\max_{Z}\;\Delta R(Z) = R(Z)\;-\;\sum_{k=1}^K R(U_k^\top Z),$$7.
• Memory usage: $$\max_{Z}\;\Delta R(Z) = R(Z)\;-\;\sum_{k=1}^K R(U_k^\top Z),$$8 for activations, versus $$\max_{Z}\;\Delta R(Z) = R(Z)\;-\;\sum_{k=1}^K R(U_k^\top Z),$$9 for full softmax attention.
• Special cases: CBSA recovers full softmax (when $U_k \in O(d, p)$0 and $U_k \in O(d, p)$1), linear attention (when $U_k \in O(d, p)$2 are principal components of $U_k \in O(d, p)$3), and channel attention (when $U_k \in O(d, p)$4 are fixed orthonormal bases) (Wen et al., 21 Sep 2025).

4. Broadcast Attention and Redundancy in Diffusion

Broadcast attention has further applications in iterative architectures like diffusion models, where per-step computations exhibit high redundancy. In Pyramid Attention Broadcast (PAB) for DiT-based video diffusion, broadcast attention directly reuses the output of an attention layer over multiple timesteps, conditioned on measured redundancy.

• Formal definition: If $U_k \in O(d, p)$5 is the output at broadcast origin $U_k \in O(d, p)$6, then for $U_k \in O(d, p)$7, set $U_k \in O(d, p)$8 instead of recomputing attention at each step.
• Redundancy diagnosis: The per-step output change $U_k \in O(d, p)$9 is empirically U-shaped, with high variance at the ends but minimal in the central 70% of diffusion steps (Zhao et al., 2024).

5. Pyramid-Style and Distributed Broadcast Mechanisms

PAB adapts broadcast window sizes hierarchically according to attention type:

• Spatial attention: Shortest broadcast range ($K$0)
• Temporal attention: Intermediate ($K$1)
• Cross attention: Longest ($K$2)

The algorithm proceeds as follows:

$K$6

PAB further extends to distributed settings (broadcast sequence parallel), eliminating inter-GPU communication during broadcasted temporal attention steps, reducing required bandwidth and increasing throughput (Zhao et al., 2024).

6. Empirical Findings and Comparative Performance

CBSA

• Compression via representatives: Coding-rate $K$3 progressively decreases over CBSA layers, well-correlated with classification accuracy; contraction over a handful of representatives suffices.
• Role of broadcast: Ablation experiments removing the broadcast step (i.e., using identity for $K$4) result in considerable accuracy degradation.
• Emergent segmentation and robustness: Early layers show emergent object-segmentation properties; model is robust to perturbations of the basis $K$5.
• Efficiency: On ImageNet-1K, a CBSA-Small model matches ViT-Small accuracy (≈71.4%) with ∼40% of the pairwise similarity operations of standard attention.

PAB

• Speedup: On single-GPU (Open-Sora, 30 steps, 480p video), PAB with (2,4,6) broadcasting achieves 1.34× acceleration with negligible quality drop (<1% VBench loss). More aggressive ranges (3,5,7) and (5,7,9) further improve speed at slight cost to quality.
• Scaling: On 8×H100 GPUs, broadcast sequence parallel yields 10.6× end-to-end speedup, with ∼50% communication reduction.
• Latencies: Attention operations constitute 10–20% of runtime, with attention-related overhead (norm, proj, reshape) ∼30%. PAB eliminates most of this overhead in redundant steps (Zhao et al., 2024).

7. Connections, Special Cases, and Implications

Both CBSA and PAB generalize classical attention via query-broadcast mechanisms:

• CBSA as a unifying framework: Varies the number of representatives, projection structure, and contraction mechanics to interpolate between full, linear, and channel attention.
• PAB in iterative models: Empirically validates that much attention activity is redundant across diffusion steps, motivating output-level reuse without retraining.

A plausible implication is that structured query-broadcast paradigms will continue to underlie both general-purpose transformers and domain-specific architectures where efficiency and interpretability are paramount. These methods further suggest that the contraction–broadcast decomposition is a powerful axis for analyzing and optimizing attention beyond mere computation reduction.

---

References:

• "Towards Interpretable and Efficient Attention: Compressing All by Contracting a Few" (Wen et al., 21 Sep 2025)
• "Real-Time Video Generation with Pyramid Attention Broadcast" (Zhao et al., 2024)