CBSA: Contract-and-Broadcast Self-Attention

• Contract-and-Broadcast Self-Attention (CBSA) is an attention mechanism that contracts global structure using a small set of representative tokens and broadcasts contracted information back to all tokens.
• It is derived from an explicit coding rate and maximal coding rate reduction objective, integrating interpretable subspace projections and learned coefficient matrices.
• CBSA improves efficiency by replacing quadratic token interactions with operations over a few representatives while enhancing model interpretability and robustness.

Contract-and-Broadcast Self-Attention (CBSA) is an attention mechanism in which a small set of representative tokens is used to contract global structure and then broadcast the contracted information back to the full token sequence. In the formulation introduced in "Towards Interpretable and Efficient Attention: Compressing All by Contracting a Few" (Wen et al., 21 Sep 2025), CBSA is derived from an explicit optimization objective based on coding rate and maximal coding rate reduction, rather than from a purely heuristic redesign of self-attention. The mechanism is presented as both inherently interpretable and computationally efficient: interpretability follows from the white-box meaning of its subspace projections, representatives, contraction, and broadcast coefficients, while efficiency follows from replacing token-token interactions over all $N$ tokens by operations over $m \ll N$ representatives. A related earlier line of work is "Scratching Visual Transformer’s Back with Uniform Attention" (Hyeon-Woo et al., 2022), whose Context Broadcasting (CB) module contracts tokens by averaging and broadcasts the result uniformly; that module is not identical to CBSA, but it supplies a closely related contract-and-broadcast design principle in Vision Transformers.

1. Formal definition and conceptual basis

CBSA is defined around the idea of compressing all tokens by contracting a few representative tokens and then broadcasting those contractions back to the full sequence (Wen et al., 21 Sep 2025). Let $Z \in \mathbb{R}^{d \times N}$ denote $N$ tokens in a $d$-dimensional feature space. CBSA introduces a set of representatives $Q \in \mathbb{R}^{d \times m}$ with $m \ll N$, together with a union of subspaces $U = \{U_k \in O(d,p)\}_{k=1}^K$, where each $U_k$ is a $d \times p$ matrix with orthonormal columns and $m \ll N$0 (Wen et al., 21 Sep 2025). The mechanism projects tokens into these subspaces, extracts representatives, contracts those representatives, and broadcasts the resulting contracted information back to all tokens.

The stated motivation is twofold. First, standard softmax self-attention is described as difficult to interpret because post-hoc attention visualizations and gradient-based explanations lack a principled theoretical foundation. Second, its $m \ll N$1 time and memory complexity becomes increasingly prohibitive for long sequences and high-resolution inputs (Wen et al., 21 Sep 2025). CBSA addresses both issues through a single optimization-based construction: the forward operator is obtained by unrolling optimization over a compression objective, so each computational stage has an explicit geometric meaning rather than being treated as a black-box module.

The core conceptual move is not to compress all tokens directly in the ambient space, but to compress a much smaller representative set within low-dimensional subspaces and then redistribute the resulting information. This creates a separation between representation of global structure and redistribution of that structure, which is central to the contract-and-broadcast paradigm (Wen et al., 21 Sep 2025). A plausible implication is that CBSA can be read as a structured decomposition of attention into summary formation and sequence-wide propagation, rather than as direct all-pairs interaction.

2. Optimization objective and derivation

The theoretical basis of CBSA is the coding-rate objective. For tokens $m \ll N$2 and precision $m \ll N$3, the coding rate is defined as

$m \ll N$4

The associated MCR$m \ll N$5 objective seeks representations that are expanded in ambient space while compressed onto a union of subspaces (Wen et al., 21 Sep 2025). With orthonormal bases $m \ll N$6 and projections $m \ll N$7, the original objective is

$m \ll N$8

with

$m \ll N$9

CBSA modifies this formulation by introducing representatives $Z \in \mathbb{R}^{d \times N}$0. Writing $Z \in \mathbb{R}^{d \times N}$1, the modified objective is

$Z \in \mathbb{R}^{d \times N}$2

Because the equality constraint is too restrictive, it is relaxed to

$Z \in \mathbb{R}^{d \times N}$3

and regularized by requiring representatives to be linear combinations of projected tokens: $Z \in \mathbb{R}^{d \times N}$4 This leads to the constrained CoCa objective

$Z \in \mathbb{R}^{d \times N}$5

In this formulation, representatives are required both to preserve subspace coding-rate structure approximately and to remain in the span of the token projections (Wen et al., 21 Sep 2025).

Unrolling gradient descent on the compression term yields the CBSA update

$Z \in \mathbb{R}^{d \times N}$6

where $Z \in \mathbb{R}^{d \times N}$7 is implemented as a learnable scalar and

$Z \in \mathbb{R}^{d \times N}$8

The inverse term is then approximated by a softmax attention matrix, following an approximation used in CRATE, producing the implemented operator

$Z \in \mathbb{R}^{d \times N}$9

In this derivation, contraction corresponds to gradient-driven reduction of representative coding rate, while broadcast applies the representative update back to the original tokens through the coefficient matrices $N$0 (Wen et al., 21 Sep 2025).

3. Architecture and computational structure

In a CBSA-based Transformer layer, tokens are first projected into subspaces: $N$1 Representatives are then initialized by pooling over input tokens, following Agent Attention and related work, and subsequently refined by a cross-attention or MSCA-style extraction step that yields both representatives $N$2 and coefficient matrices $N$3 (Wen et al., 21 Sep 2025). The contraction stage applies self-attention over representatives in each subspace,

$N$4

the broadcast stage forms

$N$5

and aggregation across heads gives

$N$6

The layer output is then updated by a residual step of the form

$N$7

The feed-forward component is interpreted as handling the expansion term $N$8, so the complete attention-plus-FF block is presented as alternating optimization over compression and expansion (Wen et al., 21 Sep 2025).

The efficiency claim follows from replacing operations over all $N$9 tokens by operations over $d$0 representatives. The reported complexity formulas are

$d$1

for full interpretable softmax attention, and

$d$2

for CBSA (Wen et al., 21 Sep 2025). The extraction and broadcast stages cost $d$3, the contraction stage costs $d$4, and the projection term $d$5 is shared with other attention mechanisms. When $d$6 is fixed or grows slowly relative to $d$7, the quadratic $d$8 term is replaced by $d$9 and $Q \in \mathbb{R}^{d \times m}$0, so the attention-specific part scales linearly in $Q \in \mathbb{R}^{d \times m}$1.

This computational profile is accompanied by an interpretability claim. The subspace bases $Q \in \mathbb{R}^{d \times m}$2 are said to represent low-dimensional structure, the representatives $Q \in \mathbb{R}^{d \times m}$3 act as token centroids or dictionary atoms, the contraction term compresses representative structure, and the broadcast coefficients $Q \in \mathbb{R}^{d \times m}$4 encode how representative changes are redistributed to tokens (Wen et al., 21 Sep 2025). This means that representative interactions and token-to-representative assignment weights are meant to be directly inspectable rather than inferred post hoc.

4. Relation to Context Broadcasting in Vision Transformers

A closely related but distinct mechanism appears in "Scratching Visual Transformer’s Back with Uniform Attention" (Hyeon-Woo et al., 2022), which introduces Context Broadcasting (CB) for Vision Transformers. That work begins from the observation that ViT attention maps are empirically high-entropy and close to dense global interactions rather than sparse ones. For attention weights $Q \in \mathbb{R}^{d \times m}$5 with $Q \in \mathbb{R}^{d \times m}$6, entropy is defined as

$Q \in \mathbb{R}^{d \times m}$7

and the reported average entropy per attention map is approximately $Q \in \mathbb{R}^{d \times m}$8, close to the maximum entropy for $Q \in \mathbb{R}^{d \times m}$9 tokens, $m \ll N$0 (Hyeon-Woo et al., 2022). The paper further analyzes the softmax Jacobian

$m \ll N$1

and the nuclear norm proxy

$m \ll N$2

arguing that this quantity is maximized by uniform attention $m \ll N$3 for all $m \ll N$4. The interpretation given is that dense, especially uniform, attention is optimization-unstable but nevertheless strongly preferred by ViTs (Hyeon-Woo et al., 2022).

CB addresses this by injecting dense interactions deterministically rather than forcing multi-head self-attention to maintain them through softmax. For tokens $m \ll N$5, the module is

$m \ll N$6

or, with per-channel scaling,

$m \ll N$7

In the default design, the layer order is MSA block $m \ll N$8 MLP block $m \ll N$9 CB, with CB applied after the MLP (Hyeon-Woo et al., 2022). The implementation can be reduced to the line $m \ll N$50 which introduces one mean over the token dimension, one broadcast add, and one scalar multiply, all of cost $U = \{U_k \in O(d,p)\}_{k=1}^K$0, while adding no parameters in the basic form (Hyeon-Woo et al., 2022).

The relation to CBSA is explicit in the technical exposition provided for (Hyeon-Woo et al., 2022): CB is described as an instantiation of the contract-and-broadcast idea implemented outside the core self-attention module. The contract step is the global average

$U = \{U_k \in O(d,p)\}_{k=1}^K$1

and the broadcast step uniformly injects $U = \{U_k \in O(d,p)\}_{k=1}^K$2 back into every token (Hyeon-Woo et al., 2022). The important distinction is that CB does not use representative extraction, subspace decomposition, or a coding-rate objective; its broadcast is uniform and deterministic rather than mediated by learned token-to-representative coefficients. This suggests that CB can be viewed as a lightweight contract-and-broadcast baseline, whereas CBSA is a more general optimization-derived attention mechanism.

5. Empirical behavior and applications

On image classification, the CBSA paper reports results for CBT-Tiny, CBT-Small, CBT-Base, and CBT-Large, pretrained on ImageNet-1K at $U = \{U_k \in O(d,p)\}_{k=1}^K$3 with patch size $U = \{U_k \in O(d,p)\}_{k=1}^K$4 (Wen et al., 21 Sep 2025). The reported model scales are CBT-Tiny with $U = \{U_k \in O(d,p)\}_{k=1}^K$5M parameters and $U = \{U_k \in O(d,p)\}_{k=1}^K$6G FLOPs, CBT-Small with $U = \{U_k \in O(d,p)\}_{k=1}^K$7M parameters and $U = \{U_k \in O(d,p)\}_{k=1}^K$8G FLOPs, CBT-Base with $U = \{U_k \in O(d,p)\}_{k=1}^K$9M parameters and $U_k$0G FLOPs, and CBT-Large with $U_k$1M parameters and $U_k$2G FLOPs. Against ViT-S, listed as $U_k$3M parameters and $U_k$4G FLOPs, CBT-Small is reported to achieve comparable top-1 accuracy on ImageNet-1K and better performance on several smaller datasets while using approximately $U_k$5 of the parameters and approximately $U_k$6 of the FLOPs (Wen et al., 21 Sep 2025). Controlled comparisons at $U_k$7 tokens further report the following top-1 accuracies: MSSA $U_k$8, CBSA $U_k$9, TSSA $d \times p$0, and Agent $d \times p$1–$d \times p$2–$d \times p$3, together with pairwise-similarity counts of $d \times p$4M/$d \times p$5M for MSSA, $d \times p$6M/$d \times p$7M for CBSA, $d \times p$8M/$d \times p$9M for TSSA, and $m \ll N$00M/$m \ll N$01M for Agent (Wen et al., 21 Sep 2025). These numbers are presented as a trade-off curve between interaction count and accuracy.

For semantic segmentation on ADE20K, CBSA is used in ViT encoders with CBSA decoders formed by stacked CBSA layers without feed-forward blocks, similarly to DEPICT (Wen et al., 21 Sep 2025). The reported result is that the CBSA decoder outperforms both DEPICT and Segmenter while using only approximately $m \ll N$02 FLOPs and a tiny fraction of pairwise similarities, specifically $m \ll N$03 of Segmenter’s in the decoder (Wen et al., 21 Sep 2025). The paper also reports robustness under Gaussian noise perturbations to attention parameters in the decoder, stating that CBSA-based decoders remain robust whereas Segmenter collapses (Wen et al., 21 Sep 2025).

The earlier Context Broadcasting work provides a complementary empirical picture. On ImageNet classification with non-distilled DeiT-style models, the reported results are: ViT-Ti baseline $m \ll N$04 top-1, $m \ll N$05 $m \ll N$06, $m \ll N$07 $m \ll N$08; ViT-S baseline $m \ll N$09, $m \ll N$10 $m \ll N$11, $m \ll N$12 $m \ll N$13; and ViT-B baseline $m \ll N$14, $m \ll N$15 $m \ll N$16, $m \ll N$17 $m \ll N$18 (Hyeon-Woo et al., 2022). On ADE20K with UperNet, ViT-Ti at $m \ll N$19K iterations improves from $m \ll N$20 mIoU to $m \ll N$21 with CB, and ViT-B from $m \ll N$22 to $m \ll N$23 (Hyeon-Woo et al., 2022). Robustness results for ViT-S include center occlusion from $m \ll N$24 to $m \ll N$25, ImageNet-A from $m \ll N$26 to $m \ll N$27, and FGSM from $m \ll N$28 to $m \ll N$29 with CB (Hyeon-Woo et al., 2022). These findings are not CBSA results proper, but they reinforce the broader contract-and-broadcast theme by showing that explicit dense global mixing can improve generalization and robustness in visual token models.

6. Special cases, limitations, and open questions

A central claim of CBSA is that it generalizes several attention mechanisms as special cases by changing the number and structure of representatives (Wen et al., 21 Sep 2025). If each token is its own representative, so that $m \ll N$30, $m \ll N$31, and $m \ll N$32, CBSA reduces to the MSSA operator

$m \ll N$33

which is the full interpretable softmax self-attention form and remains quadratic in $m \ll N$34 (Wen et al., 21 Sep 2025). If representatives are orthogonal and related to principal directions of $m \ll N$35, the resulting form is described as a linear-attention variant; if subspace bases are fixed principal directions for any input, the mechanism reduces to the Token-Statistics Self-Attention (TSSA) channel-attention form

$m \ll N$36

where $m \ll N$37 is diagonal and its entries are deterministic functions of second moments (Wen et al., 21 Sep 2025). Agent Attention is likewise presented as a CBSA variant obtained when contraction is fused into extraction or removed, so that representatives act as global agents without explicit contraction (Wen et al., 21 Sep 2025).

Several limitations are also stated explicitly. CBSA does not eliminate the projection bottleneck $m \ll N$38 associated with $m \ll N$39, so although the attention-specific term is linearized in $m \ll N$40, total complexity remains sensitive to large $m \ll N$41 (Wen et al., 21 Sep 2025). Its modeling assumption—that tokens lie near a union of low-dimensional subspaces—is said to be well-motivated for images but may be less accurate for some modalities or tasks (Wen et al., 21 Sep 2025). The softmax replacement of the contraction inverse introduces overparameterization and potential sign ambiguities that are absorbed in $m \ll N$42, which weakens theoretical rigor for practical performance (Wen et al., 21 Sep 2025). The paper also notes a decompression phenomenon in early layers, where coding rate sometimes increases before later compression, and identifies extension to language and very long sequences as an open question (Wen et al., 21 Sep 2025).

The Context Broadcasting precursor raises related but distinct caveats. Its benefits are larger on smaller ViTs than on ViT-B; as the number of attention heads increases, its marginal benefit declines (Hyeon-Woo et al., 2022). Some CB-S settings slightly reduce ADE20K mIoU, and class-token-based context broadcasting is reported to decrease ImageNet accuracy from $m \ll N$43 to $m \ll N$44, whereas average pooling is more effective (Hyeon-Woo et al., 2022). Those observations suggest that the choice of contract operator and the depth at which broadcasting is applied are consequential design variables. In the CB paper, applying broadcasting only in upper layers, denoted $m \ll N$45, improves ViT-Ti from $m \ll N$46 to $m \ll N$47 and ViT-S from $m \ll N$48 to $m \ll N$49, relative to applying CB in all layers (Hyeon-Woo et al., 2022). A plausible implication for CBSA is that representative-based broadcasting may be especially beneficial in deeper semantic layers, where the model’s preference for dense global interaction is strongest.

Taken together, these works define a spectrum of contract-and-broadcast mechanisms. Context Broadcasting (Hyeon-Woo et al., 2022) supplies a deterministic global average branch that relieves self-attention from learning uniform dense interactions, while CBSA (Wen et al., 21 Sep 2025) embeds contraction and broadcast inside a principled representative-based attention operator derived from coding-rate optimization. The common premise is that dense global mixing need not be realized solely through all-pairs softmax attention; it can instead be factored into explicit summarization and redistribution steps, with different trade-offs among interpretability, efficiency, and expressive power.