---
title: 'CBSA: Contract-and-Broadcast Self-Attention'
url: https://www.emergentmind.com/topics/contract-and-broadcast-self-attention-cbsa
type: topic
---

# CBSA: Contract-and-Broadcast Self-Attention

Contract-and-Broadcast Self-Attention (CBSA) is an attention mechanism in which a small set of representative tokens is used to contract global structure and then broadcast the contracted information back to the full token sequence. In the formulation introduced in "Towards Interpretable and Efficient Attention: Compressing All by Contracting a Few" [2509.16875], CBSA is derived from an explicit optimization objective based on coding rate and maximal coding rate reduction, rather than from a purely heuristic redesign of self-attention. The mechanism is presented as both inherently interpretable and computationally efficient: interpretability follows from the white-box meaning of its subspace projections, representatives, contraction, and broadcast coefficients, while efficiency follows from replacing token-token interactions over all $N$ tokens by operations over $m \ll N$ representatives. A related earlier line of work is "Scratching Visual Transformer’s Back with Uniform Attention" [2210.08457], whose Context Broadcasting (CB) module contracts tokens by averaging and broadcasts the result uniformly; that module is not identical to CBSA, but it supplies a closely related contract-and-broadcast design principle in Vision Transformers.

## 1. Formal definition and conceptual basis

CBSA is defined around the idea of **compressing all tokens by contracting a few representative tokens and then broadcasting those contractions back to the full sequence** [2509.16875]. Let $Z \in \mathbb{R}^{d \times N}$ denote $N$ tokens in a $d$-dimensional feature space. CBSA introduces a set of representatives $Q \in \mathbb{R}^{d \times m}$ with $m \ll N$, together with a union of subspaces $U = \{U_k \in O(d,p)\}_{k=1}^K$, where each $U_k$ is a $d \times p$ matrix with orthonormal columns and $pK=d$ [2509.16875]. The mechanism projects tokens into these subspaces, extracts representatives, contracts those representatives, and broadcasts the resulting contracted information back to all tokens.

The stated motivation is twofold. First, standard softmax self-attention is described as difficult to interpret because post-hoc attention visualizations and gradient-based explanations lack a principled theoretical foundation. Second, its $O(N^2)$ time and memory complexity becomes increasingly prohibitive for long sequences and high-resolution inputs [2509.16875]. CBSA addresses both issues through a single optimization-based construction: the forward operator is obtained by unrolling optimization over a compression objective, so each computational stage has an explicit geometric meaning rather than being treated as a black-box module.

The core conceptual move is not to compress all tokens directly in the ambient space, but to compress a much smaller representative set within low-dimensional subspaces and then redistribute the resulting information. This creates a separation between **representation of global structure** and **redistribution of that structure**, which is central to the contract-and-broadcast paradigm [2509.16875]. A plausible implication is that CBSA can be read as a structured decomposition of attention into summary formation and sequence-wide propagation, rather than as direct all-pairs interaction.

## 2. Optimization objective and derivation

The theoretical basis of CBSA is the coding-rate objective. For tokens $Z \in \mathbb{R}^{d \times N}$ and precision $\epsilon > 0$, the coding rate is defined as
\[
R(Z) \doteq \frac{1}{2} \log \det\left(I_N + \frac{d}{N \epsilon^2} Z^\top Z\right).
\]
The associated MCR$^2$ objective seeks representations that are expanded in ambient space while compressed onto a union of subspaces [2509.16875]. With orthonormal bases $U_k \in O(d,p)$ and projections $Z_k \doteq U_k^\top Z$, the original objective is
\[
\max_Z \Delta R(Z) \doteq R(Z) - R_c(Z \mid U),
\]
with
\[
R_c(Z \mid U) \doteq R(Z) - \sum_{k=1}^K R(Z_k).
\]

CBSA modifies this formulation by introducing representatives $Q$. Writing $Q_k = U_k^\top Q \in \mathbb{R}^{p \times m}$, the modified objective is
\[
\max_Z \left[ R(Z) - \sum_{k=1}^K R(Q_k) \right]
\quad \text{s.t.} \quad
\sum_{k=1}^K R(Q_k) = \sum_{k=1}^K R(Z_k).
\]
Because the equality constraint is too restrictive, it is relaxed to
\[
|R(Q_k)-R(Z_k)| \le \tau, \quad \forall k,
\]
and regularized by requiring representatives to be linear combinations of projected tokens:
\[
Q_k = Z_k A_k, \quad A_k \in \mathbb{R}^{N \times m}.
\]
This leads to the constrained CoCa objective
\[
\begin{aligned}
\min_Z & \quad \sum_{k=1}^K R(Q_k) \\
\text{s.t.} & \quad |R(Q_k)-R(Z_k)| \le \tau, \\
& \quad Q_k = Z_k A_k, \quad \forall k \in [K].
\end{aligned}
\]
In this formulation, representatives are required both to preserve subspace coding-rate structure approximately and to remain in the span of the token projections [2509.16875].

Unrolling gradient descent on the compression term yields the CBSA update
\[
Z \gets Z - \kappa \,\operatorname{CBSA}(Z \mid U),
\]
where $\kappa$ is implemented as a learnable scalar and
\[
\operatorname{CBSA}(Z \mid U)
\doteq
\sum_{k=1}^K
U_k
\underbrace{
U_k^\top Q
\left(
I_m + \frac{p}{m^2}(U_k^\top Q)^\top(U_k^\top Q)
\right)^{-1}
}_{\text{Contraction}}
\underbrace{A_k^\top}_{\text{Broadcast}}.
\]
The inverse term is then approximated by a softmax attention matrix, following an approximation used in CRATE, producing the implemented operator
\[
\operatorname{CBSA}(Z \mid U)
\doteq
\sum_{k=1}^K
U_k
\underbrace{
U_k^\top Q \,
\operatorname{softmax}\left((U_k^\top Q)^\top(U_k^\top Q)\right)
}_{\text{Contraction via self-attention}}
\underbrace{A_k^\top}_{\text{Broadcast}}.
\]
In this derivation, contraction corresponds to gradient-driven reduction of representative coding rate, while broadcast applies the representative update back to the original tokens through the coefficient matrices $A_k$ [2509.16875].

## 3. Architecture and computational structure

In a CBSA-based Transformer layer, tokens are first projected into subspaces:
\[
Z_k = U_k^\top Z \in \mathbb{R}^{p \times N}.
\]
Representatives are then initialized by pooling over input tokens, following Agent Attention and related work, and subsequently refined by a cross-attention or MSCA-style extraction step that yields both representatives $Q_k$ and coefficient matrices $A_k$ [2509.16875]. The contraction stage applies self-attention over representatives in each subspace,
\[
C_k = U_k^\top Q \,\operatorname{softmax}\left((U_k^\top Q)^\top(U_k^\top Q)\right),
\]
the broadcast stage forms
\[
B_k = C_k A_k^\top \in \mathbb{R}^{p \times N},
\]
and aggregation across heads gives
\[
\operatorname{CBSA}(Z) = \sum_{k=1}^K U_k B_k.
\]
The layer output is then updated by a residual step of the form
\[
Z_{\text{out}} = Z - \kappa\,\operatorname{CBSA}(Z) + \text{(expansion / FF via ISTA)}.
\]
The feed-forward component is interpreted as handling the expansion term $R(Z)$, so the complete attention-plus-FF block is presented as alternating optimization over compression and expansion [2509.16875].

The efficiency claim follows from replacing operations over all $N$ tokens by operations over $m$ representatives. The reported complexity formulas are
\[
\Omega(\mathrm{MSSA}) = 2Nd^2 + 2N^2d,
\]
for full interpretable softmax attention, and
\[
\Omega(\mathrm{CBSA}) = 2Nd^2 + 3Nmd + 2m^2d,
\]
for CBSA [2509.16875]. The extraction and broadcast stages cost $O(Nmd)$, the contraction stage costs $O(m^2d)$, and the projection term $O(Nd^2)$ is shared with other attention mechanisms. When $m$ is fixed or grows slowly relative to $N$, the quadratic $N^2d$ term is replaced by $Nmd$ and $m^2d$, so the attention-specific part scales linearly in $N$.

This computational profile is accompanied by an interpretability claim. The subspace bases $U_k$ are said to represent low-dimensional structure, the representatives $Q$ act as token centroids or dictionary atoms, the contraction term compresses representative structure, and the broadcast coefficients $A_k^\top$ encode how representative changes are redistributed to tokens [2509.16875]. This means that representative interactions and token-to-representative assignment weights are meant to be directly inspectable rather than inferred post hoc.

## 4. Relation to Context Broadcasting in Vision Transformers

A closely related but distinct mechanism appears in "Scratching Visual Transformer’s Back with Uniform Attention" [2210.08457], which introduces **Context Broadcasting (CB)** for Vision Transformers. That work begins from the observation that ViT attention maps are empirically high-entropy and close to dense global interactions rather than sparse ones. For attention weights $a \in \mathbb{R}^N$ with $\sum_j a_j=1$, entropy is defined as
\[
H(a) = -\sum_{j=1}^N a_j \log a_j,
\]
and the reported average entropy per attention map is approximately $4.4$, close to the maximum entropy for $N=197$ tokens, $\log N \approx 5.3$ [2210.08457]. The paper further analyzes the softmax Jacobian
\[
J_{jk} = \frac{\partial a_j}{\partial s_k} = \lambda(\delta_{j=k}a_j-a_ja_k),
\]
and the nuclear norm proxy
\[
\|J\|_* = \sum_j (a_j-a_j^2),
\]
arguing that this quantity is maximized by uniform attention $a_j=\frac{1}{N}$ for all $j$. The interpretation given is that dense, especially uniform, attention is optimization-unstable but nevertheless strongly preferred by ViTs [2210.08457].

CB addresses this by injecting dense interactions deterministically rather than forcing multi-head self-attention to maintain them through softmax. For tokens $x_i \in \mathbb{R}^d$, the module is
\[
CB(x_i)=\frac{x_i + \frac{1}{N}\sum_{j=1}^N x_j}{2},
\]
or, with per-channel scaling,
\[
CB_S(x_i)=x_i+\lambda \odot \left(\frac{1}{N}\sum_{j=1}^N x_j\right).
\]
In the default design, the layer order is MSA block $\rightarrow$ MLP block $\rightarrow$ CB, with CB applied after the MLP [2210.08457]. The implementation can be reduced to the line
```python
X = 0.5 * X + 0.5 * X.mean(dim=1, keepdim=True)
```
which introduces one mean over the token dimension, one broadcast add, and one scalar multiply, all of cost $O(Nd)$, while adding no parameters in the basic form [2210.08457].

The relation to CBSA is explicit in the technical exposition provided for [2210.08457]: CB is described as an instantiation of the contract-and-broadcast idea implemented outside the core self-attention module. The **contract** step is the global average
\[
\bar{x}=\frac{1}{N}\sum_{j=1}^N x_j,
\]
and the **broadcast** step uniformly injects $\bar{x}$ back into every token [2210.08457]. The important distinction is that CB does not use representative extraction, subspace decomposition, or a coding-rate objective; its broadcast is uniform and deterministic rather than mediated by learned token-to-representative coefficients. This suggests that CB can be viewed as a lightweight contract-and-broadcast baseline, whereas CBSA is a more general optimization-derived attention mechanism.

## 5. Empirical behavior and applications

On image classification, the CBSA paper reports results for CBT-Tiny, CBT-Small, CBT-Base, and CBT-Large, pretrained on ImageNet-1K at $224\times224$ with patch size $16$ [2509.16875]. The reported model scales are CBT-Tiny with $1.8$M parameters and $1.1$G FLOPs, CBT-Small with $6.7$M parameters and $4.0$G FLOPs, CBT-Base with $25.7$M parameters and $15.1$G FLOPs, and CBT-Large with $83.1$M parameters and $47.3$G FLOPs. Against ViT-S, listed as $22.1$M parameters and $9.8$G FLOPs, CBT-Small is reported to achieve comparable top-1 accuracy on ImageNet-1K and better performance on several smaller datasets while using approximately $30\%$ of the parameters and approximately $40\%$ of the FLOPs [2509.16875]. Controlled comparisons at $192$ tokens further report the following top-1 accuracies: MSSA $64.7/72.1$, CBSA $63.2/71.4$, TSSA $61.2/68.5$, and Agent $62.7$–$63.8/70.6$–$71.8$, together with pairwise-similarity counts of $1.4$M/$2.8$M for MSSA, $0.53$M/$1.1$M for CBSA, $0.45$M/$0.91$M for TSSA, and $0.52$M/$1.0$M for Agent [2509.16875]. These numbers are presented as a trade-off curve between interaction count and accuracy.

For semantic segmentation on ADE20K, CBSA is used in ViT encoders with CBSA decoders formed by stacked CBSA layers without feed-forward blocks, similarly to DEPICT [2509.16875]. The reported result is that the CBSA decoder outperforms both DEPICT and Segmenter while using only approximately $20\%$ FLOPs and a tiny fraction of pairwise similarities, specifically $0.06\%$ of Segmenter’s in the decoder [2509.16875]. The paper also reports robustness under Gaussian noise perturbations to attention parameters in the decoder, stating that CBSA-based decoders remain robust whereas Segmenter collapses [2509.16875].

The earlier Context Broadcasting work provides a complementary empirical picture. On ImageNet classification with non-distilled DeiT-style models, the reported results are: ViT-Ti baseline $72.2\%$ top-1, $+CB$ $73.2\%$, $+CB_S$ $73.5\%$; ViT-S baseline $79.9\%$, $+CB$ $80.5\%$, $+CB_S$ $80.4\%$; and ViT-B baseline $81.8\%$, $+CB$ $82.0\%$, $+CB_S$ $82.1\%$ [2210.08457]. On ADE20K with UperNet, ViT-Ti at $40$K iterations improves from $35.5$ mIoU to $36.5$ with CB, and ViT-B from $44.3$ to $45.1$ [2210.08457]. Robustness results for ViT-S include center occlusion from $73.0$ to $74.0$, ImageNet-A from $19.0$ to $21.2$, and FGSM from $27.2$ to $32.3$ with CB [2210.08457]. These findings are not CBSA results proper, but they reinforce the broader contract-and-broadcast theme by showing that explicit dense global mixing can improve generalization and robustness in visual token models.

## 6. Special cases, limitations, and open questions

A central claim of CBSA is that it generalizes several attention mechanisms as special cases by changing the number and structure of representatives [2509.16875]. If each token is its own representative, so that $Q=Z$, $m=N$, and $A_k=I_N$, CBSA reduces to the MSSA operator
\[
\operatorname{MSSA}(Z \mid U)
\doteq
\sum_{k=1}^K
U_k U_k^\top Z
\operatorname{softmax}\left((U_k^\top Z)^\top(U_k^\top Z)\right),
\]
which is the full interpretable softmax self-attention form and remains quadratic in $N$ [2509.16875]. If representatives are orthogonal and related to principal directions of $U_k^\top Z$, the resulting form is described as a linear-attention variant; if subspace bases are fixed principal directions for any input, the mechanism reduces to the Token-Statistics Self-Attention (TSSA) channel-attention form
\[
\sum_{k=1}^K U_k D_k U_k^\top Z,
\]
where $D_k$ is diagonal and its entries are deterministic functions of second moments [2509.16875]. Agent Attention is likewise presented as a CBSA variant obtained when contraction is fused into extraction or removed, so that representatives act as global agents without explicit contraction [2509.16875].

Several limitations are also stated explicitly. CBSA does not eliminate the projection bottleneck $O(Nd^2)$ associated with $Z \mapsto U_k^\top Z$, so although the attention-specific term is linearized in $N$, total complexity remains sensitive to large $d$ [2509.16875]. Its modeling assumption—that tokens lie near a union of low-dimensional subspaces—is said to be well-motivated for images but may be less accurate for some modalities or tasks [2509.16875]. The softmax replacement of the contraction inverse introduces overparameterization and potential sign ambiguities that are absorbed in $\kappa$, which weakens theoretical rigor for practical performance [2509.16875]. The paper also notes a decompression phenomenon in early layers, where coding rate sometimes increases before later compression, and identifies extension to language and very long sequences as an open question [2509.16875].

The Context Broadcasting precursor raises related but distinct caveats. Its benefits are larger on smaller ViTs than on ViT-B; as the number of attention heads increases, its marginal benefit declines [2210.08457]. Some CB-S settings slightly reduce ADE20K mIoU, and class-token-based context broadcasting is reported to decrease ImageNet accuracy from $79.9$ to $79.3$, whereas average pooling is more effective [2210.08457]. Those observations suggest that the choice of contract operator and the depth at which broadcasting is applied are consequential design variables. In the CB paper, applying broadcasting only in upper layers, denoted $CB^\dagger$, improves ViT-Ti from $73.2$ to $73.4$ and ViT-S from $80.5$ to $80.8$, relative to applying CB in all layers [2210.08457]. A plausible implication for CBSA is that representative-based broadcasting may be especially beneficial in deeper semantic layers, where the model’s preference for dense global interaction is strongest.

Taken together, these works define a spectrum of contract-and-broadcast mechanisms. Context Broadcasting [2210.08457] supplies a deterministic global average branch that relieves self-attention from learning uniform dense interactions, while CBSA [2509.16875] embeds contraction and broadcast inside a principled representative-based attention operator derived from coding-rate optimization. The common premise is that dense global mixing need not be realized solely through all-pairs softmax attention; it can instead be factored into explicit summarization and redistribution steps, with different trade-offs among interpretability, efficiency, and expressive power.

Source: https://www.emergentmind.com/topics/contract-and-broadcast-self-attention-cbsa