---
title: Factorized Self-Attention in Transformers
url: https://www.emergentmind.com/topics/factorized-self-attention-c50cf3d0-2846-442c-843d-c7d9faf89d13
type: topic
---

# Factorized Self-Attention in Transformers

Factorized self-attention refers broadly to families of architectures, analytical techniques, and parameterizations in which the standard dense attention operations in Transformers are decomposed or factorized—algebraically or structurally—along one or more axes. This factorization targets either (a) greater efficiency (reducing computational and memory complexity), (b) interpretability (disentangling semantic or positional interactions), or (c) architectural modularity (explicitly capturing structure such as spatial, temporal, or spectral separations). Factorization may be realized at the matrix algebra level (e.g., low-rank factorization or Kronecker structure), along data axes (e.g., separable spatiotemporal, spectral-spatial, or interlaced blocks), or even at the level of learned attention weights themselves. This article surveys rigorous methodologies, theoretical frameworks, and empirical outcomes connected to factorized self-attention mechanisms.

## 1. Analytical Frameworks: Bi-Orthogonal Factor Decomposition in Vision Transformers

The Bi-Orthogonal Factor Decomposition (BFD) framework provides a principled methodology for parsing the informational contents exchanged via self-attention in Vision Transformers (ViT) [2601.05328]. BFD proceeds in two complementary steps:

1. **ANOVA-style Decomposition of Token Embeddings:** For each token embedding $f_\ell^{(p)}(x) \in \mathbb{R}^d$ at layer $\ell$ (patch $p$, image $x$), embeddings are decomposed into three mutually orthogonal components (all expectations are over data/image and patch):
   - **Global mean:** $\mu_L = \mathbb{E}_{x,p}[f_\ell^{(p)}(x)]$
   - **Positional bias:** $\mu_P^{(p)} = \mathbb{E}_x[f_\ell^{(p)}(x)] - \mu_L$
   - **Content residual:** $\mu_C^{(p)}(x) = f_\ell^{(p)}(x) - \mu_L - \mu_P^{(p)}$
   By construction, these components are exactly orthogonal.

2. **Spectral (Bi-Orthogonal) Decomposition of Query-Key Matrix:** For a single attention head,
   - Form $Q = A W_Q$ and $K = A W_K$ (with $A$ the token embedding matrix), giving $QK^\top = A W A^\top$, where $W=W_Q W_K^\top$.
   - Singular-value-decompose $W = U \Sigma V^\top$, with $(u_i, v_i, \sigma_i)$ defining bi-orthogonal mode triplets.
   - The raw query–key interaction matrix can be expressed as $QK^\top = \sum_{i=1}^d (A u_i)\,\sigma_i\,(A v_i)^\top$.

Further factorizing the token embedding components, BFD projects each component along these singular directions, enabling precise ANOVA decomposition of attention "energy" into content–content (C–C), content–position (C–P), and position–position (P–P) contributions. Empirically, across ViTs and self-supervised DINOv2 models, the C–C channel dominates, with DINOv2 displaying richer and more distributed mode spectra, stronger C–P coupling, and tighter functional specialization by head and mode [2601.05328].

## 2. Structured and Axis-Wise Factorizations: Spatiotemporal and Spectral-Spatial Decompositions

Factorization by data axis is exemplified by models such as FactoFormer [2309.09431] and spatiotemporal transformers in activity recognition [2310.14416]. These architectures exploit the natural separability of input domains (e.g., space vs. time; spectral vs. spatial) to construct independent transformer branches:

- **FactoFormer:** For hyperspectral data, tokens are constructed as either entire spectral bands at each spatial location ("spectral tokens") or as the full pixel spectrum at each spatial position ("spatial tokens"). Spectral and spatial branches are each processed by independent transformer layers, and their outputs are fused at the classification token ("CLS") level. This enables separate pretraining (self-supervised masking strategies) and is proven to reduce computational cost from $O((BS^2)^2)$ to $O(B^2 + S^4)$ per cube, with B the number of bands and S the patch dimension [2309.09431].

- **Spatiotemporal Factorization (ConViViT):** For video, input is $X \in \mathbb{R}^{T \times N \times d}$ (T frames, N patches), and two independent attention steps are performed: frame-wise spatial self-attention, followed by patch-wise temporal self-attention (or vice versa). Each step applies standard multi-head self-attention along its axis. The overall cost drops from $O((TN)^2 d)$ to $O(T N^2 d + N T^2 d)$, yielding orders-of-magnitude savings and improved performance on activity recognition [2310.14416].

| Model/Method    | Factorization Type          | Main Efficiency Benefit                        |
|-----------------|----------------------------|------------------------------------------------|
| BFD (ViT)       | Statistical factor + SVD   | Interpretable C–C, C–P, P–P decomposition      |
| FactoFormer     | Spectral/Spatial axis      | $O(B^2 + S^4)$ vs $O((B S^2)^2)$               |
| ConViViT/VIViT  | Temporal/Spatial axis      | $O(T N^2 + N T^2)$ vs $O((T N)^2)$             |

## 3. Algebraic Factorization: Low-Rank, Sparse, and Hybrid Attention Mechanisms

Direct algebraic factorization of the attention operation targets memory and compute savings, and includes both low-rank and sparse approaches.

- **Low-Rank Multi-Head Factorization:** In LAMA [1912.00835], each head's bilinear weight matrix $W_i$ is factorized as $P_i Q_i^\top$ (with $P_i, Q_i$ of size $2h \times k, k \ll 2h$), yielding linear-in-sequence (O($T m h$)) rather than quadratic complexity. Multiple heads are bundled via reshaping $P$ and $Q$, and attention is bilinear with respect to a single global context vector.

- **Factorized Synthesizer:** Dot-product attention is replaced by a learned low-rank alignment matrix $S = U V$ ($U \in \mathbb{R}^{N \times k}$, $V \in \mathbb{R}^{k \times N}$). This head is input-agnostic and the softmax is row-wise over $S$. Despite being divorced from the explicit query-key mechanism, in encoding scenarios the factorized Synthesizer matches or outperforms Linformer and Transformer baselines at only $2 N k$ parameter cost [2005.00743].

- **Interlaced Sparse Self-Attention:** The affinity matrix is factorized multiplicatively: $A \approx A^S A^L$, with $A^L$ and $A^S$ each block-diagonal and constructed over long-range and short-range groupings, respectively. The composition ensures any two positions in a feature map communicate in two steps, achieving $O(N \sqrt{N} C)$ compute vs $O(N^2 C)$ for $N$ positions, $C$ channels [1907.12273].

| Variant                 | Main Factorization      | Expressiveness/Trade-off          |
|-------------------------|------------------------|-----------------------------------|
| LAMA                    | Low-rank bilinear      | Context-dependent, linear         |
| Factorized Synthesizer  | Static low-rank align. | Input-agnostic, very efficient    |
| Interlaced Sparse SA    | Block-diag $\times$    | Full coverage, $O(N \sqrt{N})$    |

## 4. Eigenanalysis and Reconstruction: Low-Rank Subspace Structure in Attention

Attention mechanisms in standard Transformers have been empirically observed to inhabit low-dimensional subspaces, as shown by global eigenanalysis of the attention logit matrix over large data distributions [2106.08823]. The principal findings are:

- The covariance of flattened attention logits $S$ (from $Q K^\top$) is nearly low-rank, with the leading $r$ eigencomponents capturing most variance.
- Attention scores for new inputs can be reconstructed from a small sampled subset ($k \ll n^2$) via linear regression using the empirical covariance. This reduces computation from $O(n^2 d)$ to $O(n k d)$ per layer with tolerable error.
- Subspace structure is highly shared across layers and models; thus, pre-computation of reconstruction weights is feasible.

This supports data-driven, empirical low-rank factorization schemes for efficient and accurate self-attention [2106.08823].

## 5. Empirical Outcomes, Design, and Interpretability Implications

Factorized self-attention contributes both to greater interpretability and to practical performance gains:

- **Semantic-Positional Disentanglement:** BFD reveals that most attention energy in both supervised and self-supervised ViTs is allocated to content–content channels. Content–position and position–position channels are nonetheless crucial for specific functional properties, with DINOv2 dedicating systematically more energy to content–position coupling and exhibiting richer, less mode-aligned spectra [2601.05328].
- **Head and Mode Specialization:** Singular modes within attention heads cluster into distinguishable functional classes (C–C, C–P, P–P). This specialization is tighter in self-supervised models.
- **Efficient Factorizations:** Low-rank and axis-structured variants (e.g., LAMA, FactoFormer, Interlaced SA) achieve substantial savings in runtime and parameter count, without sacrificing—or often improving—downstream accuracy on classification and segmentation benchmarks [1912.00835, 2309.09431, 1907.12273].
- **Interpretability:** Mechanistic explanations are enabled: e.g., BFD provides head/mode-level traces to specific (semantic or spatial) factor pairs, interlaced sparse SA covers all positions by design, and low-rank factor models elucidate principal subspaces underlying contextual dependencies.
- **Design Levers:** Factorization introduces architectural and training levers, such as increasing the rank/capacity of query-key interaction ($W_Q W_K^\top$), explicit cross-modal gating, and spectrum-regularizing penalties for attention diversification [2601.05328].

## 6. Generalizations and Application Domains

Factorized self-attention is broadly applicable across data modalities and use cases:

- **Multi-axis Data:** FactoFormer’s dual-branch paradigm extends directly to video (temporal vs. spatial), multivariate time series (feature vs. time), and volumetric imaging (slice vs. region) [2309.09431].
- **Efficient Long-Range Modeling:** Sparse and low-rank factorized models enable tractable self-attention for high-resolution imagery, long documents, videos, and large-scale multimodal fusion.
- **Self-Supervised and Hybrid Training:** Self-supervised, axis-specific pretraining (e.g., masked band or patch prediction) is enabled by axis factorization. Hybrid losses leveraging both positional alignment and semantic contrast are supported [2309.09431, 2601.05328].

## 7. Limitations, Open Problems, and Future Directions

Present factorization schemes involve trade-offs in model expressiveness, input-dependency, and coverage. Static low-rank models (e.g., factorized Synthesizer) are highly efficient but may not capture context-specific dependencies. Sparse/factorized models reconstruct full affinity structure with fewer resources but may require careful groupings to cover all position pairs. Open problems include:

- Systematic evaluation of expressiveness vs. efficiency among algebraic factorization families in high-capacity models.
- Automated selection of optimal rank/cutoffs or permutation groupings for block-sparse compositions.
- Integration of interpretability tools such as BFD with efficient axis-wise or algebraic factorization schemes to unify mechanistic transparency with operational efficiency.
- Exploration of hybrid and dynamically-adaptive factorization strategies, potentially conditioned on input modality or content.

Factorized self-attention provides both an epistemic lens for analyzing arbitrary attention mechanisms and a constructive toolkit for engineering next-generation, efficient, and interpretable transformer architectures across modalities and tasks.

Source: https://www.emergentmind.com/topics/factorized-self-attention-c50cf3d0-2846-442c-843d-c7d9faf89d13