---
title: Factorized Self-Attention
url: https://www.emergentmind.com/topics/factorized-self-attention
type: topic
---

# Factorized Self-Attention

Factorized self-attention encompasses a family of architectural and algorithmic strategies for decomposing, approximating, or structurally modifying the standard dot-product self-attention mechanism in order to reduce computational and memory overhead, control parameter footprint, or enforce inductive priors. Across diverse neural models—including Transformers, convolutional recurrent cells, and hybrid architectures—factorization techniques have been exploited to enable efficient long-sequence modeling, enhance spatial-temporal expressiveness, and provide interpretability advantages. These approaches can involve low-rank factorizations of the attention (alignment) matrix, sequential or blockwise decomposition, structured sparsification, or analytic disentanglement of positional and content-related communication.

## 1. Mathematical Principles of Factorized Self-Attention

Factorized self-attention emerges from the observation that the canonical attention alignment matrix, $A = \mathrm{softmax}(QK^T)$, is often highly redundant: either intrinsically low-rank or amenable to sparse or structured factor representations in practical data regimes.

- **Low-rank (bilinear) factorization**: The alignment matrix $A\in\mathbb{R}^{N\times N}$ may be approximated by $A\approx AB$ where $A\in\mathbb{R}^{N\times k}$, $B\in\mathbb{R}^{k\times N}$ for $k\ll N$, as in the Factorized Random Synthesizer and LAMA attention [2005.00743][1912.00835].
- **Structured block factorization**: Decompose attention along spatial, temporal, or head/channel axes, performing attention within sub-blocks, then fusing outputs. Examples include spatial-temporal factorizations for video (ConViViT, ViViT) and blockwise sparse affinity products as in interlaced sparse self-attention [2310.14416][1907.12273].
- **Algorithmic low-rank approximation**: Use partial eigen-decomposition or covariance-driven schemes to reconstruct full attention from a small set of computed affinities [2106.08823].
- **Synthesis via learned factors**: Replace $QK^T$ entirely with a parameterized or random factorized progression, as in Synthesizers, decoupling attention from explicit query-key token interactions [2005.00743].
  
These factorizations decrease quadratic cost in sequence/spatial length to linear or subquadratic, with modest trade-offs in expressivity.

## 2. Taxonomy of Factorization Techniques

A broad division of factorized self-attention approaches can be made as follows:

| Approach                       | Method                         | Core Formula or Strategy                         |
|------------------------------- |------------------------------- |--------------------------------------------------|
| Low-rank factor synthesis      | Random or learnable $R_1R_2$   | $A = \mathrm{softmax}(R_1 R_2)$                  |
| Dense MLP-based factorization  | Per-token param. $a_i, b_j$    | $A_{i,j}\sim H_A(a_i) \cdot H_B(b_j)$           |
| Bilinear compact models        | Global context, low-rank maps  | $f_t=c^TPQ^Tu_t$, $W_i\approx PQ^T$              |
| Eigen/covariance reconstruction| Subset computation + SVD/Sigma | $S_r = U_rU_r^TS$, $a_{P̄}=R^* a_P$              |
| Block-structural decomposition | Spatial/temporal/focal blocks  | $A \approx \prod_k A^{(k)}$                      |
| Sparse/dilated window factor   | Window select, fusion          | Split input, sample/sparse keys, fuse            |

Methods such as the Factorized Random Synthesizer [2005.00743], LAMA [1912.00835], and Linformer employ low-rank parameterizations. Structured variants for spatiotemporal or spatial arrangements (FaViT [2312.08614], ConViViT [2310.14416], interlaced sparse [1907.12273]) use block decomposition or interleaving. Reconstruction-based methods exploit the empirical low effective rank of real attention matrices [2106.08823].

## 3. Algorithmic Workflows and Complexity

### Factorized Random Synthesizer ([2005.00743]):
1. **Parameter initialization**: $R_1\in\mathbb{R}^{N\times k}$, $R_2\in\mathbb{R}^{k\times N}$.
2. **Alignment logits**: $C = R_1 R_2$ ($O(Nk)$).
3. **Row-softmax**: $A = \mathrm{softmax}(C)$ ($O(N^2)$).
4. **Value projection & output**: $Y = AV$ ($O(N^2 d)$).

### Factorized Dense Synthesizer:
1. **Per-position factor extraction**: $a_i=F_A(x_i)$, $b_j=F_B(x_j)$.
2. **Matrix assembly**: $C_{i,j}=[H_A(a_i)]_j \cdot [H_B(b_j)]_i$.
3. **Softmax, output as above**.

### Bilinear/LAMA Compact Form ([1912.00835]):
- Replace standard per-position queries with a single global context $c$ and utilize low-rank factorization $W_i \approx PQ^T$.

### Covariance/Eigendecomposition ([2106.08823]):
- Compute small subset of attention scores, reconstruct remainder via trained $R^*$ or projection in eigenbasis.

### Block-Sparse/Interlaced ([1907.12273], [2312.08614]):
- Partition feature/function space, perform local sparse attention, then combine outputs via permutation or aggregation to achieve global mixing.

### Complexity Reduction:
- Standard self-attention: $O(N^2 d)$ (memory: $O(N^2)$).
- Factorized random: $O(N^2 d + N k)$, with memory $O(N k)$.
- Windowed/block: $O(N M^2 d)$ for window size $M \ll N$.
- Eigen/covariance: $O(n k d + n k)$ for $k$ computed entries per row.

## 4. Applications, Benchmarks, and Empirical Findings

Factorized self-attention has demonstrated competitive or superior performance across machine translation, language modeling, classification, semantic segmentation, time series, and video understanding:

- **Synthesizer variants**: Nearly matches or marginally trails vanilla Transformers on WMT'14 MT (27.30 BLEU vs 27.67, $O(Nk)$ parameter cost) and LM1B (40.6 PPL vs 38.1), but as a hybrid significantly outperforms on GLUE/SuperGLUE (+0.6, +1.9 over T5 base) [2005.00743].
- **FaSA/FaViT**: Matches Swin-T's efficiency but surpasses in both accuracy (+1%) and robustness (+6.6pp on ImageNet-C), and further improves instance/semantic segmentation [2312.08614].
- **LAMA**: Provides up to a 65% parameter reduction on text tasks (News, Reuters, IMDB), with similar or slightly better accuracy than BERT or CNN/GRU models [1912.00835].
- **Eigen/covariance reconstruction**: Yields $25$–$50\%$ FLOP reduction at cost of only $2$pp drop in MNLI downstream accuracy [2106.08823].
- **Spatiotemporal factorization**: In video, factorized SA (spatial then temporal) outperforms both full and parallel (dot-product) attention, as evidenced by ConViViT’s state-of-the-art results on HMDB51 (90.05%) and others [2310.14416].

Across these tasks, pure factorized modules alone typically trail full dot-product SA by a small margin, but hybridization (composition) yields best-in-class results, indicating strong complementarity [2005.00743].

## 5. Implementation Strategies and Practical Caveats

- **Parameter dependency on sequence length**: Most factorized schemes (especially explicit $R_1/R_2$ ones) scale parameter count with $N$ (the longest sequence), requiring either truncation or tiling for variable-length data [2005.00743].
- **Factor size/rank selection**: Optimal $k$ ($k \ll N$) must balance expressivity and compactness to avoid underfitting or overfitting; practical values are often $k=8$–$32$ [2005.00743][1912.00835].
- **Stability/regularization**: Lower parameter count aids overfitting control; no special stabilization tricks are needed beyond standard training recipes [2005.00743].
- **Sparse/global context**: Some factorized blocks allow information propagation across the global context by permutation, fusion, or occasional global steps (e.g., axial attention every $K$ steps in FAConvLSTM [2601.10914]).
- **Implementation details**: Fast variants exploit modern deep learning primitives (unfold/sampling/aggregation), and can support dynamic cropping or windowing at inference [2312.08614][1907.12273].

## 6. Variants and Theoretical Analysis

- **Statistical/analytic factorization**: Recent analytic approaches such as Bi-Orthogonal Factor Decomposition (BFD) do not modify the architecture but instead provide decompositional insight into the separation of positional and content effects in learned attention matrices [2601.05328].
- **Partial/approximate computation**: Covariance-driven selection schemes offer theoretical guarantees on error vs. cost and exploit the empirical concentration of attention in low-dimensional eigenspaces [2106.08823].
- **Interpretability**: Attention heads or singular modes induced by factorization acquire specialized roles (content-content vs. content-position), which correlate with robust shape/semantic sensitivity in self-supervised vision transformers [2601.05328].

## 7. Limitations and Future Extensions

- **Granularity and information loss**: Factorization imposes structural biases and may discard subtle correlations (e.g., very sparse keys in FaSA might miss certain dependencies) [2312.08614].
- **Sequence length scaling remains nontrivial**, as most schemes retain $O(N^2)$ computation for value mixing unless projection or window size is aggressively minimized.
- **Potential enhancements**: Learnable or adaptive fusion across blocks, hybridization with global tokens, and integration with efficient patch-embedding or MLP architectures constitute active directions. Statistical diagnostic methods may further refine or inform new factorization strategies [2312.08614][2601.05328].

---

Factorized self-attention subsumes a diverse set of methods for reducing the computational and memory burden of self-attention, exploiting empirical rank structure or modularizing attention for spatial, temporal, or hybrid architectures. These techniques offer strong empirical utility on large-scale tasks and serve as a foundation for further theoretical and practical advances in efficient neural sequence modeling [2005.00743][2312.08614][1907.12273][1912.00835][2106.08823][2310.14416][2601.05328][2601.10914].

Source: https://www.emergentmind.com/topics/factorized-self-attention