Factorized Self-Attention Architecture

• Factorized self-attention architecture is a method that reduces computational complexity by decomposing dense attention matrices into efficient, lower-dimensional forms.
• It leverages techniques like low-rank approximation, block-diagonal sparsification, and separable attention to maintain long-range dependencies while decreasing memory and compute costs.
• Empirical benchmarks in NLP, vision, and edge-device applications show competitive performance with significant speedups and reduced parameter footprints.

Factorized self-attention architectures are a class of models that reduce the computational and parameter complexity of standard self-attention by decomposing or approximating the dense $N\times N$ affinity or attention matrices into lower-rank, block-diagonal, sparse, or separable forms. The primary motivation is to overcome the quadratic scaling bottleneck of the attention mechanism with respect to the sequence length (or image size), without sacrificing the ability to model long-range dependencies. These architectures appear in diverse settings, including natural language processing, computer vision, and edge-device-friendly networks, and often lead to dramatic improvements in speed, memory footprint, and robustness.

1. Core Principles of Factorized Self-Attention

Factorized self-attention refers to methods that replace the standard dense dot-product self-attention with an attention computation that is decomposed—either by low-rank approximation, blocking/sparsification, parameter sharing, or separability. The classical attention computes $A = \mathrm{softmax}(QK^\top/\sqrt{d})$, where $A \in \mathbb{R}^{N\times N}$. For large $N$, this step dominates FLOPs and memory.

Different factorization strategies have been employed:

• Low-Rank Decomposition: Approximating $A$ by $UV^\top$ with $U, V \in \mathbb{R}^{N\times r}$ and $r \ll N$.
• Block-Diagonal/Sparse Factorization: Partitioning $A$ into smaller submatrices that capture only a restricted set of interactions per block, reducing complexity to $O(N^{3/2})$ or lower.
• Separable Attention: Decoupling spatial and temporal (or channel) interactions, so attention operates along one axis at a time.
• Synthetic/Parameter-Generated Attention: Learning attention maps directly as parameterized matrices or via MLPs, sometimes further factored into smaller tensors.
• Hierarchical and Multi-Group Factorized Heads: Distributing the attention computation over several groups or heads, each responsible for integrating information at different granularities.

2. Formal Design Patterns and Mathematical Formulations

Representative architectures instantiate factorization in distinct ways:

2.1. Random and Dense Factorized Synthesizer (Tay et al., 2020)

• Random Factorized Synthesizer: Instead of computing $A = QK^\top$, each attention head learns two smaller matrices $R_1 \in \mathbb{R}^{N \times k}$, $R_2 \in \mathbb{R}^{k\times N}$, forming $C = R_1 R_2$. The attention matrix is $A = \mathrm{softmax}(C)$.
• Dense Factorized Synthesizer: Per token $i$, compute $A_i$ and $B_i$ via MLPs, then outer-product/reshape $C(i, j) = A_i B_j$ to assemble the $N\times N$ weights.

2.2. Low-Rank Multi-Head Structures (Mehta et al., 2019)

• For input $X \in \mathbb{R}^{n \times d}$, project into low-rank spaces with $U = X W_U$, $V = X W_V$ ($W_U, W_V \in \mathbb{R}^{d\times r}$), so $A \approx U V^\top$ ($r \ll n$).
• For bilinear attention with global context vector $c$, attention scores become $$f_t = c^\top W_i u_t = (P^\top c) \circ (Q^\top u_t)$$, with $W_i=P Q^\top$.

2.3. Block-Factorization and Interlaced Sparse Attention (Huang et al., 2019)

• Deconstruct the dense $N\times N$ affinity into the product $$\mathbf{A} \approx \mathbf{A}^{(2)}\mathbf{A}^{(1)}$$, where both $\mathbf{A}^{(1)}$ and $\mathbf{A}^{(2)}$ are highly sparse block-diagonal matrices. One captures long-range, another short-range dependencies.

2.4. Double-Condensing Attention Condenser (Wong et al., 2022)

• Employs two sequential condensation operations: first, spatial/channel pooling plus channel projection, then a further projection to produce ultra-low-dimensional $Q, K, V$, followed by attention in the small space and an expansion back to the original resolution.

2.5. Structured or Axis-Aligned Factorization

• In spatiotemporal or video models (Dokkar et al., 2023), factorization applies attention over space and time separately: first, per-frame attention, then per-patch temporal attention.
• In vision transformers (Qin et al., 2023), the attention matrix is approximated by aggregating channel-wise groups that operate on sparse, dilated windows across the spatial domain.

3. Computational Complexity and Parameterization Analysis

Model Type	Parameter Count	Attention Cost	Scaling
Standard $N\times N$ dot-product	$O(d^2)$ per $Q,K,V$ proj	$O(N^2 d)$	Quadratic
Random Synthesizer ($N\times N$ learnable)	$O(N^2)$	$O(N^2)$	Quadratic
Factorized Random Synthesizer ($N\times k$)	$O(Nk)$ per factor	$O(Nk)$	Linear
Low-Rank Factorized (LAMA, (Mehta et al., 2019))	$O(ndr)$, $r\ll n$	$O(n r d)$	Linear in $n$
Interlaced Sparse SA (Huang et al., 2019)	$O(nC^2)$ blocks, $O(n^{3/2})$ total	$O(N^{3/2} C)$	Subquadratic
Double-Condensing (Wong et al., 2022)	$O((d_1+d)C)$, $d,d_1 \ll C$	$O(N d_1 d)$	Subquadratic
FaSA (Qin et al., 2023)	$O(N M^2 C)$	$O(N M^2 C)$ ($M \ll \sqrt{N}$)	Linear

By reducing the effective rank, block size, or token count participating in each attention computation, these architectures achieve dramatic reductions in both parameters and floating-point operations, particularly as $N$ grows.

4. Empirical Effects and Benchmark Results

Factorized self-attention designs have been empirically validated across modalities:

• Encoding-Only NLP Tasks: Factorized Random Synthesizer matches or exceeds Linformer and vanilla Transformer on AGNews and Movie Review classification, and mixtures with vanilla heads outperform both (Tay et al., 2020).
• Machine Translation and Language Modeling: BLEU and perplexity remain highly competitive; e.g., WMT14 En–De BLEU: Factorized Random (27.30) vs. Transformer (27.67).
• Video and Vision: ConViViT factorized space-time attention achieves state-of-the-art activity recognition on HMDB51 (90.05%), UCF101, and ETRI-Activity3D (Dokkar et al., 2023).
• Semantic Segmentation: Interlaced Sparse Self-Attention (ISA) on ResNet-101 obtains 81.4% mIoU on Cityscapes, matching or exceeding dense-attention baselines, with reduced memory and compute (Huang et al., 2019).
• Edge/TinyML: AttendNeXt, based on Double-Condensing Attention Condenser, achieves 75.8% ImageNet accuracy with a 3.2MB model and $>$10× speedup versus MobileOne and MobileNetV3-L (Wong et al., 2022).
• Robustness and Generalization: FaViT (FaSA) improves ImageNet-C robustness by 7% and classification accuracy by 1% over Swin-T (Qin et al., 2023).

5. Architectural Variants and Hybrid Designs

A spectrum of architectures incorporate factorized attention:

• Synthesizer Family (Tay et al., 2020): Includes fully random, factorized random, dense, factorized dense, and mixtures with vanilla dot-product heads.
• LAMA (Mehta et al., 2019): Factorized bilinear multi-head attention using a global context vector.
• ConViViT (Dokkar et al., 2023): CNN-based spatial encoder followed by sequential factorized spatial and temporal attention layers.
• ISA (Huang et al., 2019): Two-stage block-diagonal sparse attention for high-resolution vision.
• FaSA/FaViT (Qin et al., 2023): Channel-wise decomposition with multi-dilated windowed sub-attentions, reassembled for hierarchical integration.
• AttendNeXt (Wong et al., 2022): Multi-columner, multi-stage stacking of double-condensing factorized attention modules.

6. Limitations, Applicability, and Theoretical Considerations

Factorized self-attention excels when $N$ is large, such as in document classification, long-context modeling, high-resolution vision, or video processing. However, several trade-offs and limitations have been observed:

• Global Pattern Limitation: Methods that learn attention weights independently of input tokens (e.g., Random Synthesizer) are not suitable for cross-attention scenarios (QA, NLI), as the same global pattern is used for all samples (Tay et al., 2020).
• Short-Sequence Inefficiency: For small $N$, factorization overhead may offset benefits.
• Approximation Error: Sparsified/block-diagonal decompositions approximate the full affinity matrix, but empirical results demonstrate negligible error for typical data due to inherent low-rank structure (Huang et al., 2019).
• Inductive Bias and Interpretability: Explicitly factorized models in space/time yield more structured inductive bias, e.g., first fusing spatial features, then modeling their evolution, which can aid interpretability (Dokkar et al., 2023).

7. Future Directions and Generalization

Emerging trends in factorized self-attention research include:

• Hybridization of dense and factorized heads within a shared multi-head block to balance expressivity and efficiency (Tay et al., 2020).
• Data-dependent or adaptive rank/block selection to optimize trade-offs per sequence or modality (Mehta et al., 2019, Wong et al., 2022).
• Hierarchical and multi-level condensation beyond two-stage factorizations.
• Extension to arbitrary graph structures, higher-dimensional data, or multimodal transformers.

Factorized self-attention remains a fundamental direction for scaling transformers and attention-based models to longer inputs, larger images, and specialized inference regimes, while maintaining or enhancing model quality and robustness.