---
title: Factorized Random Synthesizer
url: https://www.emergentmind.com/topics/factorized-random-synthesizer
type: topic
---

# Factorized Random Synthesizer

A Factorized Random Synthesizer is a variant of the Synthesizer architecture that replaces the standard self-attention mechanism in Transformers with a synthetic attention pattern generated by low-rank, trainable matrices. Unlike dot-product attention, which computes attention weights via token-token (query-key) interactions, the Factorized Random Synthesizer employs a low-rank decomposition of a learned alignment matrix, resulting in reduced parameter costs while maintaining competitive empirical performance across diverse tasks such as machine translation, language modeling, and encoding benchmarks [2005.00743].

## 1. Mathematical Formulation

The standard Transformer self-attention employs query ($Q_h$), key ($K_h$), and value ($V_h$) projections:
- $Q_h = XW^Q_h$, $K_h = XW^K_h$, $V_h = XW^V_h$ for input $X \in \mathbb R^{L \times d}$ (sequence of length $L$ and model dimension $d$).
- Attention weights: $A_h^{\mathrm{dot}} = \mathrm{softmax}(Q_h K_h^T / \sqrt{d_k}) \in \mathbb R^{L \times L}$.
- Output: $Y_h = A_h^{\mathrm{dot}} V_h$.

The (full) Random Synthesizer omits $Q$ and $K$, introducing instead a learned alignment matrix $R_h \in \mathbb R^{L \times L}$:
- Attention: $A_h^{\mathrm{rand}} = \mathrm{softmax}(R_h)$ (row-wise).
- Output: $Y_h = A_h^{\mathrm{rand}} V_h$.

In the Factorized Random Synthesizer, $R_h$ is approximated via a low-rank factorization:
$$
R_h = U_h V_h^T,\qquad U_h \in \mathbb R^{L \times r},\; V_h \in \mathbb R^{L \times r},\; r \ll L.
$$
The resultant attention weights and output are:
$$
A_h = \mathrm{softmax}(U_h V_h^T), \qquad Y_h = A_h V_h(X).
$$
This replaces the $L^2$ parameter matrix with two $L \times r$ matrices ($2Lr$ parameters per head).

## 2. Architectural Differences and Parameter Analysis

Compared to the vanilla Transformer:
- Standard attention head (per-head dimension $d_k = d/h$) requires $2d d_k$ parameters for $Q+K$ projections.
- Full Random Synthesizer head forgoes $Q,K$, adds $L^2$ trainable parameters for $R_h$.
- Factorized Random Synthesizer head forgoes $Q,K$, adds $2Lr$ parameters for $U_h,V_h$.
- In both Synthesizer variants, the $V$ projection (dimension $d \times d_k$), output projection, and feed-forward sublayers are unchanged.

Relative to baseline, the factorized variant decreases parameter count (saving the $Q,K$ projections), trading it for low-rank factor parameters. When $r \ll L$, the reduction is substantial.

| Method                    | Project Q/K | Alignment Matrix         | #Parameters (per head)  |
|---------------------------|-------------|-------------------------|-------------------------|
| Vanilla Self-Attention    | Yes         | None                    | $2dd_k$                 |
| Full Random Synthesizer   | No          | $R_h \in \mathbb R^{L\times L}$  | $L^2$                   |
| Factorized Random Synthesizer | No      | $U_h,V_h \in \mathbb R^{L\times r}$ | $2Lr$                   |

## 3. Complexity Analysis

Let $n=L$ and $d_k\approx d/h$:
- **Dot-product Attention**: Time $O(n^2 d_k)$ (for $QK^T$ and $A_h V_h$), memory $O(n^2)$ for $A_h$, parameters $O(d d_k)$.
- **Full Random Synthesizer**: Time $O(n^2 + n^2 d_k) \approx O(n^2 d_k)$, identical memory, parameters $n^2$.
- **Factorized Random Synthesizer**: Forms $U_h V_h^T$ ($O(n^2 r)$), softmax, multiply $A_h V_h(X)$ ($O(n^2 d_k)$); total $O(n^2(r + d_k))$.
  - If $A_h$ is not stored but contractively applied, memory/time can reduce to $O(nr + r d_k)$ and $O(n r d_k)$, though the reference implementation does not exploit this.

The factorized form maintains $O(n^2)$ time/memory complexity when $r$ is small relative to $n$, but with lower parameter cost.

## 4. Empirical Performance Across Tasks

The Factorized Random Synthesizer mirrors the full random variant's performance with much lower parameter overhead:
- **Machine Translation** (WMT’14 En→De, En→Fr; BLEU): Factorized Random (r=8) yields 27.30 (EnDe) and 41.12 (EnFr), nearly matching full random (27.27/41.12) and falling ~0.4 BLEU below vanilla Transformer (27.67/41.57).
- **Language Modeling** (LM1B, PPL): Factorized Random (r=8) PPL=42.40, versus full random 40.60, and vanilla 38.21; thus, $\sim$2 PPL worse than vanilla.
- **Masked LM on C4**: Full Random Synthesizer achieves log PPL=1.972 at 4.26 steps/sec (about 60% faster than Dynamic Convolution, which gets log PPL=2.040 at 2.65 steps/sec).
- **Multi-task Fine-tuning (GLUE/SuperGLUE)**: Pure Random or Dense Synthesizers underperform on cross-sentence tasks due to lack of cross-attention mechanisms, but mixture models (Random+Vanilla, Dense+Vanilla) outperform base T5 (e.g., GLUE: T5=83.5, +Random+Vanilla=84.1; SuperGLUE: T5=70.3, +Random+Vanilla=72.2).
- **Encoding Tasks/Comparison to Linformer**: On AG News and MR Reviews (Linformer at $k=32$ yields 86.50%/82.86%), Factorized Random (r=32) matches or slightly exceeds (86.53%/83.39%).

These results indicate that for a wide range of tasks, low-rank Synthesizers achieve competitive or superior results to efficient attention baselines such as Dynamic Convolution and Linformer, with similar or improved speed [2005.00743].

## 5. Effect of Factorization Rank and Learned Patterns

Empirical ablations varying rank $r$ show minor performance degradation up to $r\approx 8$ or $16$; at $r=8$, nearly the full performance of full-rank random Synthesizer is recovered. Visualizations demonstrate that, following training, the learned $U_hV_h^T$ matrices exhibit both local and global alignment patterns analogous to those produced by dot-product attention. This suggests that low-rank random attention can spontaneously capture structurally meaningful alignments. Mixture models that combine small amounts of Dense Synthesizer or vanilla self-attention restore or exceed base Transformer performance, underscoring that the factorized global structure is the principal contributor to alignment [2005.00743].

## 6. Limitations and Observed Constraints

Pure Random and Dense Synthesizer models (lacking $Q,K$) underperform on cross-sentence tasks, reflecting their inability to model certain long-range dependencies when cross-attention is required. Mixture architectures mitigate this deficiency. Since the reference implementation does not exploit memory-efficient computation exploiting low-rank structure for $A_h$, memory and time complexity remain $O(L^2)$ for long sequence lengths, although parameter savings are realized. A plausible implication is that specialized kernels leveraging factorization could further scale efficiency for extreme $L$.

## 7. Relationship to Efficient-Attention and Transformer Variants

The factorized random approach contrasts with other efficient-attention architectures by decoupling attention construction from input-dependent ($Q,K$) products, instead generating synthetic alignment through learned low-rank interactions. Compared to Linformer, which projects key/value matrices along the sequence axis to reduce sequence length, and Dynamic Convolution, which uses localized convolutions, the Factorized Random Synthesizer achieves similar or better accuracy and throughput on encoding-based tasks. Compositional variants—combining factorized random with dot-product attention—consistently surpass Transformer and alternative baselines in aggregate benchmarks, as evidenced by improvements on GLUE/SuperGLUE [2005.00743].

Source: https://www.emergentmind.com/topics/factorized-random-synthesizer