Factorized Random Synthesizer

• Factorized Random Synthesizer is a Transformer variant that replaces standard Q/K projections with a low-rank factorization of a learned alignment matrix.
• It achieves significant parameter savings by decomposing the full attention matrix into two smaller matrices while retaining competitive empirical performance.
• Empirical results show that low-rank approximations recover nearly full performance on tasks like machine translation and language modeling, despite being about 0.4 BLEU or 2 PPL behind vanilla Transformers.

A Factorized Random Synthesizer is a variant of the Synthesizer architecture that replaces the standard self-attention mechanism in Transformers with a synthetic attention pattern generated by low-rank, trainable matrices. Unlike dot-product attention, which computes attention weights via token-token (query-key) interactions, the Factorized Random Synthesizer employs a low-rank decomposition of a learned alignment matrix, resulting in reduced parameter costs while maintaining competitive empirical performance across diverse tasks such as machine translation, language modeling, and encoding benchmarks (Tay et al., 2020).

1. Mathematical Formulation

The standard Transformer self-attention employs query ($Q_h$), key ($K_h$), and value ($V_h$) projections:

• $Q_h = XW^Q_h$, $K_h = XW^K_h$, $V_h = XW^V_h$ for input $X \in \mathbb R^{L \times d}$ (sequence of length $L$ and model dimension $d$).
• Attention weights: $$A_h^{\mathrm{dot}} = \mathrm{softmax}(Q_h K_h^T / \sqrt{d_k}) \in \mathbb R^{L \times L}$$.
• Output: $K_h$0.

The (full) Random Synthesizer omits $K_h$1 and $K_h$2, introducing instead a learned alignment matrix $K_h$3:

• Attention: $K_h$4 (row-wise).
• Output: $K_h$5.

In the Factorized Random Synthesizer, $K_h$6 is approximated via a low-rank factorization:

$K_h$7

The resultant attention weights and output are:

$K_h$8

This replaces the $K_h$9 parameter matrix with two $V_h$0 matrices ($V_h$1 parameters per head).

2. Architectural Differences and Parameter Analysis

Compared to the vanilla Transformer:

• Standard attention head (per-head dimension $V_h$2) requires $V_h$3 parameters for $V_h$4 projections.
• Full Random Synthesizer head forgoes $V_h$5, adds $V_h$6 trainable parameters for $V_h$7.
• Factorized Random Synthesizer head forgoes $V_h$8, adds $V_h$9 parameters for $Q_h = XW^Q_h$0.
• In both Synthesizer variants, the $Q_h = XW^Q_h$1 projection (dimension $Q_h = XW^Q_h$2), output projection, and feed-forward sublayers are unchanged.

Relative to baseline, the factorized variant decreases parameter count (saving the $Q_h = XW^Q_h$3 projections), trading it for low-rank factor parameters. When $Q_h = XW^Q_h$4, the reduction is substantial.

Method	Project Q/K	Alignment Matrix	#Parameters (per head)
Vanilla Self-Attention	Yes	None	$Q_h = XW^Q_h$5
Full Random Synthesizer	No	$Q_h = XW^Q_h$6	$Q_h = XW^Q_h$7
Factorized Random Synthesizer	No	$Q_h = XW^Q_h$8	$Q_h = XW^Q_h$9

3. Complexity Analysis

Let $K_h = XW^K_h$0 and $K_h = XW^K_h$1:

• Dot-product Attention: Time $K_h = XW^K_h$2 (for $K_h = XW^K_h$3 and $K_h = XW^K_h$4), memory $K_h = XW^K_h$5 for $K_h = XW^K_h$6, parameters $K_h = XW^K_h$7.
• Full Random Synthesizer: Time $K_h = XW^K_h$8, identical memory, parameters $K_h = XW^K_h$9.
• Factorized Random Synthesizer: Forms $V_h = XW^V_h$0 ($V_h = XW^V_h$1), softmax, multiply $V_h = XW^V_h$2 ($V_h = XW^V_h$3); total $V_h = XW^V_h$4.
  • If $V_h = XW^V_h$5 is not stored but contractively applied, memory/time can reduce to $V_h = XW^V_h$6 and $V_h = XW^V_h$7, though the reference implementation does not exploit this.

The factorized form maintains $V_h = XW^V_h$8 time/memory complexity when $V_h = XW^V_h$9 is small relative to $X \in \mathbb R^{L \times d}$0, but with lower parameter cost.

4. Empirical Performance Across Tasks

The Factorized Random Synthesizer mirrors the full random variant's performance with much lower parameter overhead:

• Machine Translation (WMT’14 En→De, En→Fr; BLEU): Factorized Random (r=8) yields 27.30 (EnDe) and 41.12 (EnFr), nearly matching full random (27.27/41.12) and falling ~0.4 BLEU below vanilla Transformer (27.67/41.57).
• Language Modeling (LM1B, PPL): Factorized Random (r=8) PPL=42.40, versus full random 40.60, and vanilla 38.21; thus, $X \in \mathbb R^{L \times d}$12 PPL worse than vanilla.
• Masked LM on C4: Full Random Synthesizer achieves log PPL=1.972 at 4.26 steps/sec (about 60% faster than Dynamic Convolution, which gets log PPL=2.040 at 2.65 steps/sec).
• Multi-task Fine-tuning (GLUE/SuperGLUE): Pure Random or Dense Synthesizers underperform on cross-sentence tasks due to lack of cross-attention mechanisms, but mixture models (Random+Vanilla, Dense+Vanilla) outperform base T5 (e.g., GLUE: T5=83.5, +Random+Vanilla=84.1; SuperGLUE: T5=70.3, +Random+Vanilla=72.2).
• Encoding Tasks/Comparison to Linformer: On AG News and MR Reviews (Linformer at $X \in \mathbb R^{L \times d}$2 yields 86.50%/82.86%), Factorized Random (r=32) matches or slightly exceeds (86.53%/83.39%).

These results indicate that for a wide range of tasks, low-rank Synthesizers achieve competitive or superior results to efficient attention baselines such as Dynamic Convolution and Linformer, with similar or improved speed (Tay et al., 2020).

5. Effect of Factorization Rank and Learned Patterns

Empirical ablations varying rank $X \in \mathbb R^{L \times d}$3 show minor performance degradation up to $X \in \mathbb R^{L \times d}$4 or $X \in \mathbb R^{L \times d}$5; at $X \in \mathbb R^{L \times d}$6, nearly the full performance of full-rank random Synthesizer is recovered. Visualizations demonstrate that, following training, the learned $X \in \mathbb R^{L \times d}$7 matrices exhibit both local and global alignment patterns analogous to those produced by dot-product attention. This suggests that low-rank random attention can spontaneously capture structurally meaningful alignments. Mixture models that combine small amounts of Dense Synthesizer or vanilla self-attention restore or exceed base Transformer performance, underscoring that the factorized global structure is the principal contributor to alignment (Tay et al., 2020).

6. Limitations and Observed Constraints

Pure Random and Dense Synthesizer models (lacking $X \in \mathbb R^{L \times d}$8) underperform on cross-sentence tasks, reflecting their inability to model certain long-range dependencies when cross-attention is required. Mixture architectures mitigate this deficiency. Since the reference implementation does not exploit memory-efficient computation exploiting low-rank structure for $X \in \mathbb R^{L \times d}$9, memory and time complexity remain $L$0 for long sequence lengths, although parameter savings are realized. A plausible implication is that specialized kernels leveraging factorization could further scale efficiency for extreme $L$1.

7. Relationship to Efficient-Attention and Transformer Variants

The factorized random approach contrasts with other efficient-attention architectures by decoupling attention construction from input-dependent ($L$2) products, instead generating synthetic alignment through learned low-rank interactions. Compared to Linformer, which projects key/value matrices along the sequence axis to reduce sequence length, and Dynamic Convolution, which uses localized convolutions, the Factorized Random Synthesizer achieves similar or better accuracy and throughput on encoding-based tasks. Compositional variants—combining factorized random with dot-product attention—consistently surpass Transformer and alternative baselines in aggregate benchmarks, as evidenced by improvements on GLUE/SuperGLUE (Tay et al., 2020).