---
title: FAVOR+ Attention in Scalable Transformers
url: https://www.emergentmind.com/topics/favor-attention
type: topic
---

# FAVOR+ Attention in Scalable Transformers

FAVOR+ Attention (Fast Attention Via positive Orthogonal Random Features) is a Monte Carlo kernel approximation scheme for scalable attention in neural sequence models, designed to approximate the softmax attention mechanism with provable accuracy while reducing time and space complexity from quadratic to linear in sequence length. This mechanism is central to Performer and subsequent architectures such as DF-Conformer, enabling practical modeling of long sequences in settings where explicit computation of the attention matrix is computationally prohibitive [2009.14794][2511.02454].

## 1. Motivation and Problem Setting

Traditional softmax attention in Transformers requires computation and storage of a full $L \times L$ attention matrix (where $L$ is the sequence length), resulting in $O(L^2 d)$ time and $O(L^2)$ memory complexity. For tasks with long contexts (e.g., sequence lengths $L \gg 10^3$), these requirements are prohibitively expensive. To address this, FAVOR+ constructs an unbiased, low-variance approximation to the softmax attention kernel, reducing complexity without introducing sparsity or low-rank constraints [2009.14794].

## 2. Mathematical Foundation: Kernelization of Softmax Attention

The central observation is that softmax attention is kernelizable:
\[
A_{ij} = \exp\left( \frac{q_i \cdot k_j}{\sqrt{d}} \right) = K(q_i, k_j)
\]
with $K(x, y) = \exp(x^\top y)$. For any positive-definite kernel $K(x, y)$ expressible as an expectation over features, $K(x, y) = \mathbb{E}_\omega[\phi(x; \omega)\phi(y; \omega)]$, the attention mechanism can be approximated in feature space. This approach enables reframing matrix products involving $A$ in linear rather than quadratic time, provided a suitable $\phi$ can be constructed [2009.14794][2511.02454].

## 3. FAVOR+ Random Feature Maps

FAVOR+ uses positive orthogonal random features to approximate the exponential kernel. The construction proceeds as follows:

- For each input $x \in \mathbb R^d$, and a set of $m$ orthogonal random vectors $\omega_1, \dots, \omega_m$, define
  \[
  \phi_{\omega_i}^+(x) = \exp(\omega_i^\top x - \|x\|^2/2), \quad i=1,\dots,m
  \]
- For reduced variance, a "hyp+" variant uses pairs $(\omega_i, -\omega_i)$ and outputs
  \[
  \phi^{\mathrm{hyp+}}_{\omega_i}(x) = \frac{1}{\sqrt{2}} \left( \exp(\omega_i^\top x - \|x\|^2/2), \exp(-\omega_i^\top x - \|x\|^2/2) \right)
  \]
- Orthogonality of $\omega_i$ is achieved by sampling a Gaussian matrix, then orthonormalizing its rows (gram–schmidt, Householder, or Hadamard methods).

These features are positive-valued, preserve unbiasedness, and crucially reduce variance relative to i.i.d. sampled features [2009.14794][2511.02454].

## 4. Linear-Time Attention via FAVOR+

Given matrices $Q, K, V \in \mathbb{R}^{L \times d}$, the transformation proceeds as follows:

1. Compute random-feature projections:
   \[
   Q' = \phi(Q) \in \mathbb{R}^{L \times r}, \qquad K' = \phi(K) \in \mathbb{R}^{L \times r}
   \]
2. Approximate the attention output:
   \[
   \mathrm{approxAtt}(Q, K, V) = D^{-1} Q' (K'^\top V)
   \]
   where $D = \mathrm{diag}(Q' (K'^\top 1_L))$

Time and space complexity per block become $O(L r d)$ and $O(L r)$, with $r$ typically $256$–$1024$, representing a substantial reduction compared to the $O(L^2 d)$ and $O(L^2)$ baseline [2009.14794][2511.02454].

## 5. Theoretical Guarantees

FAVOR+ is a provably unbiased estimator:
\[
\mathbb{E}_{\omega}[\phi_\omega(x)\phi_\omega(y)] = \exp(x^\top y)
\]
Variance bounds are established, with mean-squared error for the approximation decreasing as $O(1/m)$. Orthogonal features further reduce variance, with
\[
\mathrm{Var}_{\mathrm{ORF}+} \leq \mathrm{Var}_{\mathrm{PRF}+} - \Delta(d, m)
\]
Uniform convergence guarantees ensure that, for sufficient $m = O(d/\delta^2 \cdot \log(\sigma \cdot \mathrm{diam} / \delta))$, the attention matrix approximation satisfies $\max_{i,j} |{\hat A_{ij} - A_{ij}}| \leq \varepsilon$ with high probability [2009.14794].

## 6. Implementation in Performer and DF-Conformer Architectures

FAVOR+ first appears as the core attention mechanism in Performer [2009.14794], replacing only the self-attention block and maintaining compatibility with the rest of the Transformer architecture (residuals, layer normalization, feed-forward network). In DF-Conformer [2511.02454], FAVOR+ is embedded in each block's attention sublayer as follows:

- For $h$ heads and per-head dimension $d_h = d/h$, apply random-feature projection $W_i \in \mathbb{R}^{r \times d_h}$
- Compute random-feature matrices $\Phi_{Q_i}$, $\Phi_{K_i}$
- Linearly aggregate values through $\Phi_{K_i}^\top V_i$, then combine using $\Phi_{Q_i}$
- Per-row normalization with $D_{i, t} = \sum_{j=1}^T [\Phi_{Q_i}]_{t:} [\Phi_{K_i}]_{j:}^T$
- Favor+ is often sandwiched between local convolutional modules (e.g., depthwise dilated convolution) for joint modeling of local and global dependencies

Typical hyperparameters include $d \in \{256, 512\}$, $h \in \{4, 8\}$, per-head random features $r \approx 128$–$256$, and rotary embeddings prior to projection. All operations are amenable to efficient GPU/TPU acceleration due to batching and matrix multiplications [2009.14794][2511.02454].

## 7. Empirical Behavior, Limitations, and Successors

FAVOR+ enables scaling attention models to sequence lengths of $8~\mathrm{k}$–$32~\mathrm{k}$, where baseline Transformers are infeasible due to memory exhaustion. Empirical studies demonstrate linear scaling of compute and memory, up to $8\times$ speedup in large-$L$ regimes, and retention of modeling accuracy on diverse tasks—image generation, text modeling, and protein sequence modeling. In domains such as speech enhancement (Genhancer), FAVOR+ underpins efficient variants like DF-Conformer, though recent research highlights potential gains from replacing FAVOR+ with state-space sequence models for global dependency modeling while retaining linear complexity [2511.02454].

A salient aspect is the trade-off between unbiased approximation and approximation error: while FAVOR+ reduces computational burden and preserves full-rank attention properties, a finite number of random features can introduce deviations from softmax attention. Fine-tuning the number of random features and redrawing projection matrices can mitigate some limitations. The integration with local convolutions is designed to maintain strong locality without loss of global context [2009.14794][2511.02454].

---

**References**

- "Rethinking Attention with Performers" [2009.14794]
- "Improving DF-Conformer Using Hydra For High-Fidelity Generative Speech Enhancement on Discrete Codec Token" [2511.02454]

Source: https://www.emergentmind.com/topics/favor-attention