---
title: Focused Linear Attention (FLatten)
url: https://www.emergentmind.com/topics/focused-linear-attention-flatten
type: topic
---

# Focused Linear Attention (FLatten)

Focused Linear Attention (FLatten) refers to a family of attention mechanisms designed to combine the computational efficiency of linear attention with the sharp focus and feature diversity characteristic of classical softmax-based self-attention. FLatten mechanisms have been applied in computer vision and sequence modeling, most notably in transformers for high-resolution vision tasks, local feature matching, and speech separation. The core innovation of FLatten is a focused nonlinear mapping that sharpens attention distributions and a rank restoration module based on depthwise convolution, resulting in linear time and memory complexity while mitigating the rank deficiency and smoothness typically observed in vanilla linear attention. This approach is exemplified by the FLatten Transformer [2308.00442], LoFLAT [2410.22710], and FLASepformer [2508.19528].

## 1. Mathematical Foundations

Standard self-attention with softmax is given by
$$
\operatorname{Att}_{\text{soft}}(Q, K, V) = \operatorname{softmax}\left(\frac{Q K^\top}{\sqrt{d}}\right)V,
$$
which incurs $\mathcal{O}(N^2 d)$ time and memory for $N$ tokens of dimension $d$. Linear attention approximates softmax using a positive mapping $\phi(\cdot)$:
$$
\operatorname{Att}_{L}(Q, K, V) = \phi(Q) [\phi(K)^\top V],
$$
with cost $\mathcal{O}(N d^2)$ for fixed $d$.

FLatten introduces a focused kernel $\phi_p$, defined as follows. For $x \in \mathbb{R}^d$:
$$
\phi_p(x) = f_p(\operatorname{ReLU}(x)), \quad f_p(x) = \frac{\|x\|_2}{\|x^{**p}\|_2} x^{**p},
$$
where $x^{**p}$ denotes elementwise exponentiation and $\operatorname{ReLU}(x)$ ensures nonnegativity. The resulting focused linear attention computes:
$$
O = \phi_p(Q) [\phi_p(K)^\top V] + \operatorname{DWC}(V),
$$
where $\operatorname{DWC}(V)$ is a depthwise convolution applied channel-wise to $V$ for rank restoration [2308.00442][2410.22710][2508.19528].

## 2. Focus Sharpening and Feature Diversity

Softmax-based attention yields highly concentrated maps allowing queries to selectively attend to informative keys. Vanilla linear kernels tend to produce diffuse, low-rank attention, making them suboptimal in high-resolution or long-sequence regimes. The $\phi_p$ mapping in FLatten increases the angular separation between query and key vectors as $p$ grows, enhancing focus on dominant features and suppressing irrelevant dimensions. Proposition 1 in [2308.00442] formalizes that $f_p$ increases cosine similarity for shared dominant indices, leading to sharper, more discriminative attention.

Linear attention's rank is bounded by $d$ ($\min\{\operatorname{rank}(\phi(Q)), \operatorname{rank}(\phi(K))\} \leq d \ll N$), causing the effective attention map to lack diversity. Injection of a small depthwise convolution (kernel $k \times k$ or 1D for sequential data) over $V$ is mathematically equivalent to adding a sparse, full-rank correction, thus restoring per-token expressiveness and breaking the degenerate rank constraint [2308.00442][2410.22710][2508.19528].

## 3. Computational Complexity and Efficiency

The computational and memory complexity of different attention schemes is summarized below:

| Attention Type         | Time Complexity      | Memory Complexity    | Expressiveness   |
|-----------------------|---------------------|---------------------|------------------|
| Softmax (MHSA)        | $\mathcal{O}(N^2d)$ | $\mathcal{O}(N^2)$  | High             |
| Linear (kernel)       | $\mathcal{O}(Nd^2)$ | $\mathcal{O}(Nd)$   | Low (Rank $<d$)  |
| Focused Linear (FLatten) | $\mathcal{O}(Nd^2 + Ndk^2)$ | $\mathcal{O}(Nd)$ | High (Full rank) |

Depthwise convolution's cost ($k^2$ for images, $k$ for sequences) is minimal relative to the quadratic component eliminated by linearization. Empirically, FLatten achieves up to $2\times$ speedup with either comparable or better accuracy on high-resolution tasks [2308.00442]. In speech separation, FLASepformer attains $1.91\times$ faster inference and only 20.9% GPU memory usage compared to SepReformer while closely matching SI-SNRi accuracy [2508.19528]. LoFLAT demonstrates similar efficiency with increased accuracy over LoFTR for local feature matching [2410.22710].

## 4. Architectural Integration and Variants

FLatten has been integrated into diverse transformer-based architectures:

- **Vision Transformers**: FLatten replaces the softmax attention in early ViT stages and is compatible with DeiT, PVT, Swin, and similar models [2308.00442]. The depthwise convolution operates spatially.
- **Local Feature Matching (LoFLAT)**: The Feature Transformer Module applies FLatten with a $3\times3$ 2D depthwise convolution per channel. Feature Extraction utilizes ResNet + FPN. A coarse-to-fine Matching Module exploits the improved map focus and diversity for robust matching [2410.22710].
- **Speech Separation (FLASepformer)**: FLASepformer [2508.19528] deploys FLatten (with 1D depthwise convolution) in the global attention modules of SepReformer and in the temporal blocks of TF-Locoformer, both replacing quadratic MHSA. A gated MLP module with LayerNorm and channel-wise gating further enhances per-token modulations. Hyperparameters such as focus exponent $p=3$ and kernel $k=7$ are used for robust performance.

## 5. Empirical Results and Comparisons

Extensive benchmarks validate FLatten's efficacy:

- **Image Classification** [2308.00442]: DeiT-Tiny accuracy improves from 72.2% (softmax) to 74.1% (FLatten); Swin-Tiny improves from 81.3% to 82.1%. Performance gains are consistent across segmentation (ADE20K mIoU) and object detection (COCO AP).
- **Speech Separation** [2508.19528]: FLA-SepReformer matches or nearly matches SepReformer SI-SNRi while enabling $2\times$ acceleration and linear memory scaling. FLA-TFLocoformer achieves similar SI-SNRi with $\sim20\%$ of the original GPU memory budget.
- **Local Feature Matching** [2410.22710]: On MegaDepth, LoFLAT increases AUC @5°, @10°, and @20° by 2.7%, 1.9%, and 0.9% respectively over LoFTR, yielding denser and more robust matches.

Ablation studies confirm that the focused mapping improves accuracy significantly over vanilla linear attention, and further gains accrue when the depthwise convolution rank-restoration module is added. The choice of exponent $p$ is robust; $p=3$ is typically optimal [2308.00442][2508.19528].

## 6. Implementation Guidelines and Limitations

Key implementation suggestions include:

- **Focus Exponent**: Default $p=3$; variation in $p$ has minor effects on accuracy unless set very high or low [2308.00442][2508.19528].
- **Depthwise Convolution**: Kernel size $k=3$ for images and $k=7$ for sequential data are effective; size variations yield diminishing returns [2410.22710][2508.19528].
- **Gating (for FLASepformer)**: LayerNorm prior to gating and a single or two-layer linear projection with nonlinearities are effective.
- **Integration Points**: Plugging FLatten into early, high-resolution transformer blocks yields maximum efficiency benefit [2308.00442].

Limitations include a need to tune the additional focus parameter $p$, potential ineffectiveness of depthwise convolution if local structure is weak, and the under-weighting of extremely long-range interactions—a generic limitation of all linear attention methods [2410.22710]. In speech separation, moderate SI-SNRi drops of 0.2–0.3 dB are observed versus full MHSA, but with large gains in scalability and speed [2508.19528].

## 7. Research Directions and Outlook

Potential future research includes:

- **Learnable Mapping Exponents**: Making $p$ learnable for task adaptation.
- **Multi-Kernel Rank Restoration**: Using richer or adaptive convolutional kernels to further increase feature diversity.
- **Applicability to Cross-Attention**: Extending FLatten to decoder or cross-attention mechanisms in advanced architectures.
- **Broader Modalities**: Due to linear scaling, practical for large-context, high-resolution, or long-sequence modeling across vision, audio, and multimodal domains [2308.00442][2410.22710][2508.19528].

The Focused Linear Attention paradigm offers a unifying, resource-efficient framework that matches or outperforms prior linear attention methods and, in several regimes, surpasses softmax self-attention in accuracy‐throughput trade-offs.

Source: https://www.emergentmind.com/topics/focused-linear-attention-flatten