---
title: 'Fourier-VLM: Efficient Vision–Language Compression'
url: https://www.emergentmind.com/topics/fourier-vlm
type: topic
---

# Fourier-VLM: Efficient Vision–Language Compression

Searching arXiv for Fourier-VLM and closely related papers to ground the entry.
Fourier-VLM is a vision–language modeling method that compresses visual representations in the frequency domain before they are injected into a backbone large language model. In the formulation introduced in “Fourier-VLM: Compressing Vision Tokens in the Frequency Domain for Large Vision-Language Models” [2508.06038], the method exploits the observation that vision encoder features exhibit concentrated energy in low-frequency components, applies a low-pass filter with a two-dimensional Discrete Cosine Transform (DCT), and reconstructs a smaller set of visual tokens with no additional parameters. The resulting design targets the dominant systems bottleneck of contemporary VLMs: the large number of image tokens that replace the textual `<image>` placeholder and substantially increase context length, FLOPs, KV-cache usage, and prefilling latency [2508.06038].

## 1. Problem setting and architectural motivation

Modern VLMs such as LLaVA and Qwen-VL use a vision encoder to convert an image into a grid of patch embeddings, then project those embeddings into the token space of a backbone LLM. In the notation used for Fourier-VLM, an input image yields low-level grid features $\hat{\boldsymbol{H}}^{v} \in \mathbb{R}^{N \times N \times h_c}$ after a CNN stem and high-level visual representations $\boldsymbol{H}^{v} \in \mathbb{R}^{N^{2} \times h_v}$ after the vision transformer. These $N^2$ vision tokens replace the `<image>` placeholder in the prompt, so the LLM operates on a sequence that concatenates system text, visual tokens, and textual tokens [2508.06038].

The method is motivated by the quadratic dependence of Transformer self-attention on sequence length. Large visual token counts increase attention cost, enlarge the KV cache, and slow Time to First Token. The problem intensifies for high-resolution images, multi-image prompts, and video inputs, where visual tokens can dominate the context. Previous mitigation strategies—token pruning, token merging, learnable queries, and prefusion—reduce token count but introduce extra overhead, task-specific heuristics, or additional trainable modules, and may degrade performance under high compression [2508.06038].

Fourier-VLM addresses this bottleneck by compressing visual features after the vision encoder rather than altering the core VLM stack. A plausible implication is that the method preserves the inductive biases of the pretrained encoder and LLM while relocating the compression step to a deterministic transform layer. The paper’s stated design goal is a parameter-free, architecture-agnostic module that can be inserted between the vision encoder and the projector [2508.06038].

## 2. Frequency-domain formulation and the Frequency Feature Compressor

The core empirical premise is that vision encoder outputs, when reshaped into an $N \times N \times h_v$ grid and analyzed with a 2D DCT, concentrate most of their energy in low-frequency components near the top-left of the spectrum. The paper reports this behavior for real images across both CLIP ViT-L/336px in LLaVA-v1.5 and the re-engineered ViT used in Qwen-2-VL, while noting that a random RGB image does not show the same concentration [2508.06038].

Fourier-VLM operationalizes this observation through a Frequency Feature Compressor (FFC). Given visual features $\boldsymbol{H}^{v} \in \mathbb{R}^{N^{2} \times h_v}$, the method first reshapes them into a 2D spatial grid,
$$
\boldsymbol{G}^{v} = \text{Reshape}(\boldsymbol{H}^{v}), \quad \boldsymbol{G}^{v} \in \mathbb{R}^{N \times N \times h_v}.
$$
A 2D DCT is then applied over the spatial dimensions for each channel,
$$
\hat{\boldsymbol{F}}^{v}_{m,n,:} = \alpha_m \alpha_n \sum_{p=0}^{N-1} \sum_{q=0}^{N-1} \boldsymbol{G}^{v}_{p,q,:} \cdot \phi_N(m,p) \cdot \phi_N(n,q),
$$
where $\hat{\boldsymbol{F}}^{v} \in \mathbb{R}^{N \times N \times h_v}$, $\phi_N(x,y)=\cos\left[\frac{\pi}{N} x\left(y + \frac{1}{2}\right)\right]$, and the normalization is
$$
\alpha_m =
\begin{cases}
\sqrt{\frac{1}{N}}, & m = 0, \\
\sqrt{\frac{2}{N}}, & \text{otherwise.}
\end{cases}
$$
The retained spectrum is a simple top-left crop,
$$
\boldsymbol{F}^{v} = \hat{\boldsymbol{F}}^{v}[0\!:\!C,\, 0\!:\!C, :], \quad \boldsymbol{F}^{v} \in \mathbb{R}^{C \times C \times h_v}, \ C < N.
$$
This low-pass filter discards all higher-frequency coefficients and keeps only $C^2$ frequency tokens [2508.06038].

A 2D inverse DCT reconstructs a coarser spatial grid,
$$
\boldsymbol{T}^{v}_{i,j,:} = \sum_{p=0}^{C-1}\sum_{q=0}^{C-1} \alpha_p \alpha_q \cdot \boldsymbol{F}^{v}_{p,q,:} \cdot \phi_C(p,i) \cdot \phi_C(q,j),
$$
with $\boldsymbol{T}^{v} \in \mathbb{R}^{C \times C \times h_v}$. Flattening yields compressed visual tokens,
$$
\boldsymbol{H}^{v}_c = \text{Flatten}(\boldsymbol{T}^{v}), \quad \boldsymbol{H}^{v}_c \in \mathbb{R}^{C^{2} \times h_v}.
$$
These compressed tokens replace the original $N^2$ tokens in the VLM pipeline [2508.06038].

The DCT itself is implemented via FFT. For a sequence $\langle x_i \rangle = \{x_0, x_1, \dots, x_{N-1}\}$, the DCT-II is written as
$$
f_m = \alpha_m \sum_{i=0}^{N-1} x_i \cdot \phi_N(m, i), \quad m \in \{0,\dots,N-1\},
$$
with inverse
$$
x_i = \sum_{k=0}^{N-1} \alpha_k \, f_k \, \phi_N(k, i).
$$
The paper further rewrites DCT computation through a rearranged sequence and an FFT, yielding $\mathcal{O}(N \log N)$ complexity in 1D and $\mathcal{O}(N^2 \log N)$ in 2D [2508.06038].

This design is deliberately minimal. There is no learned mask, no new attention block, and no parameterized merger inside FFC. This suggests that the compression bias is entirely spectral: semantic redundancy is assumed to be concentrated in low-frequency vision features, and compression is imposed by deterministic truncation rather than token importance estimation.

## 3. Integration into LLaVA and Qwen-VL

In Fourier-LLaVA, the FFC is inserted after the CLIP vision encoder and before the standard 2-layer MLP projector. The original 576 visual tokens produced by CLIP ViT-L/336px in LLaVA-v1.5 are replaced by compressed token sets of size $C^2 \in \{256, 144, 64, 36\}$ [2508.06038]. The resulting compression ratios are stated explicitly: 256 tokens correspond to 55.6% compression, 144 tokens to 75.0%, 64 tokens to 88.9%, and 36 tokens to 93.75% [2508.06038].

Training follows the original two-stage LLaVA pipeline with FFC always active. In stage 1, feature alignment is performed on 558k image–caption pairs from a LAION-CC-SBU subset for 1 epoch, with the vision encoder and LLM frozen and only the 2-layer MLP projector trainable. In stage 2, visual instruction tuning is performed on 665k multimodal instruction-tuning samples for 2 epochs, with the vision encoder frozen and the projector plus LLM trainable through LoRA with rank $r = 128$ and scale $\alpha = 256$ [2508.06038].

In Fourier-Qwen, the FFC is inserted after Qwen-VL’s original MLP-based vision-language merger, which already compresses visual tokens by approximately 75%. The Qwen models operate at arbitrary input resolutions, so the number of visual tokens is variable. During training, the paper sets the visual token count between 256 and 2304 and performs continued fine-tuning on 600k single-image conversation samples from LLaVA-NeXT. The projector and LLM are trainable, and the vision encoder uses a small learning rate; FFC itself remains fixed [2508.06038].

The paper emphasizes that these integrations do not alter the architecture of the vision encoder, projector, or LLM, apart from changing the number of visual tokens consumed downstream. A plausible implication is that the method is primarily a systems intervention rather than a representational redesign: it compresses the sequence presented to the LLM while reusing the established multimodal alignment machinery.

## 4. Computational characteristics and efficiency profile

The computational argument for Fourier-VLM is stated in terms of asymptotic and measured cost. For visual features of shape $(B, N^2, h_v)$, the paper lists the cost of an MLP as $\mathcal{O}(B \cdot h_v^2 \cdot N^2)$, self-attention over $N^2$ tokens as $\mathcal{O}(B \cdot h_v \cdot N^4)$, and a query transformer with $M$ learnable queries as $\mathcal{O}(B \cdot h_v \cdot N^2 \cdot M)$. By contrast, FFC requires
$$
\mathcal{O}(B \cdot h_v \cdot N^2 \log N + B \cdot h_v \cdot C^2 \log C)
\approx \mathcal{O}(B \cdot h_v \cdot N^2 \log N),
$$
which the paper describes as substantially cheaper than attention-based or query-based compression under the typical regime $h_v \gg M > N$ [2508.06038].

Measured efficiency gains are central to the method’s significance. Using calflops on an RTX 4090, baseline LLaVA-v1.5-7B at 576 tokens requires 8.54 T FLOPs per inference. Fourier-LLaVA reduces this to 4.30 T at 256 tokens, 2.81 T at 144 tokens, 1.75 T at 64 tokens, and 1.38 T at 36 tokens, corresponding to reductions of 49.6%, 67.1%, 79.5%, and 83.8%, respectively [2508.06038].

On an A100 40G GPU, Fourier-LLaVA reduces KV-cache usage by up to 86.4% relative to the LLaVA-v1.5-7B baseline and improves Time to First Token by up to 31.2% [2508.06038]. The paper also states that Fourier-LLaVA outperforms MQT-LLaVA in TTFT at equal visual token counts, which it attributes to the lighter cost of DCT/iDCT relative to a query transformer [2508.06038].

These results position Fourier-VLM as a compression method whose overhead is not merely smaller than the savings it enables, but sufficiently small to preserve the practical advantage even when inserted into already optimized VLM pipelines. This suggests that the method is especially relevant for deployments constrained by prefilling latency, memory budgets, or multi-image scaling.

## 5. Empirical behavior on image and video benchmarks

The image-based evaluation follows eight benchmarks: VQA-v2, GQA, ScienceQA, TextVQA, POPE, MMBench, LLaVA Wild, and MMMU. On LLaVA-v1.5-7B, the baseline model with 576 tokens obtains an average score of 64.6. Fourier-LLaVA with 256 tokens also achieves 64.6, and Fourier-LLaVA with 144 tokens again achieves 64.6, matching the baseline average while using 44% and 25% of the original visual tokens, respectively [2508.06038].

At more aggressive compression, the degradation remains limited. Fourier-LLaVA with 64 tokens reaches an average of 63.3, and with 36 tokens reaches 62.1. The paper characterizes the 36-token setting as maintaining more than 96% of baseline average performance while using only 6.25% of the original visual tokens [2508.06038]. It further reports that Fourier-LLaVA with 64 or 36 tokens still outperforms MQT-LLaVA and ATP-LLaVA at similar token budgets [2508.06038].

The larger LLaVA-v1.5-13B backbone exhibits a similar pattern. The baseline 13B model at 576 tokens records an average of 67.3, while Fourier-LLaVA at 144 tokens records 65.9, equal to the reported PruMerge result at the same token count [2508.06038]. The paper states that Fourier-VLM therefore scales favorably to larger backbones.

On Qwen-VL derivatives, the method again compresses visual tokens substantially. Qwen-2-VL-2B has a baseline of approximately 553 visual tokens per image and average performance 66.8; Fourier-Qwen-2 reduces the average visual token count to approximately 236 and attains 65.7. Qwen-2.5-VL-3B has a baseline average of 71.1, while Fourier-Qwen-2.5 at approximately 236 visual tokens attains 72.6 [2508.06038]. The paper presents this as evidence that the compression generalizes across architectures and can even improve performance on some tasks.

The method is also evaluated on zero-shot video understanding with MVBench. In this setting, each video frame contributes visual tokens, so token count multiplies by the number of frames. Baseline LLaVA-v1.5 uses 2304 visual tokens per video, while Fourier-LLaVA uses 288 and reduces average performance from 45.6 to 44.0. Baseline Qwen-2-VL and Qwen-2.5-VL use 3193 visual tokens per video; their Fourier variants use 1328, with averages changing from 61.4 to 59.8 and from 65.2 to 62.9, respectively [2508.06038]. The paper highlights that these are strong results given that the models were trained only on image data.

Ablation behavior is closely tied to task type. The paper notes that reasoning-heavy benchmarks such as ScienceQA and MMMU are often preserved or improved under compression, whereas OCR-heavy tasks such as TextVQA degrade more noticeably at extreme compression [2508.06038]. This suggests that low-frequency truncation preserves coarse semantics and structure more robustly than fine-grained textual or high-frequency detail.

## 6. Interpretation, limitations, and terminological scope

The primary limitation emphasized by the empirical results is task dependence. High-frequency-dependent tasks, particularly OCR-heavy benchmarks, are more sensitive to aggressive compression. The paper also reports that inserting FFC into a pretrained VLM without fine-tuning leads to distortions and hallucinations, illustrated qualitatively by progressive misdescription of a heart-shaped sign under stronger compression; proper fine-tuning mitigates these distortions [2508.06038]. Another explicit limitation is the use of a fixed low-pass mask: the retained frequency region is always the simple crop $[0\!:\!C,0\!:\!C]$, with no image- or task-adaptive selection [2508.06038].

The broader literature contains several other “Fourier” formulations in multimodal or adjacent settings that should not be conflated with Fourier-VLM as defined above. “Fourier-Attentive Representation Learning” introduces a CLIP-style adaptation framework that decomposes images into phase-based structural and amplitude-based stylistic cues and injects disentangled tokens asymmetrically into the text and image encoders for few-shot generalization [2512.04395]. Although both methods operate in the frequency domain, FARL is a representation-learning and adaptation strategy rather than a token-compression module.

Outside vision–language modeling, the name can overlap with unrelated Fourier-based frameworks. “Variational Matrix-Learning Fourier Networks for Parametric Multiphysics Surrogates” presents a variational matrix-learning Fourier network for PDE-governed surrogates in system-technology co-optimization [2605.02280], and “Variable Elimination in the Fourier Domain” studies Fourier representations for approximate inference in Boolean graphical models [1508.04032]. These works share Fourier structure but neither addresses the compression of visual tokens in large VLMs.

Within the VLM literature, Fourier-VLM in the strict sense therefore denotes the parameter-free FFC framework introduced for LLaVA and Qwen-VL. Its distinguishing features are the use of 2D DCT on vision encoder outputs, deterministic low-pass truncation, inverse reconstruction into a smaller token grid, and efficient integration into existing VLM stacks with substantial reductions in FLOPs, KV-cache usage, and prefilling latency [2508.06038].

Source: https://www.emergentmind.com/topics/fourier-vlm