Fourier-VLM: Efficient Vision–Language Compression

• Fourier-VLM is a frequency-domain vision–language technique that compresses visual tokens via a low-pass 2D DCT, reducing computational cost.
• It compresses high-dimensional visual features into fewer tokens, thereby lowering FLOPs, KV-cache usage, and latency without extra parameters.
• Integration into models like LLaVA and Qwen-VL demonstrates minimal performance drop even at aggressive compression levels on image and video benchmarks.

Searching arXiv for Fourier-VLM and closely related papers to ground the entry. Fourier-VLM is a vision–language modeling method that compresses visual representations in the frequency domain before they are injected into a backbone LLM. In the formulation introduced in “Fourier-VLM: Compressing Vision Tokens in the Frequency Domain for Large Vision-LLMs” (Wang et al., 8 Aug 2025), the method exploits the observation that vision encoder features exhibit concentrated energy in low-frequency components, applies a low-pass filter with a two-dimensional Discrete Cosine Transform (DCT), and reconstructs a smaller set of visual tokens with no additional parameters. The resulting design targets the dominant systems bottleneck of contemporary VLMs: the large number of image tokens that replace the textual <image> placeholder and substantially increase context length, FLOPs, KV-cache usage, and prefilling latency (Wang et al., 8 Aug 2025).

1. Problem setting and architectural motivation

Modern VLMs such as LLaVA and Qwen-VL use a vision encoder to convert an image into a grid of patch embeddings, then project those embeddings into the token space of a backbone LLM. In the notation used for Fourier-VLM, an input image yields low-level grid features $$\hat{\boldsymbol{H}}^{v} \in \mathbb{R}^{N \times N \times h_c}$$ after a CNN stem and high-level visual representations $$\boldsymbol{H}^{v} \in \mathbb{R}^{N^{2} \times h_v}$$ after the vision transformer. These $N^2$ vision tokens replace the <image> placeholder in the prompt, so the LLM operates on a sequence that concatenates system text, visual tokens, and textual tokens (Wang et al., 8 Aug 2025).

The method is motivated by the quadratic dependence of Transformer self-attention on sequence length. Large visual token counts increase attention cost, enlarge the KV cache, and slow Time to First Token. The problem intensifies for high-resolution images, multi-image prompts, and video inputs, where visual tokens can dominate the context. Previous mitigation strategies—token pruning, token merging, learnable queries, and prefusion—reduce token count but introduce extra overhead, task-specific heuristics, or additional trainable modules, and may degrade performance under high compression (Wang et al., 8 Aug 2025).

Fourier-VLM addresses this bottleneck by compressing visual features after the vision encoder rather than altering the core VLM stack. A plausible implication is that the method preserves the inductive biases of the pretrained encoder and LLM while relocating the compression step to a deterministic transform layer. The paper’s stated design goal is a parameter-free, architecture-agnostic module that can be inserted between the vision encoder and the projector (Wang et al., 8 Aug 2025).

2. Frequency-domain formulation and the Frequency Feature Compressor

The core empirical premise is that vision encoder outputs, when reshaped into an $N \times N \times h_v$ grid and analyzed with a 2D DCT, concentrate most of their energy in low-frequency components near the top-left of the spectrum. The paper reports this behavior for real images across both CLIP ViT-L/336px in LLaVA-v1.5 and the re-engineered ViT used in Qwen-2-VL, while noting that a random RGB image does not show the same concentration (Wang et al., 8 Aug 2025).

Fourier-VLM operationalizes this observation through a Frequency Feature Compressor (FFC). Given visual features $$\boldsymbol{H}^{v} \in \mathbb{R}^{N^{2} \times h_v}$$, the method first reshapes them into a 2D spatial grid,

$$\boldsymbol{G}^{v} = \text{Reshape}(\boldsymbol{H}^{v}), \quad \boldsymbol{G}^{v} \in \mathbb{R}^{N \times N \times h_v}.$$

A 2D DCT is then applied over the spatial dimensions for each channel,

$$\hat{\boldsymbol{F}}^{v}_{m,n,:} = \alpha_m \alpha_n \sum_{p=0}^{N-1} \sum_{q=0}^{N-1} \boldsymbol{G}^{v}_{p,q,:} \cdot \phi_N(m,p) \cdot \phi_N(n,q),$$

where $$\hat{\boldsymbol{F}}^{v} \in \mathbb{R}^{N \times N \times h_v}$$, $$\phi_N(x,y)=\cos\left[\frac{\pi}{N} x\left(y + \frac{1}{2}\right)\right]$$, and the normalization is

$$\alpha_m = \begin{cases} \sqrt{\frac{1}{N}}, & m = 0, \ \sqrt{\frac{2}{N}}, & \text{otherwise.} \end{cases}$$

The retained spectrum is a simple top-left crop,

$$\boldsymbol{H}^{v} \in \mathbb{R}^{N^{2} \times h_v}$$0

This low-pass filter discards all higher-frequency coefficients and keeps only $$\boldsymbol{H}^{v} \in \mathbb{R}^{N^{2} \times h_v}$$1 frequency tokens (Wang et al., 8 Aug 2025).

A 2D inverse DCT reconstructs a coarser spatial grid,

$$\boldsymbol{H}^{v} \in \mathbb{R}^{N^{2} \times h_v}$$2

with $$\boldsymbol{H}^{v} \in \mathbb{R}^{N^{2} \times h_v}$$3. Flattening yields compressed visual tokens,

$$\boldsymbol{H}^{v} \in \mathbb{R}^{N^{2} \times h_v}$$4

These compressed tokens replace the original $$\boldsymbol{H}^{v} \in \mathbb{R}^{N^{2} \times h_v}$$5 tokens in the VLM pipeline (Wang et al., 8 Aug 2025).

The DCT itself is implemented via FFT. For a sequence $$\boldsymbol{H}^{v} \in \mathbb{R}^{N^{2} \times h_v}$$6, the DCT-II is written as

$$\boldsymbol{H}^{v} \in \mathbb{R}^{N^{2} \times h_v}$$7

with inverse

$$\boldsymbol{H}^{v} \in \mathbb{R}^{N^{2} \times h_v}$$8

The paper further rewrites DCT computation through a rearranged sequence and an FFT, yielding $$\boldsymbol{H}^{v} \in \mathbb{R}^{N^{2} \times h_v}$$9 complexity in 1D and $N^2$0 in 2D (Wang et al., 8 Aug 2025).

This design is deliberately minimal. There is no learned mask, no new attention block, and no parameterized merger inside FFC. This suggests that the compression bias is entirely spectral: semantic redundancy is assumed to be concentrated in low-frequency vision features, and compression is imposed by deterministic truncation rather than token importance estimation.

3. Integration into LLaVA and Qwen-VL

In Fourier-LLaVA, the FFC is inserted after the CLIP vision encoder and before the standard 2-layer MLP projector. The original 576 visual tokens produced by CLIP ViT-L/336px in LLaVA-v1.5 are replaced by compressed token sets of size $N^2$1 (Wang et al., 8 Aug 2025). The resulting compression ratios are stated explicitly: 256 tokens correspond to 55.6% compression, 144 tokens to 75.0%, 64 tokens to 88.9%, and 36 tokens to 93.75% (Wang et al., 8 Aug 2025).

Training follows the original two-stage LLaVA pipeline with FFC always active. In stage 1, feature alignment is performed on 558k image–caption pairs from a LAION-CC-SBU subset for 1 epoch, with the vision encoder and LLM frozen and only the 2-layer MLP projector trainable. In stage 2, visual instruction tuning is performed on 665k multimodal instruction-tuning samples for 2 epochs, with the vision encoder frozen and the projector plus LLM trainable through LoRA with rank $N^2$2 and scale $N^2$3 (Wang et al., 8 Aug 2025).

In Fourier-Qwen, the FFC is inserted after Qwen-VL’s original MLP-based vision-language merger, which already compresses visual tokens by approximately 75%. The Qwen models operate at arbitrary input resolutions, so the number of visual tokens is variable. During training, the paper sets the visual token count between 256 and 2304 and performs continued fine-tuning on 600k single-image conversation samples from LLaVA-NeXT. The projector and LLM are trainable, and the vision encoder uses a small learning rate; FFC itself remains fixed (Wang et al., 8 Aug 2025).

The paper emphasizes that these integrations do not alter the architecture of the vision encoder, projector, or LLM, apart from changing the number of visual tokens consumed downstream. A plausible implication is that the method is primarily a systems intervention rather than a representational redesign: it compresses the sequence presented to the LLM while reusing the established multimodal alignment machinery.

4. Computational characteristics and efficiency profile

The computational argument for Fourier-VLM is stated in terms of asymptotic and measured cost. For visual features of shape $N^2$4, the paper lists the cost of an MLP as $N^2$5, self-attention over $N^2$6 tokens as $N^2$7, and a query transformer with $N^2$8 learnable queries as $N^2$9. By contrast, FFC requires

$N \times N \times h_v$0

which the paper describes as substantially cheaper than attention-based or query-based compression under the typical regime $N \times N \times h_v$1 (Wang et al., 8 Aug 2025).

Measured efficiency gains are central to the method’s significance. Using calflops on an RTX 4090, baseline LLaVA-v1.5-7B at 576 tokens requires 8.54 T FLOPs per inference. Fourier-LLaVA reduces this to 4.30 T at 256 tokens, 2.81 T at 144 tokens, 1.75 T at 64 tokens, and 1.38 T at 36 tokens, corresponding to reductions of 49.6%, 67.1%, 79.5%, and 83.8%, respectively (Wang et al., 8 Aug 2025).

On an A100 40G GPU, Fourier-LLaVA reduces KV-cache usage by up to 86.4% relative to the LLaVA-v1.5-7B baseline and improves Time to First Token by up to 31.2% (Wang et al., 8 Aug 2025). The paper also states that Fourier-LLaVA outperforms MQT-LLaVA in TTFT at equal visual token counts, which it attributes to the lighter cost of DCT/iDCT relative to a query transformer (Wang et al., 8 Aug 2025).

These results position Fourier-VLM as a compression method whose overhead is not merely smaller than the savings it enables, but sufficiently small to preserve the practical advantage even when inserted into already optimized VLM pipelines. This suggests that the method is especially relevant for deployments constrained by prefilling latency, memory budgets, or multi-image scaling.

5. Empirical behavior on image and video benchmarks

The image-based evaluation follows eight benchmarks: VQA-v2, GQA, ScienceQA, TextVQA, POPE, MMBench, LLaVA Wild, and MMMU. On LLaVA-v1.5-7B, the baseline model with 576 tokens obtains an average score of 64.6. Fourier-LLaVA with 256 tokens also achieves 64.6, and Fourier-LLaVA with 144 tokens again achieves 64.6, matching the baseline average while using 44% and 25% of the original visual tokens, respectively (Wang et al., 8 Aug 2025).

At more aggressive compression, the degradation remains limited. Fourier-LLaVA with 64 tokens reaches an average of 63.3, and with 36 tokens reaches 62.1. The paper characterizes the 36-token setting as maintaining more than 96% of baseline average performance while using only 6.25% of the original visual tokens (Wang et al., 8 Aug 2025). It further reports that Fourier-LLaVA with 64 or 36 tokens still outperforms MQT-LLaVA and ATP-LLaVA at similar token budgets (Wang et al., 8 Aug 2025).

The larger LLaVA-v1.5-13B backbone exhibits a similar pattern. The baseline 13B model at 576 tokens records an average of 67.3, while Fourier-LLaVA at 144 tokens records 65.9, equal to the reported PruMerge result at the same token count (Wang et al., 8 Aug 2025). The paper states that Fourier-VLM therefore scales favorably to larger backbones.

On Qwen-VL derivatives, the method again compresses visual tokens substantially. Qwen-2-VL-2B has a baseline of approximately 553 visual tokens per image and average performance 66.8; Fourier-Qwen-2 reduces the average visual token count to approximately 236 and attains 65.7. Qwen-2.5-VL-3B has a baseline average of 71.1, while Fourier-Qwen-2.5 at approximately 236 visual tokens attains 72.6 (Wang et al., 8 Aug 2025). The paper presents this as evidence that the compression generalizes across architectures and can even improve performance on some tasks.

The method is also evaluated on zero-shot video understanding with MVBench. In this setting, each video frame contributes visual tokens, so token count multiplies by the number of frames. Baseline LLaVA-v1.5 uses 2304 visual tokens per video, while Fourier-LLaVA uses 288 and reduces average performance from 45.6 to 44.0. Baseline Qwen-2-VL and Qwen-2.5-VL use 3193 visual tokens per video; their Fourier variants use 1328, with averages changing from 61.4 to 59.8 and from 65.2 to 62.9, respectively (Wang et al., 8 Aug 2025). The paper highlights that these are strong results given that the models were trained only on image data.

Ablation behavior is closely tied to task type. The paper notes that reasoning-heavy benchmarks such as ScienceQA and MMMU are often preserved or improved under compression, whereas OCR-heavy tasks such as TextVQA degrade more noticeably at extreme compression (Wang et al., 8 Aug 2025). This suggests that low-frequency truncation preserves coarse semantics and structure more robustly than fine-grained textual or high-frequency detail.

6. Interpretation, limitations, and terminological scope

The primary limitation emphasized by the empirical results is task dependence. High-frequency-dependent tasks, particularly OCR-heavy benchmarks, are more sensitive to aggressive compression. The paper also reports that inserting FFC into a pretrained VLM without fine-tuning leads to distortions and hallucinations, illustrated qualitatively by progressive misdescription of a heart-shaped sign under stronger compression; proper fine-tuning mitigates these distortions (Wang et al., 8 Aug 2025). Another explicit limitation is the use of a fixed low-pass mask: the retained frequency region is always the simple crop $N \times N \times h_v$2, with no image- or task-adaptive selection (Wang et al., 8 Aug 2025).

The broader literature contains several other “Fourier” formulations in multimodal or adjacent settings that should not be conflated with Fourier-VLM as defined above. “Fourier-Attentive Representation Learning” introduces a CLIP-style adaptation framework that decomposes images into phase-based structural and amplitude-based stylistic cues and injects disentangled tokens asymmetrically into the text and image encoders for few-shot generalization (Pham et al., 4 Dec 2025). Although both methods operate in the frequency domain, FARL is a representation-learning and adaptation strategy rather than a token-compression module.

Outside vision–language modeling, the name can overlap with unrelated Fourier-based frameworks. “Variational Matrix-Learning Fourier Networks for Parametric Multiphysics Surrogates” presents a variational matrix-learning Fourier network for PDE-governed surrogates in system-technology co-optimization (Li et al., 4 May 2026), and “Variable Elimination in the Fourier Domain” studies Fourier representations for approximate inference in Boolean graphical models (Xue et al., 2015). These works share Fourier structure but neither addresses the compression of visual tokens in large VLMs.

Within the VLM literature, Fourier-VLM in the strict sense therefore denotes the parameter-free FFC framework introduced for LLaVA and Qwen-VL. Its distinguishing features are the use of 2D DCT on vision encoder outputs, deterministic low-pass truncation, inverse reconstruction into a smaller token grid, and efficient integration into existing VLM stacks with substantial reductions in FLOPs, KV-cache usage, and prefilling latency (Wang et al., 8 Aug 2025).