---
title: Visual Expert Quantization (VEQ)
url: https://www.emergentmind.com/topics/visual-expert-quantization-veq
type: topic
---

# Visual Expert Quantization (VEQ)

Visual Expert Quantization (VEQ) is a post-training quantization framework for Mixture-of-Experts (MoE) vision-language models (VLMs) that treats quantization as a problem shaped by two distinct forms of heterogeneity: the discrepancy between visual and language tokens, and the unequal contribution of different experts. Introduced as a dual-aware quantization method for low-bit weight-only compression, VEQ modifies both expert-level reconstruction objectives and token-level second-order calibration so that pivotal experts and modality-sensitive token directions receive greater protection under quantization. Its two principal components are **Modality-expert-aware Quantization (VEQ-ME)** and **Modality-affinity-aware Quantization (VEQ-MA)**, implemented respectively as AWQ-style and GPTQ-style extensions for MoE VLMs [2602.01037].

## 1. Problem formulation and motivating heterogeneity

VEQ is defined for **post-training quantization (PTQ)** of **MoE vision-language models**, with emphasis on **weight-only quantization** such as **W3A16** and **W4A16**. The motivating models are MoE VLMs including **Kimi-VL-Instruct** and **Qwen3-VL-30B-A3B-Instruct**, where quantization is attractive because full models remain expensive in memory and inference cost even though MoE activates only a subset of experts per token [2602.01037].

The framework begins from the claim that conventional PTQ methods such as **RTN**, **AWQ**, and **GPTQ** are inadequate for MoE VLMs because they assume a comparatively homogeneous calibration regime. VEQ argues that this assumption fails in two ways. First, **vision and text tokens are not statistically equivalent**. Vision tokens are numerous and spatially redundant, whereas text tokens are fewer, semantically dense, and more influential on final outputs. Using the supervised fine-tuning loss gradient as a sensitivity proxy, the reported average gradient magnitude of text tokens exceeds that of vision tokens by a factor of **22.4** on **128 COCO samples**, and the text-to-vision gradient ratio is around **15** in a representative sample. Second, **experts are not equally important**. MoE routing is sparse and imbalanced: some experts are hot, some are nearly dormant, some are modality-specialized, and only a few experts typically receive large routing probabilities for a given token [2602.01037].

These observations define VEQ’s central problem. A calibration set with many image tokens can dominate unweighted statistics even when text-token perturbations are more harmful, and a uniform expert-wise objective can allocate equal effort to hot and dormant experts even though their downstream effect is highly unequal. VEQ therefore treats successful compression of MoE VLMs as requiring joint modeling of **modality heterogeneity** and **expert heterogeneity**, rather than treating quantization as a layer-uniform approximation problem [2602.01037].

## 2. Modality-expert-aware quantization

VEQ-ME addresses the macro-level question of **which experts deserve more protection**. It extends an AWQ-like calibration objective by weighting each expert according to a modality-balanced importance score derived from routed token counts and gradient sensitivity [2602.01037].

The construction uses four calibration statistics for expert \(i\): the number of routed text tokens \(N_i^{\text{text}}\), the number of routed vision tokens \(N_i^{\text{vis}}\), the total text-token count \(T_{\text{text}}\), and the total vision-token count \(T_{\text{vis}}\). It also uses the modality sensitivity ratio
\[
\gamma = \frac{\|\nabla_{\text{text}}\|}{\|\nabla_{\text{vis}}\|},
\]
together with the quantity normalization factor
\[
\beta = \frac{T_{\text{text}}}{T_{\text{vis}}}.
\]
The resulting expert importance score is
\[
S_i = \gamma \cdot N_i^{\text{text}} + \beta \cdot N_i^{\text{vis}}.
\]

This score explicitly amplifies text-token traffic through \(\gamma\) and rescales visual-token traffic through \(\beta\). The standard expert-wise reconstruction objective
\[
\mathcal{L}_{\text{Standard}} = \sum_{i=1}^{M} \left\| \mathbf{W}_i \mathbf{X}_i - \hat{\mathbf{W}}_i \mathbf{X}_i \right\|_F^2
\]
is then replaced by the weighted objective
\[
\mathcal{L}_{\text{Weighted}} = \sum_{i=1}^{M} S_i \cdot \left\| \mathbf{W}_i \mathbf{X}_i - \hat{\mathbf{W}}_i \mathbf{X}_i \right\|_F^2,
\]
where \(M\) is the number of experts, \(\mathbf{W}_i\) and \(\hat{\mathbf{W}}_i\) are the full-precision and quantized weights of expert \(i\), and \(\mathbf{X}_i\) is the routed input to that expert [2602.01037].

The significance of VEQ-ME is not that it minimizes uniform numerical error more aggressively, but that it minimizes **task-relevant** error more selectively. Frequently used experts, and especially experts heavily used by text tokens, receive larger weights during scale search and calibration. A common misconception is that MoE PTQ can be made multimodal simply by balancing image and text tokens globally. VEQ-ME rejects that view: modality balancing is carried out at the **expert level**, where routing imbalance and expert specialization actually appear [2602.01037].

## 3. Modality-affinity-aware quantization

VEQ-MA addresses the micro-level question of **which token directions should dominate second-order calibration within an expert**. It extends GPTQ by replacing the usual uniform Hessian surrogate with an affinity- and modality-weighted Hessian that emphasizes tokens strongly associated with the current expert and disproportionately weights text-token directions [2602.01037].

In standard GPTQ notation, the Hessian approximation is
\[
H = 2XX^\top,
\]
where \(X\) is the calibration matrix. VEQ-MA argues that this form is inappropriate for MoE VLMs because it assigns equal influence to all routed tokens even though some are weakly associated with the expert and some arise from the less sensitive visual modality. For token \(j\), let \(p_j\) be its affinity to the target expert. VEQ-MA defines
\[
c_j = p_j \cdot \alpha_j, \qquad
\alpha_j =
\begin{cases}
\gamma, & x_j \text{ is text token},\\
1, & x_j \text{ is vision token}.
\end{cases}
\]
With diagonal matrix \(\mathbf{C}\) whose entries are \(c_j\), the enhanced Hessian is
\[
\tilde{H} = (X \cdot \sqrt{\mathbf{C}})(X \cdot \sqrt{\mathbf{C}})^\top = X \mathbf{C} X^\top.
\]

This replacement changes GPTQ’s inverse-Hessian-based updates so that the quantizer protects directions that are simultaneously **expert-relevant** and **modality-sensitive**. High-affinity tokens contribute more because they better represent the expert’s actual operating region; text tokens contribute more because their gradients are larger and their semantic density is higher. In this way VEQ-MA aligns second-order calibration with MoE routing structure rather than with undifferentiated token covariance [2602.01037].

The method also appears in a sensitivity study through the interpolation
\[
(1-\lambda)\cdot \mathbf{1} + \lambda \cdot \mathbf{p},
\]
where \(\lambda=1\) recovers raw router confidence and \(\lambda=0\) removes affinity weighting. The reported best stability occurs at \(\lambda=1\) and \(\gamma=\gamma_0\), which the paper interprets as evidence that router affinity is genuinely informative rather than incidental [2602.01037].

## 4. Empirical performance and ablation evidence

VEQ is evaluated on **Kimi-VL-Instruct** and **Qwen3-VL-30B-A3B-Instruct** over **MMMU**, **AI2D**, **InfoVQA**, **TextVQA**, **RealWorldQA**, **ScienceQA**, **VizWiz-VQA**, **MMBench**, and **MME-RealWorld**, using the **lmms-eval** framework and **SGLang** as the inference backend. The emphasis is on **weight-only PTQ** under **W3** and **W4**, with the abstract highlighting **W3A16** [2602.01037].

The headline result is that VEQ sets a new state of the art at **W3A16**. On **Kimi-VL**, average accuracies are **60.74** for RTN, **63.37** for AWQ, **62.93** for MBQ, **62.33** for GPTQ, **64.00** for VEQ-ME, and **65.41** for VEQ-MA, yielding a **+2.04** average gain over the best baseline. On **Qwen3-VL**, the corresponding averages are **63.17**, **62.22**, **60.97**, **64.16**, **63.93**, and **67.14**; the abstract reports a **3.09%** average gain over prior state of the art, while the tabulated W3 averages place VEQ-MA roughly three points above GPTQ. At **W4**, gains are smaller but remain consistent: **73.57** versus **72.96** on Kimi-VL, and **74.36** versus **73.20** on Qwen3-VL [2602.01037].

| Setting | Kimi-VL average | Qwen3-VL average |
|---|---:|---:|
| W3 best baseline | 63.37 | 64.16 |
| W3 VEQ-MA | 65.41 | 67.14 |
| W4 best baseline | 72.96 | 73.20 |
| W4 VEQ-MA | 73.57 | 74.36 |

Task-level behavior is consistent with the method design. The paper specifically highlights **TextVQA** under **Kimi-VL W3**, where **GPTQ** scores **64.30** and **VEQ-MA** reaches **78.30**. More generally, VEQ is reported as especially robust on reasoning-intensive and OCR-heavy tasks such as **MMMU**, **InfoVQA**, and **TextVQA**, where preserving text-sensitive directions and decisive experts matters most [2602.01037].

The ablation studies separate the two modules cleanly. For VEQ-ME, removing the gradient factor by setting \(\gamma=1\) reduces the average to **62.02**, and removing the quantity factor by setting \(\beta=1\) gives **62.30**, compared with **63.27** for full VEQ-ME. For VEQ-MA, removing affinity by setting \(p=1\) gives **63.63**, removing modality weighting by setting \(\alpha=1\) gives **62.46**, and full VEQ-MA gives **63.96**. These results indicate that both routing confidence and modality-aware weighting matter, with modality-aware weighting contributing more strongly in the reported averages [2602.01037].

## 5. Relation to adjacent quantization research

VEQ belongs to a broader shift in visual and multimodal quantization research in which quantization is treated as **structure-aware** rather than architecture-agnostic. Within that landscape, its closest neighboring formulation is **Quant Experts (QE)**, which also targets VLM PTQ through a mixture-of-experts design. QE divides important channels into token-independent and token-dependent groups, uses a **shared expert** for global quantization error compensation, and introduces **routed experts** implemented as multiple routed low-rank adapters for token-specific local error. It is evaluated on **Qwen2VL** and **InternVL2** under **W4A6**, **W4A8**, and **W3A16**, and reports consistent gains from **2B** to **72B** models [2602.24059]. VEQ and QE are closely related in motivation but differ in mechanism: VEQ modifies calibration objectives and Hessian structure on expert weights, whereas QE reconstructs residual quantization error through expert-conditioned low-rank adapters. This suggests two complementary interpretations of “expert-aware” quantization in VLMs: one based on **importance-weighted calibration**, the other on **expertized residual compensation**.

A second neighboring line concerns hard visual discretization and routing rather than MoE PTQ itself. **DiVeQ** and **SF-DiVeQ** treat vector quantization as a differentiable reparameterization problem, keeping hard nearest-neighbor assignment in the forward pass while improving gradient flow and, in the space-filling variant, improving codebook utilization through assignment to line segments between codewords [2509.26469]. Although not framed as MoE VLM compression, this work is directly relevant to VEQ-like systems whenever hard expert or codeword assignment blocks end-to-end training.

A third line addresses large-scale or semantically aligned visual tokenization rather than expert-aware PTQ. **AlignedVQ** quantizes normalized intermediate visual token features in a partitioned VLM, using dual linear projections around the quantizer and achieving approximately **1365x** compression of transmitted features with small VQA accuracy change across eight datasets [2411.05961]. **IBQ** addresses codebook collapse in large learned visual vocabularies by applying a straight-through estimator to the one-hot categorical index, enabling codebooks as large as \(2^{18}\) with high utilization [2412.02692]. **ViQ** and **CVQ** further show that visual quantization can be text-aligned, any-resolution, or channel-wise rather than patch-wise [2606.27313], [2605.26089]. These methods are not VEQ in the strict MoE PTQ sense, but they show that modern visual quantization increasingly depends on alignment, utilization, routing geometry, and structural heterogeneity rather than on uniform scalar approximation alone.

## 6. Limitations, interpretation, and future directions

VEQ is deliberately scoped. It depends on calibration data to estimate modality token counts, the gradient ratio \(\gamma\), expert activation frequencies, and router affinities. If calibration data is unrepresentative, the weighting scheme can become suboptimal. The reported experiments focus on **weight-only PTQ**, especially **W3A16** and **W4A16**, rather than fully quantized activations or end-to-end ultra-low-bit deployment of all components. The method also does not provide a dedicated analysis of **router quantization** itself, even though it relies on router outputs for affinity-aware calibration [2602.01037].

Implementation complexity is another explicit limitation. Compared with vanilla AWQ or GPTQ, VEQ requires token modality labels, expert-routing statistics, and affinity-weighted Hessian construction. The published evaluation is also limited to two MoE VLM families, **Kimi-VL** and **Qwen3-VL**, so broader architectural validation remains open [2602.01037].

These limitations clarify a common misconception. VEQ is not a generic claim that “multimodal PTQ should weight text more heavily.” Its specific contribution is the joint treatment of **which experts matter most** and **which token directions matter most within each expert**. Expert awareness alone would still allow abundant vision tokens to dominate curvature estimation; modality awareness alone would still ignore MoE routing imbalance. VEQ’s two modules are intended to work together: VEQ-ME protects the right experts at the macro level, and VEQ-MA protects the right token directions at the micro level [2602.01037].

The natural extensions named for the framework are **activation quantization**, **mixed precision across experts**, **router-aware end-to-end compression**, and **more adaptive or online calibration for domain-shifted multimodal inputs** [2602.01037]. A plausible implication is that future VEQ-style research will increasingly combine modality-aware statistics, expert-aware allocation, and dynamic token-level structure rather than treating MoE VLMs as ordinary dense transformers with more parameters.

Source: https://www.emergentmind.com/topics/visual-expert-quantization-veq