Visual Expert Quantization (VEQ)

• Visual Expert Quantization (VEQ) is a framework designed to address modality and expert heterogeneity in MoE vision-language models.
• It integrates VEQ-ME and VEQ-MA modules to tailor calibration objectives and Hessian approximations for protecting high-impact experts and token directions.
• Empirical results demonstrate that VEQ, especially its VEQ-MA component, achieves state-of-the-art improvements in low-bit weight-only quantization accuracy.

Visual Expert Quantization (VEQ) is a post-training quantization framework for Mixture-of-Experts (MoE) vision-LLMs (VLMs) that treats quantization as a problem shaped by two distinct forms of heterogeneity: the discrepancy between visual and language tokens, and the unequal contribution of different experts. Introduced as a dual-aware quantization method for low-bit weight-only compression, VEQ modifies both expert-level reconstruction objectives and token-level second-order calibration so that pivotal experts and modality-sensitive token directions receive greater protection under quantization. Its two principal components are Modality-expert-aware Quantization (VEQ-ME) and Modality-affinity-aware Quantization (VEQ-MA), implemented respectively as AWQ-style and GPTQ-style extensions for MoE VLMs (Qin et al., 1 Feb 2026).

1. Problem formulation and motivating heterogeneity

VEQ is defined for post-training quantization (PTQ) of MoE vision-LLMs, with emphasis on weight-only quantization such as W3A16 and W4A16. The motivating models are MoE VLMs including Kimi-VL-Instruct and Qwen3-VL-30B-A3B-Instruct, where quantization is attractive because full models remain expensive in memory and inference cost even though MoE activates only a subset of experts per token (Qin et al., 1 Feb 2026).

The framework begins from the claim that conventional PTQ methods such as RTN, AWQ, and GPTQ are inadequate for MoE VLMs because they assume a comparatively homogeneous calibration regime. VEQ argues that this assumption fails in two ways. First, vision and text tokens are not statistically equivalent. Vision tokens are numerous and spatially redundant, whereas text tokens are fewer, semantically dense, and more influential on final outputs. Using the supervised fine-tuning loss gradient as a sensitivity proxy, the reported average gradient magnitude of text tokens exceeds that of vision tokens by a factor of 22.4 on 128 COCO samples, and the text-to-vision gradient ratio is around 15 in a representative sample. Second, experts are not equally important. MoE routing is sparse and imbalanced: some experts are hot, some are nearly dormant, some are modality-specialized, and only a few experts typically receive large routing probabilities for a given token (Qin et al., 1 Feb 2026).

These observations define VEQ’s central problem. A calibration set with many image tokens can dominate unweighted statistics even when text-token perturbations are more harmful, and a uniform expert-wise objective can allocate equal effort to hot and dormant experts even though their downstream effect is highly unequal. VEQ therefore treats successful compression of MoE VLMs as requiring joint modeling of modality heterogeneity and expert heterogeneity, rather than treating quantization as a layer-uniform approximation problem (Qin et al., 1 Feb 2026).

2. Modality-expert-aware quantization

VEQ-ME addresses the macro-level question of which experts deserve more protection. It extends an AWQ-like calibration objective by weighting each expert according to a modality-balanced importance score derived from routed token counts and gradient sensitivity (Qin et al., 1 Feb 2026).

The construction uses four calibration statistics for expert $i$: the number of routed text tokens $N_i^{\text{text}}$, the number of routed vision tokens $N_i^{\text{vis}}$, the total text-token count $T_{\text{text}}$, and the total vision-token count $T_{\text{vis}}$. It also uses the modality sensitivity ratio

$$\gamma = \frac{\|\nabla_{\text{text}}\|}{\|\nabla_{\text{vis}}\|},$$

together with the quantity normalization factor

$\beta = \frac{T_{\text{text}}}{T_{\text{vis}}}.$

The resulting expert importance score is

$$S_i = \gamma \cdot N_i^{\text{text}} + \beta \cdot N_i^{\text{vis}}.$$

This score explicitly amplifies text-token traffic through $\gamma$ and rescales visual-token traffic through $\beta$. The standard expert-wise reconstruction objective

$N_i^{\text{text}}$0

is then replaced by the weighted objective

$N_i^{\text{text}}$1

where $N_i^{\text{text}}$2 is the number of experts, $N_i^{\text{text}}$3 and $N_i^{\text{text}}$4 are the full-precision and quantized weights of expert $N_i^{\text{text}}$5, and $N_i^{\text{text}}$6 is the routed input to that expert (Qin et al., 1 Feb 2026).

The significance of VEQ-ME is not that it minimizes uniform numerical error more aggressively, but that it minimizes task-relevant error more selectively. Frequently used experts, and especially experts heavily used by text tokens, receive larger weights during scale search and calibration. A common misconception is that MoE PTQ can be made multimodal simply by balancing image and text tokens globally. VEQ-ME rejects that view: modality balancing is carried out at the expert level, where routing imbalance and expert specialization actually appear (Qin et al., 1 Feb 2026).

3. Modality-affinity-aware quantization

VEQ-MA addresses the micro-level question of which token directions should dominate second-order calibration within an expert. It extends GPTQ by replacing the usual uniform Hessian surrogate with an affinity- and modality-weighted Hessian that emphasizes tokens strongly associated with the current expert and disproportionately weights text-token directions (Qin et al., 1 Feb 2026).

In standard GPTQ notation, the Hessian approximation is

$N_i^{\text{text}}$7

where $N_i^{\text{text}}$8 is the calibration matrix. VEQ-MA argues that this form is inappropriate for MoE VLMs because it assigns equal influence to all routed tokens even though some are weakly associated with the expert and some arise from the less sensitive visual modality. For token $N_i^{\text{text}}$9, let $N_i^{\text{vis}}$0 be its affinity to the target expert. VEQ-MA defines

$N_i^{\text{vis}}$1

With diagonal matrix $N_i^{\text{vis}}$2 whose entries are $N_i^{\text{vis}}$3, the enhanced Hessian is

$N_i^{\text{vis}}$4

This replacement changes GPTQ’s inverse-Hessian-based updates so that the quantizer protects directions that are simultaneously expert-relevant and modality-sensitive. High-affinity tokens contribute more because they better represent the expert’s actual operating region; text tokens contribute more because their gradients are larger and their semantic density is higher. In this way VEQ-MA aligns second-order calibration with MoE routing structure rather than with undifferentiated token covariance (Qin et al., 1 Feb 2026).

The method also appears in a sensitivity study through the interpolation

$N_i^{\text{vis}}$5

where $N_i^{\text{vis}}$6 recovers raw router confidence and $N_i^{\text{vis}}$7 removes affinity weighting. The reported best stability occurs at $N_i^{\text{vis}}$8 and $N_i^{\text{vis}}$9, which the paper interprets as evidence that router affinity is genuinely informative rather than incidental (Qin et al., 1 Feb 2026).

4. Empirical performance and ablation evidence

VEQ is evaluated on Kimi-VL-Instruct and Qwen3-VL-30B-A3B-Instruct over MMMU, AI2D, InfoVQA, TextVQA, RealWorldQA, ScienceQA, VizWiz-VQA, MMBench, and MME-RealWorld, using the lmms-eval framework and SGLang as the inference backend. The emphasis is on weight-only PTQ under W3 and W4, with the abstract highlighting W3A16 (Qin et al., 1 Feb 2026).

The headline result is that VEQ sets a new state of the art at W3A16. On Kimi-VL, average accuracies are 60.74 for RTN, 63.37 for AWQ, 62.93 for MBQ, 62.33 for GPTQ, 64.00 for VEQ-ME, and 65.41 for VEQ-MA, yielding a +2.04 average gain over the best baseline. On Qwen3-VL, the corresponding averages are 63.17, 62.22, 60.97, 64.16, 63.93, and 67.14; the abstract reports a 3.09% average gain over prior state of the art, while the tabulated W3 averages place VEQ-MA roughly three points above GPTQ. At W4, gains are smaller but remain consistent: 73.57 versus 72.96 on Kimi-VL, and 74.36 versus 73.20 on Qwen3-VL (Qin et al., 1 Feb 2026).

Setting	Kimi-VL average	Qwen3-VL average
W3 best baseline	63.37	64.16
W3 VEQ-MA	65.41	67.14
W4 best baseline	72.96	73.20
W4 VEQ-MA	73.57	74.36

Task-level behavior is consistent with the method design. The paper specifically highlights TextVQA under Kimi-VL W3, where GPTQ scores 64.30 and VEQ-MA reaches 78.30. More generally, VEQ is reported as especially robust on reasoning-intensive and OCR-heavy tasks such as MMMU, InfoVQA, and TextVQA, where preserving text-sensitive directions and decisive experts matters most (Qin et al., 1 Feb 2026).

The ablation studies separate the two modules cleanly. For VEQ-ME, removing the gradient factor by setting $T_{\text{text}}$0 reduces the average to 62.02, and removing the quantity factor by setting $T_{\text{text}}$1 gives 62.30, compared with 63.27 for full VEQ-ME. For VEQ-MA, removing affinity by setting $T_{\text{text}}$2 gives 63.63, removing modality weighting by setting $T_{\text{text}}$3 gives 62.46, and full VEQ-MA gives 63.96. These results indicate that both routing confidence and modality-aware weighting matter, with modality-aware weighting contributing more strongly in the reported averages (Qin et al., 1 Feb 2026).

5. Relation to adjacent quantization research

VEQ belongs to a broader shift in visual and multimodal quantization research in which quantization is treated as structure-aware rather than architecture-agnostic. Within that landscape, its closest neighboring formulation is Quant Experts (QE), which also targets VLM PTQ through a mixture-of-experts design. QE divides important channels into token-independent and token-dependent groups, uses a shared expert for global quantization error compensation, and introduces routed experts implemented as multiple routed low-rank adapters for token-specific local error. It is evaluated on Qwen2VL and InternVL2 under W4A6, W4A8, and W3A16, and reports consistent gains from 2B to 72B models (Jia et al., 27 Feb 2026). VEQ and QE are closely related in motivation but differ in mechanism: VEQ modifies calibration objectives and Hessian structure on expert weights, whereas QE reconstructs residual quantization error through expert-conditioned low-rank adapters. This suggests two complementary interpretations of “expert-aware” quantization in VLMs: one based on importance-weighted calibration, the other on expertized residual compensation.

A second neighboring line concerns hard visual discretization and routing rather than MoE PTQ itself. DiVeQ and SF-DiVeQ treat vector quantization as a differentiable reparameterization problem, keeping hard nearest-neighbor assignment in the forward pass while improving gradient flow and, in the space-filling variant, improving codebook utilization through assignment to line segments between codewords (Vali et al., 30 Sep 2025). Although not framed as MoE VLM compression, this work is directly relevant to VEQ-like systems whenever hard expert or codeword assignment blocks end-to-end training.

A third line addresses large-scale or semantically aligned visual tokenization rather than expert-aware PTQ. AlignedVQ quantizes normalized intermediate visual token features in a partitioned VLM, using dual linear projections around the quantizer and achieving approximately 1365x compression of transmitted features with small VQA accuracy change across eight datasets (Liu et al., 2024). IBQ addresses codebook collapse in large learned visual vocabularies by applying a straight-through estimator to the one-hot categorical index, enabling codebooks as large as $T_{\text{text}}$4 with high utilization (Shi et al., 2024). ViQ and CVQ further show that visual quantization can be text-aligned, any-resolution, or channel-wise rather than patch-wise (Yu et al., 25 Jun 2026, Song et al., 25 May 2026). These methods are not VEQ in the strict MoE PTQ sense, but they show that modern visual quantization increasingly depends on alignment, utilization, routing geometry, and structural heterogeneity rather than on uniform scalar approximation alone.

6. Limitations, interpretation, and future directions

VEQ is deliberately scoped. It depends on calibration data to estimate modality token counts, the gradient ratio $T_{\text{text}}$5, expert activation frequencies, and router affinities. If calibration data is unrepresentative, the weighting scheme can become suboptimal. The reported experiments focus on weight-only PTQ, especially W3A16 and W4A16, rather than fully quantized activations or end-to-end ultra-low-bit deployment of all components. The method also does not provide a dedicated analysis of router quantization itself, even though it relies on router outputs for affinity-aware calibration (Qin et al., 1 Feb 2026).

Implementation complexity is another explicit limitation. Compared with vanilla AWQ or GPTQ, VEQ requires token modality labels, expert-routing statistics, and affinity-weighted Hessian construction. The published evaluation is also limited to two MoE VLM families, Kimi-VL and Qwen3-VL, so broader architectural validation remains open (Qin et al., 1 Feb 2026).

These limitations clarify a common misconception. VEQ is not a generic claim that “multimodal PTQ should weight text more heavily.” Its specific contribution is the joint treatment of which experts matter most and which token directions matter most within each expert. Expert awareness alone would still allow abundant vision tokens to dominate curvature estimation; modality awareness alone would still ignore MoE routing imbalance. VEQ’s two modules are intended to work together: VEQ-ME protects the right experts at the macro level, and VEQ-MA protects the right token directions at the micro level (Qin et al., 1 Feb 2026).

The natural extensions named for the framework are activation quantization, mixed precision across experts, router-aware end-to-end compression, and more adaptive or online calibration for domain-shifted multimodal inputs (Qin et al., 1 Feb 2026). A plausible implication is that future VEQ-style research will increasingly combine modality-aware statistics, expert-aware allocation, and dynamic token-level structure rather than treating MoE VLMs as ordinary dense transformers with more parameters.