Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment

Abstract: The emergence of vision LLMs with fewer than 3 billion parameters has accelerated the implementation of on-device multimodal intelligence. However, a detailed understanding of component-wise quantization remains a bottleneck for optimal deployment. This paper presents a systematic evaluation framework for empirically validating five hypotheses across six quantization configurations on the Jetson Orin NX and AGX. By separating the vision encoder, projector, and LLM backbone yields the following results: (1) Quantization sensitivity is governed by the structural paradigm (MoE vs. dense) rather than scale alone, with MoE backbones mitigating INT4 noise where dense backbones degrade; (2) SigLIP encoders incur disproportionate INT8 latency on Jetson Ampere--a deployment-specific encoder-kernel-hardware interaction, not a SigLIP flaw; (3) Although INT4 quantization of LLMs greatly reduces VRAM consumption, it also causes slower token generation due to dequantization overhead; (4) Composite quantization errors are largely additive, except along the modality-alignment path, which is architecture-dependent; (5) The intelligence-per-joule profile varies significantly across platforms owing to memory bandwidth constraints.

• The paper presents a component-wise quantization analysis revealing that MoE architectures offer notable INT4 resilience compared to dense models.
• It employs a modular framework to benchmark five sub-3B VLMs on Jetson platforms, quantifying MME accuracy, VRAM savings, and latency overhead.
• The study highlights the need for hardware-software co-design, as energy and latency trade-offs vary significantly with different quantization configurations.

Rethinking Small VLM Quantization: Component Analysis and Hardware-Aware Edge Deployment

Introduction

The paper "Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment" (2607.08029) delivers a systematic dissection of quantization strategies for sub-3B parameter vision-LLMs (sVLMs) targeting heterogeneous edge AI platforms. In contrast to prevailing end-to-end quantization analyses on data-center hardware, this work evaluates quantization effects at the component granularity—vision encoder, projector, and LLM backbone—across a suite of sVLM families including dense and Mixture-of-Experts (MoE) architectures. The focus is empirical validation on representative Jetson Orin-series SoCs, measuring MME benchmark accuracy, VRAM, decomposed latency, and intelligence-per-joule (IPJ).

Methodology

A modular, hypothesis-driven framework is used to interrogate quantization impact. Six quantization configurations are explored, covering isolated and composite quantization of vision encoder (INT8), projector (INT8), and LLM backbone (INT4/BnB). Five diverse sVLMs (e.g., Qwen3-VL-2B, DeepSeek-VL2-Tiny, PaliGemma2-3B, LLaVA-OV-0.5B, Kosmos-2.5) are benchmarked on Jetson Orin NX and AGX platforms using the MME suite. Latency profiling is segmented by model component, and power consumption is directly measured for energy-centric evaluation. This approach enables rigorous ablation of scale vs. architectural determinants of quantization sensitivity, and exposes cross-platform deployment bottlenecks.

Empirical Findings

Structural Determinants of Quantization Sensitivity

Contrary to previous conjectures emphasizing parameter count as the dominant driver of INT4 quantization vulnerability, results demonstrate that the presence of MoE in the LLM backbone is a primary determinant—MoE architectures such as Qwen3-VL and DeepSeek-VL2 are consistently more resilient, showing INT4 quantization improvements or minimal degradation, while dense backbones (e.g., PaliGemma2, LLaVA-OV) suffer pronounced accuracy loss. Notably, Qwen3-VL (1.72B, MoE) yields a +56 MME delta under LLM INT4, while dense LLaVA-OV (0.49B) exhibits -220 MME.

Figure 1: Accuracy delta per quantization configuration across architectures, highlighting architecture-dependent INT4 sensitivity.

Hardware-Software Stack Latency Interactions

A sharp, architecture-induced latency penalty emerges in SigLIP-So400m-based vision encoders under INT8 quantization using BitsAndBytes on Jetson Ampere. While accuracy remains stable, per-token vision encoding latencies escalate by up to 4.6x compared to FP16, a property not observed in non-SigLIP encoders (e.g., Kosmos-2.5, $<1.2$x overhead). This latency anomaly is traced to quantization kernel fragmentation and synchronization overhead, owing to poor alignment with SigLIP’s ViT computation.

Resource vs. Efficiency Trade-offs under INT4

While LLM INT4 quantization supported by BitsAndBytes consistently achieves ~40–50% VRAM savings, generation latency (TPOT) rises by 10–56%, and energy consumption increases due to prohibitively expensive dequantization routines. Gains in memory footprint do not yield throughput improvements on Jetson Orin platforms, refuting naive assumptions about quantization efficiency.

Composite Quantization Error Composition

Additivity of accuracy degradation holds for LLM INT4 + projector INT8 configurations, but breaks down for LLM INT4 + vision INT8; non-additive (super- or sub-additive) interactions depend on the architecture, particularly evident in models with entangled modality alignment mechanisms. PaliGemma2 and DeepSeek-VL2 show distinct compositional patterns, highlighting the imperative for component-interaction-aware quantization planning.

Platform-Invariant Model Accuracy, but Platform-Specific Efficiency

Accuracy ranking is strictly platform-invariant for a given model and configuration, e.g., Qwen3-VL-2B $\rhd$ DeepSeek-VL2 $\rhd$ PaliGemma2, across Jetson NX and AGX. However, energy usage and IPJ efficiency are highly platform-dependent due to bandwidth, kernel, and TDP factors. For instance, Qwen3-VL achieves up to 2.5x higher IPJ on AGX vs. NX, indicating that memory interface width directly influences energy efficiency.

Implications and Future Directions

Practical Deployment Implications

• MoE sVLMs should be prioritized for aggressive LLM quantization on memory/bandwidth-limited edge HW due to their quantization resilience.
• Quantization strategies must be hardware- and software-stack-aware; for SigLIP-based models with INT8 quantization, latency bottlenecks demand custom low-level kernel optimization or architectural revision.
• Resource savings (e.g., VRAM) must be evaluated jointly with actual throughput and energy behavior; INT4 may be preferable for loadable model size but not for latency/energy unless dequantization is hardware-accelerated.

Theoretical Insights

• Quantization effects are architecture-dependent, not strictly scale-dependent, contradicting parameter-centric regularities. The combination of sparse MoE activations with decoupled gradients increases robustness to quantization noise.
• Accuracy degradation under joint quantization is not uniformly additive, signaling intricate dependency pathways between image and text branches.

Future Research Directions

• Extension to hardware-native quantization formats (e.g., FP8) and weight-activation quantization schemes (W8A8) is required to assess dequantization overhead mitigation.
• NAS-driven automatic precision allocation can further optimize the quantization schedule for specific architecture/hardware pairs, facilitating true HW-SW co-design for energy-scaled AI.

Conclusion

This work establishes that effective sVLM deployment on edge platforms demands component-wise, architecture/hardware-aware quantization policies. It underscores that dominant architectural features (e.g., MoE) determine INT4 robustness, that vision encoder quantization can manifest hardware/software stack interaction effects uncorrelated with accuracy, and that energy efficiency is a product of the underlying HW-SW co-design, not just quantization settings. The findings motivate a shift toward precision scheduling guided by component interactions and deployment constraints, rather than simplistic uniform quantization. The modular evaluation framework herein provides a blueprint for future hardware-in-the-loop sVLM optimization.