SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference

Abstract: Low-bit floating-point formats and semi-structured sparsity are increasingly supported by modern accelerators, yet combining them for LLM activation compression remains challenging: activations contain input-dependent outliers that dominate block scales in FP4 quantization, and directly applying N:M sparsity masks discards moderate values, coupling sparsification loss with quantization error. We introduce SharQ, a training-free inference method that bridges activation sparsity and FP4 quantization through an online sparse--dense decomposition. For each activation tensor, SharQ generates an input-adaptive N:M mask to extract an outlier-dominated sparse backbone, quantizes it to FP4, and defines a dense residual relative to the quantized sparse backbone rather than the unquantized sparse values. A sparse FP4 GEMM processes the backbone while a dense FP4 GEMM compensates for both mask-induced activation loss and sparse-path quantization error. The two paths share a single FP4 weight payload with path-specific scale views, and a fused preparation kernel absorbs mask generation, residual construction, and layer normalization into one operator. SharQ requires no calibration data, retraining, or model-specific tuning. Evaluated on Llama-3.1-8B, Qwen2.5-7B, Qwen3-30B-A3B, and Qwen3-VL-8B, SharQ recovers 43--63% of the NVFP4-to-FP16 accuracy gap across language and vision-language tasks, and generalizes across NVFP4, HiF4, and MXFP4 formats. On an RTX 5090, SharQ delivers 2.2--2.4$\times$ latency reduction over FP16 and 1.2--1.4$\times$ throughput improvement over FP8 in LLM serving, and up to 1.58$\times$ speedup on Wan2.2-T2V-A14B video generation when combined with SageAttention. Our code is available at https://github.com/actypedef/SharQ.

• The paper presents a training-free sparse-dense decomposition that synergizes FP4 quantization with N:M sparsity, reducing the FP4-to-FP16 accuracy gap.
• It deploys a shared-weight, dual-GEMM kernel combining a sparse FP4 backbone and a dense residual path to mitigate quantization and sparsification errors.
• Empirical evaluations on various LLMs demonstrate significant improvements in latency and accuracy, with robust performance across multiple FP4 formats.

SharQ: Synergizing Activation Sparsity and FP4 Quantization for LLM Inference

Motivation and Problem Analysis

The proliferation of LLMs has underscored linear layers as system bottlenecks in both computational and memory cost, especially at deployment scale. Contemporary accelerators support both fine-grained, block-scaled 4-bit floating-point (FP4) numeric formats, such as NVFP4, MXFP4, and HiF4, and hardware-accelerated N:M semi-structured sparsity patterns. However, jointly leveraging FP4 quantization and N:M sparsity for activation compression in inference with minimal accuracy loss remains unsolved.

The core challenge arises due to input-dependent, high-magnitude activation outliers. In block-scaled FP4, these outliers set local group scales, reducing effective quantization precision for non-outlier values. Furthermore, naively applying N:M activation sparsity online is excessively lossy, since the mask is generated dynamically without retraining, discarding moderate-magnitude activation information—a regime intractable for static masks and prone to catastrophic accuracy degradation.

Prior solutions—outlier-aware quantization (LLM.int8, SmoothQuant, AWQ, GPTQ) or mixed-precision designs—either introduce hardware-inefficient auxiliary branches, irregular data flow, or significant storage overhead. The interaction between quantization and sparsity is entangled, not additive: the error from one influences the other, requiring algorithmic designs that explicitly model this coupling.

Methodology: Input-Adaptive Sparse–Dense Decomposition

SharQ introduces an online, training-free sparse–dense decomposition of activations, reframing sparsity from a "hard drop" to a "soft routing" mechanism that preserves hardware efficiency and numerical fidelity. The high-level pipeline is visualized below:

Figure 1: High-level overview of the SharQ pipeline. SharQ applies an input-adaptive N:M sparse mask to select outlier-dominated activations, defines a dense residual relative to the quantized sparse backbone, and executes both paths with shared FP4 weights, enabling joint compensation of sparsification and quantization errors within hardware-optimized kernels.

Sparse Backbone Extraction

For each activation tensor, SharQ dynamically computes a hardware-valid N:M mask (e.g., 4:8 in-pairs for NVFP4, 2:4 for HiF4). The mask selects the largest-magnitude elements in each block, forming a sparse "backbone" that captures the dominant outlier structure. This backbone is then quantized into FP4.

Dense Residual Construction

Instead of dropping moderate values outside the N:M mask, SharQ defines a dense residual as the difference between the original activations and the quantized sparse backbone output. This residual thus simultaneously encodes both mask-induced activation loss and quantization error from the sparse path. The dense residual is also quantized to FP4.

Shared-Weight, Dual-GEMM Execution

Both the sparse backbone and the dense residual paths use a single FP4 weight payload, with path-specific scale layouts (e.g., group-32 for sparse GEMM, group-16 for dense GEMM). The execution consists of:

• A hardware-accelerated sparse FP4 GEMM on the backbone
• A dense FP4 GEMM on the residual that accumulates directly into the output

This is realized via a fused activation preparation kernel, which performs mask generation, FP4 quantization, block normalization, and residual computation in a single pass, minimizing data movement and kernel overhead.

Figure 2: Kernel-level dataflow for SharQ on NVFP4. The preparation kernel jointly handles RMSNorm/PTS, mask generation, sparse compression, dense residual calculation, and weight quantization with path-specific views, culminating in sparse and dense GEMMs that cooperatively build the output.

Theoretical Perspective

By isolating outliers, the dense residual becomes well-suited to FP4 quantization, as its local dynamic range is less distorted by extreme values. SharQ's decomposition converts the coupled error from sparsity and quantization into a single residual approximation problem, absorbed by the dense path. Under fixed low-bit weights, the output perturbation is provably bounded by the quality of this residual compensation.

Experimental Results

Accuracy Recovery

SharQ is evaluated on Llama-3.1-8B, Qwen2.5-7B, Qwen3-30B-A3B (MoE LLM), and Qwen3-VL-8B (vision-language LLM), always using zero calibration data and no retraining. The main findings are:

• On Llama-3.1-8B, SharQ raises the average zero-shot accuracy from 70.32% (NVFP4) to 71.55%, closing 56% of the FP4-to-FP16 gap.
• WikiText2 perplexity improves from 6.94 to 6.73.
• MMLU score increases by 1.8 percentage points.
• On Qwen2.5-7B and Qwen3-30B-A3B, the FP4-to-FP16 gap closure is 63% and ~40%, respectively, with consistent improvements across all evaluated tasks.
• On Qwen3-VL-8B, SharQ closes 43% of the FP4 accuracy gap for multimodal benchmarks, showing strong generalization to vision-LLMs.

The results are plug-and-play: SharQ yields consistent recovery across diverse architectures and scales without dataset- or model-specific tuning.

Efficiency

SharQ demonstrates robust speedups in real-world serving scenarios:

Figure 3: End-to-end LLM serving efficiency on RTX 5090 (vLLM, Llama-3.1-8B, batch=4). SharQ outperforms FP16 and FP8 in request latency and throughput at all sequence lengths.

• Latency: 2.2–2.4× lower than FP16, 1.2–1.4× faster than FP8 throughput across all sequence lengths.
• Video Generation: On Wan2.2-T2V-A14B the method delivers up to 1.58× speedup when combined with SageAttention, indicating composability with attention-level optimizations.

  Figure 4: (a) SharQ fuses quantization, mask generation, and normalization, saving prefilling time; (b) Sparse NVFP4 GEMMs reach 0.66–0.72× the latency of dense. FP8 GEMMs are significantly slower.

Kernel benchmarking reveals that the sparse FP4 backbone provides 28–34% GEMM speedup via hardware N:M sparsity, while the residual path overhead is mitigated by kernel fusion. This enables SharQ’s strong accuracy-latency tradeoff.

Format Generalization and Ablations

SharQ generalizes effectively to all major block-scaled FP4 formats:

• On NVFP4, HiF4, and MXFP4, SharQ always lifts accuracy and reduces perplexity versus vanilla RTN-based FP4 quantization, especially for the most quantization-sensitive tasks.
• Path ablation shows that both the sparse backbone (for efficiency) and the dense residual (for accuracy) are essential; removing the residual is catastrophic for quality, while dual dense paths forgo all sparsity benefits.

Implications and Future Directions

SharQ reifies a new regime of training-free LLM inference: algorithm–hardware co-designed, plug-and-play, sparse–dense low-bit execution, requiring no calibration, auxiliary precision, or model-specific pipelines. It transforms activation sparsity from a purely theoretical advantage to a practical accelerator-aligned primitive by fusing it with block-scaled FP4 quantization.

The method’s flexibility across FP4 dialects, model architectures, and modalities implies broad utility as hardware and software FP4 stacks mature. Potential future work may include:

• Extending the dual-path decomposition to dynamic granularity adjustment depending on per-layer or per-token statistics.
• Joint optimization of weight and activation quantization under the SharQ regime.
• Exploring its interaction with ultra-low-bit (<4b) activations and SNN/GNN architectures.

Conclusion

SharQ provides an inference-time solution that bridges FP4 quantization and N:M activation sparsity via online sparse–dense decomposition. A fused kernel design, shared-weight reuse with scale remapping, and accumulation-based residual correction synergistically yield state-of-the-art efficiency–accuracy trade-offs for modern LLMs and multimodal models. It sets the foundation for future research in full-stack algorithm–hardware co-design for LLM inference.

Reference: See original paper "SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference" (2606.26587).