SharQ: Training-Free FP4 Sparse–Dense Inference

• SharQ is a training-free inference method that leverages an online sparse–dense decomposition to isolate activation outliers and compensate quantization errors.
• It generates input-adaptive N:M masks to extract an outlier-dominated sparse backbone quantized to FP4, while computing a dense residual for error correction.
• SharQ recovers 43–63% of the accuracy gap between NVFP4 and FP16 and reduces FP16 latency by 2.2–2.4× on modern accelerators.

Searching arXiv for the specified topic and closely related works. SharQ is a training-free inference method for LLMs that bridges activation sparsity and FP4 quantization through an online sparse–dense decomposition (Meng et al., 25 Jun 2026). It is designed for settings in which modern accelerators support both block-scaled 4-bit floating formats and N:M semi-structured sparsity, but where their naive combination is impaired by input-dependent activation outliers and by the coupling between sparsification loss and quantization error. For each activation tensor, SharQ generates an input-adaptive N:M mask to extract an outlier-dominated sparse backbone, quantizes it to FP4, and defines a dense residual relative to the quantized sparse backbone rather than the unquantized sparse values; inference then proceeds through a sparse FP4 GEMM for the backbone and a dense FP4 GEMM for the residual (Meng et al., 25 Jun 2026).

1. Problem formulation and design objective

SharQ is motivated by two properties of low-bit LLM inference. First, activations display input-dependent “massive” outliers. Second, inference is increasingly run on tensor cores supporting both block-scaled 4-bit floating formats and N:M semi-structured sparsity. The naive pipeline $X_{\mathrm{sparse}} = M(X) \odot X$ followed by FP4 quantization creates two intertwined problems: outliers dominate each small block’s scale, and sparsification alters the block’s distribution before quantization, so the resulting error is not simply “sparsification error + quantization error” but a coupled distortion (Meng et al., 25 Jun 2026).

Within that setting, SharQ’s key insight is to treat N:M sparsification not as final dropping but as the first stage of an online sparse–dense decomposition. The method routes outlier energy into a high-value sparse backbone path and routes everything else, including the quantization error on the backbone, into a dense residual path. Because the residual is defined relative to the already-quantized sparse backbone, the two error sources collapse into a single compensable residual term (Meng et al., 25 Jun 2026).

The paper positions this formulation against prior approaches that handle activation outliers via mixed precision, orthogonal rotations, or learned transforms, but at the cost of extra high-precision storage, irregular kernels, or offline calibration. SharQ instead requires no calibration data, retraining, or model-specific tuning (Meng et al., 25 Jun 2026).

2. Online sparse–dense decomposition

Given a dense activation tensor $X \in \mathbb{R}^{T \times K}$, SharQ approximates $XW$ by two complementary FP4 GEMMs: a sparse backbone path and a dense residual path (Meng et al., 25 Jun 2026).

The decomposition begins with input-adaptive N:M mask generation. $X$ is partitioned row-major into blocks of $M$ activations, such as $M=8$ for NVFP4 in-pairs. In each block, the $N$ entries of largest magnitude are selected to build $M(X) \in \{0,1\}^{T \times K}$ with exactly $N$ ones per block. This top-$N$ selection is performed by a fused “pair-topk” or “block-topk” primitive in hardware (Meng et al., 25 Jun 2026).

The sparse backbone is then extracted as

$X \in \mathbb{R}^{T \times K}$0

This backbone is quantized to FP4: $X \in \mathbb{R}^{T \times K}$1 Here the sparse-path quantizer and dequantizer select scales from the retained values (Meng et al., 25 Jun 2026).

The residual is defined after sparse quantization rather than before it: $X \in \mathbb{R}^{T \times K}$2 As stated in the paper, $X \in \mathbb{R}^{T \times K}$3 therefore contains both all values outside the mask and the FP4 quantization error on the retained outliers (Meng et al., 25 Jun 2026). This formulation is central: the dense path is not merely reconstructing dropped entries, but also compensating the sparse path’s own low-bit rounding error.

The final approximation to $X \in \mathbb{R}^{T \times K}$4 is

$X \in \mathbb{R}^{T \times K}$5

Both paths share a single FP4-quantized weight payload, but they view its scale metadata differently: the sparse path uses the semi-structured mask scale layout, whereas the dense path uses the standard block scales (Meng et al., 25 Jun 2026).

3. Kernel-level implementation

SharQ is built around three kernel-level optimizations and is organized so that inference remains a near-single-kernel dataflow per layer (Meng et al., 25 Jun 2026).

The first component is a fused activation preparation kernel. In one on-chip pass over the FP16 or BF16 activation $X \in \mathbb{R}^{T \times K}$6, the kernel performs layer normalization, local N:M top-k mask generation, sparse FP4 quantization and compression into the layout expected by the sparse GEMM, and dense residual computation

$X \in \mathbb{R}^{T \times K}$7

together with FP4 quantization input for the dense GEMM. The paper states that this fusion hides the mask-generation and residual cost inside the ordinary activation transform stage (Meng et al., 25 Jun 2026).

The second component is a shared FP4 weight payload with path-specific scale views. Rather than storing two separate quantized weight tensors, SharQ quantizes $X \in \mathbb{R}^{T \times K}$8 once into the hardware’s block-scaled FP4 format and retains a sparse-path scale layout and a dense-path scale layout. All share the same low-bit mantissa body in memory; only the metadata view differs (Meng et al., 25 Jun 2026). This design directly supports the two-path decomposition without duplicating the full quantized weight body.

The third component is accumulation-based residual compensation. Instead of materializing two full output branches and summing them, SharQ issues the sparse GEMM first and then uses the dense GEMM to accumulate into the same output buffer. The stated effect is removal of one extra writeback/readback for the output and preservation of a single output stream (Meng et al., 25 Jun 2026).

At the algorithmic level, the per-layer execution order is:

$X$6

The implementation is explicitly training-free and zero-calibration: all masks and scales are computed directly from the input block statistics, with no offline calibration sets or weight-oriented retraining (Meng et al., 25 Jun 2026).

4. Empirical performance

SharQ is evaluated on Llama-3.1-8B, Qwen2.5-7B, Qwen3-30B-A3B, and Qwen3-VL-8B, and the paper reports consistent recovery of 43–63% of the accuracy gap between NVFP4 and FP16 across language and vision-language tasks (Meng et al., 25 Jun 2026).

The main reported accuracy and systems results are summarized below.

Setting	Reported result	Context
Accuracy-gap recovery	43–63%	Language and vision-language tasks
Llama-3.1-8B zero-shot avg accuracy	70.32% $X \in \mathbb{R}^{T \times K}$9 71.55%	NVFP4 $XW$0 SharQ
FP16 latency reduction	2.2–2.4$XW$1	vLLM serving on RTX 5090
FP8 throughput improvement	1.2–1.4$XW$2	LLM serving
Wan2.2-T2V-A14B speedup	up to 1.58$XW$3	Combined with SageAttention

For Llama-3.1-8B, zero-shot average accuracy increases from 70.32% under NVFP4 to 71.55% under SharQ, closing 56% of the FP16 gap. The paper further reports that Qwen2.5-7B recovers approximately 63%, Qwen3-30B-A3B recovers approximately 42%, and Qwen3-VL-8B closes 43% of the NVFP4-to-FP16 gap on multimodal tasks (Meng et al., 25 Jun 2026).

On an RTX 5090, SharQ delivers 2.2–2.4$XW$4 latency reduction over FP16 in vLLM serving for batch 4 and sequence length 256–2048, and 1.2–1.4$XW$5 total token throughput improvement over FP8 weight-and-activation quantization (Meng et al., 25 Jun 2026). For video generation with Wan2.2-T2V-A14B on $XW$6 RTX 5090, SharQ alone provides 1.23$XW$7 speedup at 480 P and 1.11$XW$8 at 720 P over FP16, while SharQ combined with SageAttention reaches 1.58$XW$9 at 480 P and 1.57$X$0 at 720 P (Meng et al., 25 Jun 2026).

At the kernel level, the sparse NVFP4 GEMM with 4:8 sparsity runs at approximately 0.66–0.72$X$1 the latency of dense NVFP4, corresponding to a 28–34% arithmetic saving. The paper also states that FP8 GEMM is approximately $X$2 slower than dense NVFP4, so the sparse FP4 path is substantially more efficient than uniform FP8 (Meng et al., 25 Jun 2026).

5. Comparisons, ablations, and format generalization

The comparison framework in the paper separates SharQ from several alternative two-path or uniform low-bit strategies (Meng et al., 25 Jun 2026).

Versus uniform FP4 quantization, specifically NVFP4-only, SharQ reclaims most of the FP16 accuracy loss with only approximately 1.1$X$3 overall prefill overhead, attributed to fusion and accumulation. Versus a two-path dense FP4 design, denoted dense+dense, SharQ achieves similar accuracy but with half the arithmetic on the backbone. Versus a two-path sparse FP4 design, denoted sparse+sparse, doubling the sparse path without dense compensation collapses to the NVFP4 baseline and cannot recover sparsification loss (Meng et al., 25 Jun 2026).

The method is described as format-agnostic and is applied “as is” to three FP4 families:

• NVFP4: E2M1 block + 4:8 in-pairs
• HiF4: three-level scaling + 2:4 sparsity
• MXFP4: OCP microscaling + 2:4

In ablation on Qwen3-30B-A3B, SharQ improved NVFP4, HiF4, and MXFP4 by similar margins. The paper gives as one explicit example that MXFP4 average accuracy improved by $X$4 percentage points and perplexity by $X$5 (Meng et al., 25 Jun 2026).

A plausible implication is that SharQ’s main contribution is not tied to one vendor-specific FP4 encoding. Instead, the decomposition principle appears to transfer across distinct scale layouts and sparsity patterns, provided the hardware exposes compatible semi-structured sparse and dense low-bit GEMMs.

6. Position within the broader literature

SharQ belongs to a broader family of methods whose names are similar but whose technical goals are unrelated. It should be distinguished from the explainability framework SHARQ for association rules on relational data, which defines a Shapley-value-based measure of an element’s contribution to a set of association rules and develops exact and amortized algorithms for computing that score (Ben-Efraim et al., 2024). It should also be distinguished from SHARQ in semantic communications, where the term denotes a semantic confidence-guided HARQ mechanism that replaces CRC-based error detection with a confidence score for selective segment retransmission (Hao et al., 24 Apr 2026).

The current SharQ method is specifically an LLM inference technique for activation compression under FP4 quantization and semi-structured sparsity (Meng et al., 25 Jun 2026). In that domain, its defining feature is the online sparse–dense decomposition in which the residual is constructed relative to the quantized sparse backbone rather than the unquantized sparse values. That design choice unifies mask-induced activation loss and sparse-path quantization error into a single residual correction problem, while retaining a shared FP4 weight payload and fused preparation kernel (Meng et al., 25 Jun 2026).

Taken together, the reported results place SharQ as a systems-oriented method for low-bit inference on modern tensor-core hardware: it is training-free, zero-calibration, compatible with multiple FP4 formats, and evaluated across both language and vision-LLMs. The paper’s evidence suggests that its practical contribution lies in reconciling activation sparsity with FP4 quantization without introducing extra high-precision storage, offline calibration, or model-specific tuning (Meng et al., 25 Jun 2026).