UniQL: Unified Quantization & Compression

• UniQL is a unified framework that jointly applies structured weight-sorting, masked LoRA fine-tuning, and INT4 post-training quantization to achieve efficient low-rank compression.
• It supports diverse architectures—including Transformers, SSMs, and hybrid models—while maintaining model accuracy within 5% at 15% pruning.
• The framework enables on-device elastic pruning for adaptive deployment, yielding memory savings of up to 4×–5.7× and significant token throughput improvements.

Searching arXiv for the specified papers and closely related records to ground the article. UniQL is a unified post-training quantization and low-rank compression framework for adaptive edge LLMs that targets deployment under limited memory and shared computational resources on mobile platforms. Its central design is a one-shot cloud pipeline that combines structured weight-sorting, masked LoRA fine-tuning, and $n$-bit post-training quantization into INT4, while preserving a single quantized artifact that can be pruned on device to any global sparsity up to $35\%$ without re-compression on GPU. The framework is presented as general across pure Transformers, State Space Models (SSMs), and hybrid Mamba–Transformer architectures, with reported memory savings of $4\times$–$5.7\times$, token-throughput improvements of $2.7\times$–$3.4\times$, and accuracy maintained within $5\%$ of the original models at $15\%$ pruning (Chiang et al., 3 Dec 2025).

1. Problem setting and architectural scope

UniQL is motivated by the observation that deploying LLMs on mobile platforms is constrained not only by static memory limits but also by uncertainty in resource availability caused by the current device workload. In response, the framework introduces on-device configurable pruning rates for edge LLMs, so that a single compressed model can adapt to changing runtime conditions rather than requiring separate precompiled variants (Chiang et al., 3 Dec 2025).

The method is explicitly framed as uniform across three model families. For pure Transformers, the paper lists Llama-3.1-8B and Qwen-2.5-7B. For SSMs, it lists Mamba2-8B. For hybrid models, it lists Nemotron-H and Bamba-v2. This breadth matters because the compression stack is not restricted to a single block type: the structured sorting procedures, low-rank factorization, and deployment path are all described as architecture-aware but framework-level rather than model-specific.

A common misunderstanding would be to treat UniQL as only a quantization method. The framework is instead defined by the joint use of structured sorting, masked low-rank recovery, and INT4 post-training quantization in a single workflow. Another misunderstanding would be to view it as limited to Transformer attention/MLP blocks; the paper explicitly adds a state-aware procedure for SSMs and supports hybrid Mamba–Transformer blocks as first-class targets.

2. Joint compression formulation

UniQL’s low-rank component approximates each weight matrix $W \in \mathbb{R}^{d_{\rm in} \times d_{\rm out}}$ by

$$W \;\approx\; U\,V,\quad U\in\mathbb R^{d_{\rm in}\times r},\;V\in\mathbb R^{r\times d_{\rm out}},\;r\ll\min(d_{\rm in},d_{\rm out}),$$

where $35\%$0 are obtained by truncated SVD of $35\%$1 (Chiang et al., 3 Dec 2025).

To make that factorization aware of subsequent quantization, UniQL minimizes

$35\%$2

where $35\%$3 is the quantization error under $35\%$4-bit uniform symmetric quantization:

$35\%$5

and

$35\%$6

Within self-attention value/output matrices $35\%$7, UniQL performs two back-to-back SVDs on the activation-weighted matrix $35\%$8 and on its product with $35\%$9. The critical implementation detail is the absorption of the diagonal singular values $4\times$0 into $4\times$1:

$4\times$2

with

$4\times$3

The stated purpose is that each column of $4\times$4 carries its own scale when quantized, thereby sharply reducing the group’s quantization error $4\times$5 (Chiang et al., 3 Dec 2025).

This formulation places quantization and low-rank compression in a coupled objective rather than a serial heuristic. A plausible implication is that the method is designed to avoid the common failure mode in which a good low-rank approximation becomes a poor INT4 artifact after naïve quantization. The paper’s ablation that “Quantization-aware SVD (fusing $4\times$6 into $4\times$7) adds 3–7 pts at INT4” is consistent with that interpretation.

3. Structured sorting and architecture-specific mechanisms

A major component of UniQL is efficient structured weight-sorting. The framework groups weights by module and sorts both rows and columns using importance scores derived from calibration activations, but it avoids the $4\times$8 Moore–Penrose inverse. For the MLP-layer case, the procedure computes

$4\times$9

forms

$5.7\times$0

and uses leverage scores

$5.7\times$1

to derive the column sort matrix. The paper states that, because it never forms the pseudo-inverse of size $5.7\times$2, this step runs approximately $5.7\times$3 faster in practice, with the example $5.7\times$4 min versus $5.7\times$5 min for $5.7\times$6 (Chiang et al., 3 Dec 2025).

For SSMs, UniQL introduces state-aware sorting rather than relying on Transformer-style channel correlations. SSM blocks are written with state recurrence

$5.7\times$7

The method collects the full state matrix $5.7\times$8 for each head, forms

$5.7\times$9

and again computes leverage scores $2.7\times$0. The stated rationale is that sorting by these scores prioritizes columns most used in state propagation, which is described as a criterion unique to SSMs (Chiang et al., 3 Dec 2025).

For pruned Transformer-family models, UniQL also introduces a fused RoPE kernel. Because structured sorting reorders the head dimension of $2.7\times$1 and $2.7\times$2, the method must gather matching $2.7\times$3 pairs for rotary positional embedding:

$2.7\times$4

where $2.7\times$5 swaps half the components. By storing only half the sorted index vector $2.7\times$6, the paper describes a single Triton kernel that fuses the gather with the RoPE transform, reducing memory traffic and yielding approximately $2.7\times$7 end-to-end speedup in TPOT (Chiang et al., 3 Dec 2025).

Taken together, these mechanisms show that UniQL is not a single compression primitive but a coordinated set of architecture-specific transformations. The reported ablation that “State-aware sorting is essential to avoid >10 pts drop in SSMs” indicates that the SSM path is not merely an adaptation of Transformer heuristics.

4. One-shot cloud workflow and on-device elasticity

UniQL’s deployment model is divided into a cloud-side compression pass and an edge-side configuration step. In the cloud pipeline, the paper lists four stages.

First, it computes layer-wise Block-Influence (BI) scores

$2.7\times$8

and derives pruning fractions $2.7\times$9 for each global target $3.4\times$0 via softmax smoothing. Second, it applies structured sorting. Third, it performs masked LoRA fine-tuning by randomly sampling $3.4\times$1 each batch and masking the bottom $3.4\times$2 fraction of channels. Fourth, it quantizes the full model into INT4, specified as $3.4\times$3 bits with group size $3.4\times$4 (Chiang et al., 3 Dec 2025).

At deployment time on device, the runtime selects a desired global pruning rate $3.4\times$5, unpacks INT4 weights, slices off the bottom-ranked $3.4\times$6 channels in each layer, re-packs, and runs the fused kernels. The framework emphasizes that no re-compression on GPU is needed after the original cloud pass (Chiang et al., 3 Dec 2025).

This separation of concerns is central to the system design. The cloud pass absorbs calibration, fine-tuning, and quantization into a single artifact; the device then only performs configurable pruning within the ranking established earlier. This suggests that UniQL is designed for elastic deployment scenarios in which available memory or compute budget may vary at inference time.

5. Empirical results

The paper reports results for one-pass adaptive pruning with INT4 plus LoRA, followed by pruning at $3.4\times$7. Table 4 reports $3.4\times$8–$3.4\times$9 memory reduction. At $5\%$0 pruning, the accuracy changes listed are:

• Llama-3.1-8B: $5\%$1 $5\%$2 pts$5\%$3
• Qwen-2.5-7B: $5\%$4 $5\%$5 pts$5\%$6
• Nemotron-H-8B: $5\%$7 $5\%$8 pts$5\%$9
• Mamba2-8B: $15\%$0 $15\%$1 pts$15\%$2 (Chiang et al., 3 Dec 2025)

The model-size comparison in Table 5 gives a concrete view of the memory footprint. For Llama-3.1-8B, FP16 is $15\%$3 GB, TRT-AWQ (W4) is $15\%$4 GB, UniQL (W4) is $15\%$5 GB, and UniQL (W4, p35) is $15\%$6 GB. For Qwen-2.5-7B, the corresponding values are $15\%$7 GB, $15\%$8 GB, $15\%$9 GB, and $W \in \mathbb{R}^{d_{\rm in} \times d_{\rm out}}$0 GB. For Nemotron-H-8B, FP16 is $W \in \mathbb{R}^{d_{\rm in} \times d_{\rm out}}$1 GB, UniQL (W4) is $W \in \mathbb{R}^{d_{\rm in} \times d_{\rm out}}$2 GB, and UniQL (W4, p35) is $W \in \mathbb{R}^{d_{\rm in} \times d_{\rm out}}$3 GB.

Latency results are reported on both A6000 and Nano 8G. On A6000, Table 6 gives Llama-3.1-8B TPOT values of $W \in \mathbb{R}^{d_{\rm in} \times d_{\rm out}}$4 ms for FP16, $W \in \mathbb{R}^{d_{\rm in} \times d_{\rm out}}$5 ms for TAO-HQQ (W4), $W \in \mathbb{R}^{d_{\rm in} \times d_{\rm out}}$6 ms for UniQL, and $W \in \mathbb{R}^{d_{\rm in} \times d_{\rm out}}$7 ms for UniQL (p35). For Nemotron-H-8B, the values are $W \in \mathbb{R}^{d_{\rm in} \times d_{\rm out}}$8 ms for FP16, $W \in \mathbb{R}^{d_{\rm in} \times d_{\rm out}}$9 ms for UniQL, and $$W \;\approx\; U\,V,\quad U\in\mathbb R^{d_{\rm in}\times r},\;V\in\mathbb R^{r\times d_{\rm out}},\;r\ll\min(d_{\rm in},d_{\rm out}),$$0 ms for UniQL (p35). On Nano 8G, Table 7 gives Qwen-2.5-7B TPOT as OOM for FP16, $$W \;\approx\; U\,V,\quad U\in\mathbb R^{d_{\rm in}\times r},\;V\in\mathbb R^{r\times d_{\rm out}},\;r\ll\min(d_{\rm in},d_{\rm out}),$$1 ms for TAO-HQQ (W4), $$W \;\approx\; U\,V,\quad U\in\mathbb R^{d_{\rm in}\times r},\;V\in\mathbb R^{r\times d_{\rm out}},\;r\ll\min(d_{\rm in},d_{\rm out}),$$2 ms for UniQL, and $$W \;\approx\; U\,V,\quad U\in\mathbb R^{d_{\rm in}\times r},\;V\in\mathbb R^{r\times d_{\rm out}},\;r\ll\min(d_{\rm in},d_{\rm out}),$$3 ms for UniQL (p35). For Mamba2-8B, FP16 is also OOM, with $$W \;\approx\; U\,V,\quad U\in\mathbb R^{d_{\rm in}\times r},\;V\in\mathbb R^{r\times d_{\rm out}},\;r\ll\min(d_{\rm in},d_{\rm out}),$$4 ms for UniQL and $$W \;\approx\; U\,V,\quad U\in\mathbb R^{d_{\rm in}\times r},\;V\in\mathbb R^{r\times d_{\rm out}},\;r\ll\min(d_{\rm in},d_{\rm out}),$$5 ms for UniQL (p35) (Chiang et al., 3 Dec 2025).

The key ablations identify the contribution of individual components. Masked LoRA fine-tuning recovers $$W \;\approx\; U\,V,\quad U\in\mathbb R^{d_{\rm in}\times r},\;V\in\mathbb R^{r\times d_{\rm out}},\;r\ll\min(d_{\rm in},d_{\rm out}),$$6–$$W \;\approx\; U\,V,\quad U\in\mathbb R^{d_{\rm in}\times r},\;V\in\mathbb R^{r\times d_{\rm out}},\;r\ll\min(d_{\rm in},d_{\rm out}),$$7 pts at $$W \;\approx\; U\,V,\quad U\in\mathbb R^{d_{\rm in}\times r},\;V\in\mathbb R^{r\times d_{\rm out}},\;r\ll\min(d_{\rm in},d_{\rm out}),$$8 pruning. Quantization-aware SVD adds $$W \;\approx\; U\,V,\quad U\in\mathbb R^{d_{\rm in}\times r},\;V\in\mathbb R^{r\times d_{\rm out}},\;r\ll\min(d_{\rm in},d_{\rm out}),$$9–$35\%$00 pts at INT4. The fused RoPE kernel delivers approximately $35\%$01 TPOT speedup. State-aware sorting is essential to avoid greater than $35\%$02 pts drop in SSMs. These results frame UniQL as a system in which accuracy retention and runtime efficiency depend on the interaction of several components, not solely on weight precision.

6. Position within related nomenclature

The term “UniQL” is not unique in the literature. In “UQE: A Query Engine for Unstructured Databases,” UniQL denotes the “Universal Query Language,” a dialect of SQL that extends predicates, projections, and grouping with quoted natural-language specifications and operates through an execution engine that combines sampling, stratification, online learning, and LLM calls (Dai et al., 2024).

That usage is conceptually unrelated to UniQL as a compression framework. In UQE, UniQL is a query language with a grammar over SELECT, FROM, WHERE, GROUP BY, ORDER BY, LIMIT, and TO, together with semantics for semantic filtering, semantic projection, and abstraction aggregation. By contrast, in the edge-LLM paper, UniQL denotes a unified quantization and low-rank compression pipeline for model deployment (Chiang et al., 3 Dec 2025).

This naming collision can produce confusion in bibliographic searches or acronym-based discussion. The available evidence indicates that the two uses refer to distinct research artifacts: one is a language-and-engine stack for unstructured database analytics, and the other is a post-training compression framework for adaptive edge LLMs.

7. Interpretation and significance

Within the scope defined by the paper, UniQL addresses a deployment problem that is simultaneously algorithmic and systems-oriented. Algorithmically, it couples low-rank approximation with quantization-aware optimization and masked recovery. Systems-wise, it externalizes the expensive compression stages to a single cloud pass while reserving pruning configurability for the device. The reported support for Transformers, SSMs, and hybrid models suggests an attempt at a model-family-agnostic compression interface rather than a narrowly tuned recipe (Chiang et al., 3 Dec 2025).

The main significance of the framework lies in the combination of three properties that are usually separated: INT4 quantization, low-rank compression with accuracy recovery, and post hoc adjustable pruning on device. The paper’s reported results—memory savings of $35\%$03–$35\%$04, token-throughput improvements of $35\%$05–$35\%$06, and accuracy within $35\%$07 of the original models at $35\%$08 pruning—suggest that this combination is viable without re-running compression for each deployment profile.

A plausible implication is that UniQL is particularly relevant for edge settings in which the same application must tolerate fluctuating resource envelopes, such as memory pressure or shared compute contention. The paper does not claim universal optimality; rather, it presents a unified post-training workflow whose distinctive contribution is elastic, on-device pruning from a single quantized artifact.