Mixed Low-Rank & Quantization (MLoRQ)

• MLoRQ is a compression framework that integrates low-rank structures with quantization to efficiently reduce model size while preserving critical information.
• It assigns distinct roles where the quantized component manages bulk parameters and the low-rank component captures residual patterns and activation-sensitive directions.
• MLoRQ techniques employ alternating optimization, fused hardware kernels, and discrete search strategies to achieve faster inference and improved trade-offs in memory-constrained settings.

Mixed Low-Rank and Quantization (MLoRQ) denotes a class of compression, adaptation, and inference schemes in which a model parameter, activation tensor, or measurement operator is represented through a low-rank component and a quantized component rather than by quantization or low-rank approximation alone. In contemporary transformer compression, the canonical form is an approximation such as $W \approx Q + LR$ under an explicit memory budget, but the same principle also appears as quantized base weights with low-rank adapters during fine-tuning, as mixed-precision subspace decompositions for activations, and as low-rank recovery from quantized measurements (Gordon et al., 13 Jul 2025, Guo et al., 2023, Xu et al., 2023, Saxena et al., 2024, Lybrand et al., 2017).

1. Conceptual scope and defining forms

The defining idea of MLoRQ is the assignment of distinct representational roles to a low-rank term and a quantized term. In weight compression, the quantized component typically carries the bulk of a matrix, while the low-rank component captures either residual structure, outliers, or activation-sensitive directions. In fine-tuning, the pretrained weights may remain quantized while low-rank updates provide task adaptation. In activation quantization, a small high-precision subspace or auxiliary path preserves difficult directions while the remainder is uniformly quantized (Cho et al., 2 Jun 2025, Xu et al., 2023, Saxena et al., 2024, Lee et al., 6 Apr 2026).

Setting	Representative form	Representative methods
Weight decomposition	$W \approx Q + LR$ or $W = UV + R$	LQ-LoRA, TwinQuant, ODLRI
Quantized fine-tuning	$y=\tilde W^\top x + s(AB)^\top x$	QA-LoRA, IntLoRA, AdaLoRA-QAT
Activation or subspace decomposition	$\hat A = U\hat C_H + V\hat C_L$ or $X=X_{\mathrm{Body}}+X_{\mathrm{Aux}}$	ResQ, MUXQ

MLoRQ is therefore not a single algorithm. The literature includes post-training quantization, quantization-aware training, parameter-efficient fine-tuning, accelerator co-design, and calibration-based or calibration-free procedures. Earlier mathematically rigorous antecedents also combine low-rank structure with quantization in matrix recovery and multivariate regression, where the objective is not neural-network deployment but reconstruction from finite-precision data (Lybrand et al., 2017, Chen et al., 2023).

2. Mathematical formulations

A central formulation writes a pretrained weight matrix as the sum of a quantized matrix and a low-rank correction. Representative objectives include

$\min_{Q,L,R}\ \|(W-Q-LR)X\|_F^2$

for activation-aware decomposition, and

$$\min_{Q\in\mathcal A^{N\times N'},\,L,R}\ \|XW-X(Q+LR)\|_F^2$$

for layer-wise reconstruction with calibration activations. In transformer compression under a global memory budget, MLoRQ is also posed as selecting per-layer compressed representations $S_i$ so that $\sum_i \mathrm{WeightsMemory}(S_i)\le \psi_{WMS}$ while minimizing task loss or maximizing layer-wise SQNR under candidate-specific size constraints (Cho et al., 2 Jun 2025, Zhang et al., 31 May 2026, Gordon et al., 13 Jul 2025).

Several variants refine this basic additive picture. TwinQuant decomposes each matrix as

$W \approx Q + LR$0

then learns quantization-friendly subspaces by orthogonal rotations $W \approx Q + LR$1 on the Stiefel manifold and invertible transforms $W \approx Q + LR$2, yielding transformed factors $W \approx Q + LR$3, $W \approx Q + LR$4, and $W \approx Q + LR$5. No full-precision approximation error is introduced by this reparameterization, but the post-quantization error changes substantially, so the optimization directly targets the quantized objective rather than the real-valued energy of a residual (Wang et al., 1 Jun 2026).

A second family treats fine-tuning itself as mixed low-rank and quantized computation. In QA-LoRA, the frozen pretrained weight $W \approx Q + LR$6 is replaced during training by a low-bit quantization $W \approx Q + LR$7, and the forward pass becomes

$W \approx Q + LR$8

Group-wise operators simultaneously increase the degrees of freedom of quantization and decrease those of adaptation, reducing adapter parameters from $W \approx Q + LR$9 to $W = UV + R$0 while increasing quantization degrees of freedom to $W = UV + R$1 (Xu et al., 2023).

Activation-side formulations are likewise important. ResQ identifies a rank-$W = UV + R$2 PCA subspace $W = UV + R$3 and its orthogonal complement $W = UV + R$4, quantizes coefficients in the $W = UV + R$5 subspace to a higher bit-width and those in the $W = UV + R$6 subspace to a lower bit-width, and reconstructs

$W = UV + R$7

MUXQ instead decomposes an activation matrix into a main Body path and a small Aux path for outlier channels, then performs INT8-only inference on both paths and recombines them with integer arithmetic (Saxena et al., 2024, Lee et al., 6 Apr 2026).

Outside neural-network deployment, low-rank matrix recovery from quantized measurements uses $W = UV + R$8 quantization and nuclear-norm minimization, while quantized low-rank multivariate regression uses dithered covariates and responses to construct unbiased surrogate covariances and nuclear-norm-regularized estimators (Lybrand et al., 2017, Chen et al., 2023).

3. Optimization strategies and role assignment

A recurring optimization pattern is alternating between quantization and low-rank fitting. LQ-LoRA iterates between factorizing $W = UV + R$9 by weighted or unweighted SVD and quantizing $y=\tilde W^\top x + s(AB)^\top x$0 under a chosen NormalFloat configuration, stopping when the reconstruction error increases. MiLo alternates a quantization step based on Half-Quadratic Quantization with a low-rank compensation step based on truncated SVD, repeating until the residual converges or early-stopping at approximately $y=\tilde W^\top x + s(AB)^\top x$1 (Guo et al., 2023, Huang et al., 3 Apr 2025).

Another pattern is explicit role assignment. The ODLRI method argues that previous joint schemes tend to “lock in” poor responsibilities: quantize-first makes the low-rank term a pure error corrector, whereas low-rank-first makes the quantized term a tiny residual adjuster. ODLRI therefore identifies activation-sensitive channels from the empirical Hessian $y=\tilde W^\top x + s(AB)^\top x$2, builds a restricted Hessian $y=\tilde W^\top x + s(AB)^\top x$3, whitens the weight matrix with $y=\tilde W^\top x + s(AB)^\top x$4, and initializes the low-rank term so that it reconstructs the outlier directions before joint optimization. MPQ-DMv2 applies a related idea to diffusion-model fine-tuning: its Object-Oriented Low-Rank Initialization sets the LoRA factors from the top-$y=\tilde W^\top x + s(AB)^\top x$5 truncated SVD of the quantization residual $y=\tilde W^\top x + s(AB)^\top x$6, invoking the Eckart-Young-Mirsky theorem for the best rank-$y=\tilde W^\top x + s(AB)^\top x$7 approximation (Cho et al., 2 Jun 2025, Feng et al., 6 Jul 2025).

Resource allocation is frequently discrete rather than continuous. The transformer-compression MLoRQ method performs an intra-layer search to enumerate Pareto-optimal low-rank/quantization candidates, then an inter-layer 0–1 ILP to select one candidate per layer under a memory budget. LQ-LoRA similarly solves an ILP over quantization configurations for each matrix, minimizing offline reconstruction error under a global bit budget. QR-Adaptor extends this line to quantized fine-tuning by treating per-layer bit-widths and LoRA ranks as discrete traits, using task-driven initialization, a Pareto-Ranking Genetic Algorithm, and local Bayesian optimization on a small calibration subset (Gordon et al., 13 Jul 2025, Guo et al., 2023, Zhou et al., 2 May 2025).

Calibration assumptions vary substantially. MiLo is explicitly calibration-free and fits compensators directly on the quantization residual of the weights. TwinQuant uses a small calibration set such as 128 WikiText2 prompts. QR-Adaptor reserves 256–512 examples as calibration data for its surrogate evaluation. A plausible implication is that calibration is not intrinsic to MLoRQ itself; rather, it is a design choice tied to the optimization criterion and deployment regime (Huang et al., 3 Apr 2025, Wang et al., 1 Jun 2026, Zhou et al., 2 May 2025).

4. Fine-tuning-oriented MLoRQ

The fine-tuning branch of MLoRQ modifies the relationship between quantization and adaptation. QA-LoRA is organized around the imbalance between the degrees of freedom of quantization and those of LoRA. Its group-wise quantization and shared adapter structure allow low-bit training and direct deployment of a merged quantized model, because after fine-tuning the adapters can be absorbed into the quantization zero-point $y=\tilde W^\top x + s(AB)^\top x$8. In the reported experiments on LLaMA and LLaMA2, QA-LoRA in INT4 matches or outperforms QLoRA by $y=\tilde W^\top x + s(AB)^\top x$9–$\hat A = U\hat C_H + V\hat C_L$0 in 5-shot MMLU, remains stable down to 2 bits, and avoids the post-training quantization drop observed for QLoRA after PTQ (Xu et al., 2023).

Integer-aware merging is pushed further by IntLoRA for diffusion models. IntLoRA quantizes the adaptation weights themselves so that the effective weight is formed as

$\hat A = U\hat C_H + V\hat C_L$1

and at inference the downstream weights are obtained without any PTQ stage. In the reported storage analysis under 4-bit quantization for SDXL, per-task storage decreases from $\hat A = U\hat C_H + V\hat C_L$2 to $\hat A = U\hat C_H + V\hat C_L$3, and the merge operator becomes INT multiplication or bit-shift rather than FP addition (Guo et al., 2024).

Adapter-centric quantization is also possible. LoRAQuant first reparameterizes a LoRA update $\hat A = U\hat C_H + V\hat C_L$4 by SVD, splits it into a high-precision and a low-precision sub-LoRA according to a variance threshold $\hat A = U\hat C_H + V\hat C_L$5, quantizes the important part by RTN and the less important part by binary sign quantization, and refines each singular pair using a straight-through estimator. On LLaMA 2-7B, the reported [email protected] setting reaches an average performance of $\hat A = U\hat C_H + V\hat C_L$6 at $\hat A = U\hat C_H + V\hat C_L$7 bits, and [email protected] reaches $\hat A = U\hat C_H + V\hat C_L$8 at $\hat A = U\hat C_H + V\hat C_L$9 bits (Mirzaei et al., 30 Oct 2025).

In medical segmentation, AdaLoRA-QAT couples adaptive low-rank encoder adaptation with full quantization-aware training. The rank mask is selected by importance scores $X=X_{\mathrm{Body}}+X_{\mathrm{Aux}}$0, then frozen before stage-2 QAT. On pooled chest X-ray test data, AdaLoRA-only reaches $X=X_{\mathrm{Body}}+X_{\mathrm{Aux}}$1 Dice, and AdaLoRA plus full QAT reaches $X=X_{\mathrm{Body}}+X_{\mathrm{Aux}}$2 Dice with $X=X_{\mathrm{Body}}+X_{\mathrm{Aux}}$3 trainable-parameter reduction and $X=X_{\mathrm{Body}}+X_{\mathrm{Aux}}$4 compression; a Wilcoxon signed-rank test yields $X=X_{\mathrm{Body}}+X_{\mathrm{Aux}}$5 across Dice, IOU, and NSD (Deb et al., 1 Apr 2026).

5. Systems, kernels, and hardware realization

A persistent systems concern is that a low-rank branch can erase the practical gains of low-bit quantization unless the computation is fused or scheduled carefully. TwinQuant addresses this with a fused dual-component GPU kernel. Its low-rank path is a two-stage W4A4 computation with on-chip requantization of the intermediate tensor, and the residual path is another W4A4 GEMM; both are merged in a single epilogue, eliminating two extra global-memory writes and reads. Across LLaMA3 and Qwen3, the reported end-to-end speedup is $X=X_{\mathrm{Body}}+X_{\mathrm{Aux}}$6–$X=X_{\mathrm{Body}}+X_{\mathrm{Aux}}$7 over an FP16 baseline, while the memory and latency overhead of the low-rank path are only $X=X_{\mathrm{Body}}+X_{\mathrm{Aux}}$8–$X=X_{\mathrm{Body}}+X_{\mathrm{Aux}}$9 and $\min_{Q,L,R}\ \|(W-Q-LR)X\|_F^2$0–$\min_{Q,L,R}\ \|(W-Q-LR)X\|_F^2$1 at rank 128 (Wang et al., 1 Jun 2026).

MiLo develops hardware support for extreme quantization in sparse Mixture-of-Experts models. Its INT3 design includes zero-bit-waste 3-bit packing, Tensor Core–friendly INT3-to-FP16 dequantization, asynchronous cuda::memcpy_async, and MoE-specific tile shapes. On Mixtral-8×7B, the reported latency is $\min_{Q,L,R}\ \|(W-Q-LR)X\|_F^2$2 versus MARLIN’s $\min_{Q,L,R}\ \|(W-Q-LR)X\|_F^2$3 at batch 1 and $\min_{Q,L,R}\ \|(W-Q-LR)X\|_F^2$4 versus $\min_{Q,L,R}\ \|(W-Q-LR)X\|_F^2$5 at batch 16, corresponding to $\min_{Q,L,R}\ \|(W-Q-LR)X\|_F^2$6 and $\min_{Q,L,R}\ \|(W-Q-LR)X\|_F^2$7 speedups (Huang et al., 3 Apr 2025).

LoPRo targets fine-tuning-free PTQ in the sub-3-bit regime. It uses a rank-1 sketch–based decomposition, partial block-wise Walsh–Hadamard rotation after column permutation by importance, and scalar or vector quantization of the transformed residual. The reported quantization times are under 30 minutes for 7B, approximately 45 minutes for 13B, and 2 hours 30 minutes for a 56B MoE on A100/80G; inference overhead is below $\min_{Q,L,R}\ \|(W-Q-LR)X\|_F^2$8 latency penalty, with end-to-end speedups up to $\min_{Q,L,R}\ \|(W-Q-LR)X\|_F^2$9 versus FP16 (Gu et al., 27 Jan 2026).

Regularity of the compute pattern is another design axis. LQER reconstructs quantization error with a low-rank correction using two blocked matrix multiplications, avoiding the scatter/gather behavior of outlier-based methods. The reported W4A8 system uses $$\min_{Q\in\mathcal A^{N\times N'},\,L,R}\ \|XW-X(Q+LR)\|_F^2$$0 fewer hardware resources than the leading state-of-the-art method, and its regular pattern occupies $$\min_{Q\in\mathcal A^{N\times N'},\,L,R}\ \|XW-X(Q+LR)\|_F^2$$1 the circuit area of OmniQuant’s 8×8 reorder plus token-scale pipeline (Zhang et al., 2024).

At the accelerator level, SeVeDo separates a high-precision Low-rank Vector Core from a low-bit Residual Matrix Core. Hierarchical Group Quantization reduces FP16–INT accumulation count by $$\min_{Q\in\mathcal A^{N\times N'},\,L,R}\ \|XW-X(Q+LR)\|_F^2$$2, saving $$\min_{Q\in\mathcal A^{N\times N'},\,L,R}\ \|XW-X(Q+LR)\|_F^2$$3 energy and $$\min_{Q\in\mathcal A^{N\times N'},\,L,R}\ \|XW-X(Q+LR)\|_F^2$$4 area, while SVD-guided mixed precision eliminates a separate FP16 datapath and saves up to $$\min_{Q\in\mathcal A^{N\times N'},\,L,R}\ \|XW-X(Q+LR)\|_F^2$$5 energy and $$\min_{Q\in\mathcal A^{N\times N'},\,L,R}\ \|XW-X(Q+LR)\|_F^2$$6 area versus an FP16 baseline. In 28 nm at 250 MHz, the reported peak efficiency is $$\min_{Q\in\mathcal A^{N\times N'},\,L,R}\ \|XW-X(Q+LR)\|_F^2$$7, with $$\min_{Q\in\mathcal A^{N\times N'},\,L,R}\ \|XW-X(Q+LR)\|_F^2$$8 on ViT-Base and $$\min_{Q\in\mathcal A^{N\times N'},\,L,R}\ \|XW-X(Q+LR)\|_F^2$$9 on Llama2-7B (Choi et al., 15 Dec 2025).

6. Empirical behavior, trade-offs, and limitations

Empirical outcomes across domains show that MLoRQ is most valuable in regimes where uniform low-bit quantization is too coarse and pure low-rank compression is too expensive. In the dedicated transformer-compression formulation, the two-stage Pareto-plus-ILP pipeline combined with LoRAda adaptive rounding yields up to $S_i$0 performance improvement on Vision Transformers; at a 9.3% weight-memory point, DeiT-B reaches $S_i$1 versus ERQ at $S_i$2 and GPTQ at $S_i$3, while on GLUE with BERT-base and 16% weight size the reported average is $S_i$4 versus TFWSVD at $S_i$5 (Gordon et al., 13 Jul 2025).

For LLMs, representative PTQ results include ResQ, which retains a rank-$S_i$6 high-precision PCA subspace and quantizes the rest more aggressively. Its reported gains include up to $S_i$7 lower perplexity on Wikitext than SpinQuant and up to $S_i$8 speedup over a 16-bit baseline. TwinQuant reports W4A4 zero-shot accuracy within $S_i$9–$\sum_i \mathrm{WeightsMemory}(S_i)\le \psi_{WMS}$0 points of FP16 across models from 3B to 32B. GPTQ-intrinsic LoRA complements these empirical results with lower bounds and a near-optimal algorithmic analysis, showing layer-wise error bounds in which the usual GPTQ dependence on $\sum_i \mathrm{WeightsMemory}(S_i)\le \psi_{WMS}$1 is replaced by the rank-$\sum_i \mathrm{WeightsMemory}(S_i)\le \psi_{WMS}$2 residual $\sum_i \mathrm{WeightsMemory}(S_i)\le \psi_{WMS}$3 (Saxena et al., 2024, Wang et al., 1 Jun 2026, Zhang et al., 31 May 2026).

In sparse MoE inference, MiLo reports that low-rank compensators recover more than $\sum_i \mathrm{WeightsMemory}(S_i)\le \psi_{WMS}$4 of the accuracy lost to extreme quantization, with only a few percent memory overhead and up to $\sum_i \mathrm{WeightsMemory}(S_i)\le \psi_{WMS}$5 end-to-end inference speedups. In diffusion models, MPQ-DMv2 integrates a residual mixed quantizer, low-rank initialization, and temporal distillation; on LDM-4 with W2A4 it improves FID from $\sum_i \mathrm{WeightsMemory}(S_i)\le \psi_{WMS}$6 to $\sum_i \mathrm{WeightsMemory}(S_i)\le \psi_{WMS}$7, reaches $\sum_i \mathrm{WeightsMemory}(S_i)\le \psi_{WMS}$8 inference speedup, and reduces bit operations by $\sum_i \mathrm{WeightsMemory}(S_i)\le \psi_{WMS}$9 relative to the FP baseline (Huang et al., 3 Apr 2025, Feng et al., 6 Jul 2025).

The trade-offs are correspondingly explicit. Smaller group sizes in QA-LoRA improve quantization–adaptation balance, with group size 32 reported as a sweet spot. Higher rank improves reconstruction but increases memory overhead linearly in MiLo and average bit cost in ODLRI-style decompositions. TwinQuant still requires an offline calibration stage of approximately 10–30 minutes. QR-Adaptor’s three-stage search requires a few hours per model-task pair. SeVeDo’s static SVD decomposition may need re-tuning if activation distributions shift at deployment. ResQ assumes that post-rotation coefficients are well modeled as Gaussian and notes that domain shifts may alter the optimal subspace (Xu et al., 2023, Huang et al., 3 Apr 2025, Zhou et al., 2 May 2025, Choi et al., 15 Dec 2025, Saxena et al., 2024).

A common misconception is that the low-rank branch is always a high-precision escape hatch. The literature shows several alternatives: TwinQuant quantizes both the low-rank and residual branches to 4 bits; QA-LoRA merges the adapter into quantization parameters; LoRAQuant binarizes the less important sub-LoRA; and MUXQ keeps both Body and Aux paths in INT8. Another misconception is that MLoRQ is confined to transformer weights. The same mixed principle appears in activation decompositions, diffusion-model denoising trajectories, light-field NeRF compression, quantized low-rank regression, and low-rank matrix recovery from quantized measurements (Wang et al., 1 Jun 2026, Xu et al., 2023, Mirzaei et al., 30 Oct 2025, Lee et al., 6 Apr 2026, Shi et al., 2022, Chen et al., 2023, Lybrand et al., 2017).

Taken together, the literature presents MLoRQ as a general design principle rather than a fixed recipe: quantization handles bulk storage and throughput, while low-rank structure is used to preserve the directions that are disproportionately costly to quantize. The precise implementation—alternating decomposition, discrete search, calibration-free compensation, QAT, fused kernels, or accelerator partitioning—depends on which object is being compressed and which failure mode of low-bit representation is being targeted.