3BASiL-TM: Sparse + Low-Rank LLM Compression

• 3BASiL-TM is a framework for one-shot post-training compression of LLMs that uses a sparse plus low-rank decomposition to balance hardware efficiency and model fidelity.
• It employs a three-block ADMM algorithm for layer-wise reconstruction, ensuring convergence of sparse and low-rank components through adaptive penalty scheduling.
• A transformer matching refinement stage aligns block-level outputs, reducing perplexity gaps and outperforming pure sparsity methods on benchmark evaluations.

Searching arXiv for the cited 3BASiL-TM paper and closely related context. {"query":"(Makni et al., 2 Mar 2026) 3BASiL-TM sparse low-rank compression LLMs"} {"query":"3BASiL An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs"} 3BASiL-TM is a two-stage framework for one-shot post-training compression of LLMs by means of a sparse plus low-rank decomposition, written as $$\widehat{\mathbf{W}} \approx \mathbf{S} + \mathbf{L}$$ for each pretrained dense weight matrix. In this formulation, $\mathbf{S}$ is a sparse backbone and $\mathbf{L}$ is a low-rank correction. The framework combines 3BASiL, a 3-block ADMM procedure for layer-wise reconstruction with convergence guarantees, with Transformer-Matching (TM), a transformer-block-level refinement that jointly adjusts sparse and low-rank components across layers to align block outputs. The method is positioned within the literature on sparse plus low-rank compression of LLMs, where it is presented as an efficient alternative to prior post-training schemes that incur substantial performance degradation relative to dense models (Makni et al., 2 Mar 2026).

1. Problem setting and compression model

3BASiL-TM addresses post-training compression of LLMs using a small calibration set rather than full retraining. For a given layer, the starting point is a pretrained dense weight $$\widehat{\mathbf{W}} \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$$ and a matrix of calibration activations $\mathbf{X} \in \mathbb{R}^{N \times d_{\text{in}}}$. The compressed weight is represented as

$\widetilde{\mathbf{W}} = \mathbf{S} + \mathbf{L},$

with $\mathbf{S}\in\mathcal{S}$ and $\operatorname{rank}(\mathbf{L}) \le r$ (Makni et al., 2 Mar 2026).

The sparsity set $\mathcal{S}$ may encode unstructured sparsity at a target density or N:M semi-structured sparsity. The configurations explicitly discussed include 2:4, 3:8, and 4:8, meaning that in every group of $M$ consecutive weights only $\mathbf{S}$0 are non-zero. The low-rank component is implemented as a matrix of rank at most $\mathbf{S}$1, typically through factors such as $\mathbf{S}$2.

The rationale for the sparse plus low-rank decomposition is comparative. Pure sparsity is advantageous for memory reduction and, with N:M structure, for GPU or CPU acceleration, but high sparsity can cause large accuracy drops. Pure low rank reduces parameters, but rank often must be relatively large to approximate a full dense matrix well. The combined form uses the sparse term as a backbone structure that hardware kernels exploit, while the low-rank term captures residual fine-grained corrections and acts as a LoRA-like adapter. The same low-rank term can also serve as smart LoRA initialization for later fine-tuning (Makni et al., 2 Mar 2026).

This design implies a division of labor between the two components. A plausible implication is that the sparse backbone is intended to absorb the dominant weight structure under hardware-friendly constraints, while the low-rank term recovers approximation fidelity that would otherwise be lost under aggressive sparsification.

2. 3BASiL: three-block ADMM for layer-wise reconstruction

At the layer level, 3BASiL solves a reconstruction problem on calibration activations. The objective combines output matching on calibration data with an explicit proximity penalty to the original dense weights; the latter uses a scalar $\mathbf{S}$3 and ensures that $\mathbf{S}$4 is full-rank (Makni et al., 2 Mar 2026).

To handle sparsity cleanly, the method introduces an auxiliary copy $\mathbf{S}$5 of the sparse variable and enforces $\mathbf{S}$6 through ADMM. The algorithm alternates over three primal blocks and one dual block:

Block	Role	Update form
$\mathbf{S}$7	Dense sparse weights, unconstrained	Closed-form quadratic solve
$\mathbf{S}$8	Low-rank matrix	Best rank-$\mathbf{S}$9 weighted approximation
$\mathbf{L}$0	Sparsity-projected copy	Projection onto $\mathbf{L}$1

The $\mathbf{L}$2-update is a ridge-like least-squares step:

$\mathbf{L}$3

The method precomputes an eigendecomposition of $\mathbf{L}$4 so that $\mathbf{L}$5 can be applied cheaply for multiple values of $\mathbf{L}$6 (Makni et al., 2 Mar 2026).

The $\mathbf{L}$7-update is a reduced-rank regression with closed form:

$\mathbf{L}$8

where $\mathbf{L}$9 denotes the best rank-$$\widehat{\mathbf{W}} \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$$0 approximation via SVD. In implementation, randomized SVD is used for efficiency.

The $$\widehat{\mathbf{W}} \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$$1-update is a projection onto the sparsity set:

$$\widehat{\mathbf{W}} \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$$2

and the dual variable is updated by

$$\widehat{\mathbf{W}} \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$$3

For unstructured sparsity, projection keeps the largest entries by magnitude; for N:M sparsity, it keeps the top $$\widehat{\mathbf{W}} \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$$4 within each group of $$\widehat{\mathbf{W}} \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$$5 consecutive weights.

A distinctive feature is the increasing penalty schedule $$\widehat{\mathbf{W}} \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$$6. The procedure starts from $$\widehat{\mathbf{W}} \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$$7 and updates every 10 iterations according to the support difference

$$\widehat{\mathbf{W}} \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$$8

The schedule gradually increases $$\widehat{\mathbf{W}} \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$$9 to lock in the sparsity pattern while preserving reconstruction quality (Makni et al., 2 Mar 2026).

The convergence statement is formulated for the iterates rather than global optimality. Under non-decreasing $\mathbf{X} \in \mathbb{R}^{N \times d_{\text{in}}}$0 with $\mathbf{X} \in \mathbb{R}^{N \times d_{\text{in}}}$1, the method proves that both $\mathbf{X} \in \mathbb{R}^{N \times d_{\text{in}}}$2 and $\mathbf{X} \in \mathbb{R}^{N \times d_{\text{in}}}$3 form Cauchy sequences and that $\mathbf{X} \in \mathbb{R}^{N \times d_{\text{in}}}$4 converges to a limiting decomposition. Because the underlying problem is non-convex due to the rank and sparsity constraints, the guarantee concerns convergence of iterates, not convergence to a globally optimal solution (Makni et al., 2 Mar 2026).

3. Transformer-Matching and block-level refinement

3BASiL is layer-wise, whereas TM operates at the level of the full transformer block. After an initial sparse plus low-rank decomposition has been obtained for each linear layer in a block, TM refines the compressed block by minimizing the discrepancy between the outputs of the original dense block and those of the compressed block on calibration inputs (Makni et al., 2 Mar 2026).

For block $\mathbf{X} \in \mathbb{R}^{N \times d_{\text{in}}}$5, the refinement objective is

$\mathbf{X} \in \mathbb{R}^{N \times d_{\text{in}}}$6

where $\mathbf{X} \in \mathbb{R}^{N \times d_{\text{in}}}$7 is frozen and $\mathbf{X} \in \mathbb{R}^{N \times d_{\text{in}}}$8 uses the compressed weights $\mathbf{X} \in \mathbb{R}^{N \times d_{\text{in}}}$9. The support $\widetilde{\mathbf{W}} = \mathbf{S} + \mathbf{L},$0 found during layer-wise compression is held fixed, and only the values on that support and the low-rank factors are optimized. The ranks $\widetilde{\mathbf{W}} = \mathbf{S} + \mathbf{L},$1 are also fixed (Makni et al., 2 Mar 2026).

The refinement is described as memory-efficient because it proceeds block by block rather than through the full LLM. For each block, only the subgraph of that block participates in forward and backward passes. The procedure reuses the same small calibration set—128 sequences of length 2048—in mini-batches such as batch size 8. Optimization uses Adam with default PyTorch hyperparameters, a learning rate of $\widetilde{\mathbf{W}} = \mathbf{S} + \mathbf{L},$2, cosine annealing with $\widetilde{\mathbf{W}} = \mathbf{S} + \mathbf{L},$3, and approximately 20 epochs over the calibration sequences (Makni et al., 2 Mar 2026).

TM is characterized as universal. It does not depend on how the initial decomposition was obtained, only on having a compressed model, a support mask, and a rank specification. This means it can refine 3BASiL itself, other sparse plus low-rank methods such as OATS and HASSLE-Free variants, and pure pruning methods such as WandA, SparseGPT, and ALPS. In the purely sparse case, $\widetilde{\mathbf{W}} = \mathbf{S} + \mathbf{L},$4 and TM updates only the sparse weights on fixed supports (Makni et al., 2 Mar 2026).

This universality is significant because it separates two issues that are often conflated in compression pipelines: the initial constrained approximation of weights and the later alignment of block-level hidden-state dynamics. TM addresses the latter directly.

4. Empirical results and operating characteristics

The reported experiments cover Meta-Llama-3-8B, Llama-3.2-1B, Llama-3.2-3B, and OPT-30B. Calibration uses 128 sequences from C4 with 2048 tokens each. Evaluation includes perplexity on C4, WikiText-2, and Penn Treebank, together with zero-shot tasks via LM-Harness on PIQA, HellaSwag, ARC-Easy, ARC-Challenge, Winogrande, RTE, OpenBookQA, and BoolQ. Runs use a single GPU per configuration, specifically A100 80GB or L40 48GB (Makni et al., 2 Mar 2026).

For one-shot sparse plus low-rank compression without TM, the method already improves on prior decompositions in the examples reported. In Llama-3.2-3B under a 2:4+64 LR configuration, the dense model has WikiText-2 perplexity 7.81, Hf-ALPS has 13.79, and 3BASiL has 13.12. Adding TM improves the same configuration to 11.79 (Makni et al., 2 Mar 2026).

The paper emphasizes the relative reduction of the perplexity gap to the dense baseline. For Llama-3-8B with 2:4+64 LR, the dense model has WikiText-2 perplexity 6.14, Hf-ALPS has 11.38, and 3BASiL-TM has 9.78. The corresponding gaps to the dense model are approximately 5.24 and 3.64, which is presented as a reduction of the WikiText2 perplexity gap by over 30% relative to prior methods under that configuration (Makni et al., 2 Mar 2026).

At a fixed compression ratio $\widetilde{\mathbf{W}} = \mathbf{S} + \mathbf{L},$5 on Llama-3-8B, the strongest pure pruning baseline cited is ALPS-TM with C4 14.96, WikiText-2 10.65, and PTB 16.35. Under the same compression ratio, 3BASiL-TM with 3:8+LR reports C4 13.01, WikiText-2 8.69, and PTB 13.74. This is used to support the claim that sparse plus low-rank structure, combined with TM, outperforms pure sparsity at matched density (Makni et al., 2 Mar 2026).

Runtime is another recurrent comparison point. The abstract states that the method achieves over 2.5x faster compression runtime on an A100 GPU compared to a state-of-the-art sparse plus low-rank method. The reported explanation is algorithmic: closed-form low-rank updates replace gradient descent on factorized low-rank terms, the sparse update benefits from precomputed spectral structure, and fewer iterations are needed because of improved convergence behavior (Makni et al., 2 Mar 2026).

The same study also considers LoRA fine-tuning after decomposition. For Llama-3-8B, after 10% C4 fine-tuning with LoRA, all methods improve, but 3BASiL-TM is reported to retain its lead. This is used to support the interpretation of sparse plus low-rank decomposition as LoRA-aware initialization rather than solely as a terminal compressed representation (Makni et al., 2 Mar 2026).

5. Practical use, parameter choices, and limitations

The method is intended for one-shot post-training compression when good perplexity or zero-shot performance is required together with hardware-relevant sparsity. The paper specifically recommends it when a calibration pass is acceptable, moderate ADMM iterations per layer are feasible, and a short TM phase per transformer block can be afforded. It also notes that TM can be used directly with pure pruning when the goal is to improve a fixed-support sparse model rather than to introduce a low-rank component (Makni et al., 2 Mar 2026).

Parameter choices in the experiments are concrete. For mid-size LLaMA models, the reported sweet spot is often rank $\widetilde{\mathbf{W}} = \mathbf{S} + \mathbf{L},$6 combined with 2:4 or 3:8 sparsity; for OPT-30B, a 2:4+112 setting is used. The paper describes 2:4 as more aggressive and more acceleration-oriented, with greater accuracy loss, whereas 3:8 and 4:8 are milder and preserve accuracy better. TM is run for 20 epochs over the 128 calibration sequences with batch size 8, while 3BASiL itself uses tens of ADMM iterations per layer with an adaptive $\widetilde{\mathbf{W}} = \mathbf{S} + \mathbf{L},$7 schedule (Makni et al., 2 Mar 2026).

Several limitations are explicit. At extreme compression, dense-model performance may not be recoverable even with TM. TM is still a gradient-based procedure, so it adds non-trivial compute on top of the one-shot decomposition step. The work targets floating-point weights and leaves joint quantization plus sparse plus low-rank compression to future work. It also compresses standard transformer layers and has not been tested in the paper on mixture-of-experts or other non-standard architectures (Makni et al., 2 Mar 2026).

A common misconception is to treat TM as a substitute for the layer-wise algorithm. The reported framework does not make that claim. TM is a refinement stage layered on top of an existing decomposition, and the full designation “3BASiL-TM” refers specifically to the combination of 3BASiL and TM rather than to TM in isolation.

6. Terminological scope and disambiguation

The name 3BASiL-TM belongs to LLM compression and should be distinguished from other recent uses of similar terminology. “BASIL” in "Pathway-based Bayesian factor models for gene expression data" denotes Bayesian Analysis with gene-Sets Informed Latent space, a Bayesian factor modeling framework for RNA-sequencing data that incorporates pathway annotations into latent variable analysis (Mauri et al., 19 Jan 2026). “BAS” in "On Bismut--Ambrose--Singer manifolds" refers to Hermitian manifolds whose Bismut connection has parallel torsion and parallel curvature (Barbaro et al., 4 May 2026). Neither is related to sparse plus low-rank compression of transformers.

Within the LLM setting, 3BASiL-TM denotes a specific architectural and algorithmic pairing: a 3-block ADMM method for constrained layer-wise sparse plus low-rank decomposition, followed by a memory-efficient transformer-block-level output-matching refinement (Makni et al., 2 Mar 2026). This distinction matters because the surface similarity of the acronyms can obscure the fact that the three lines of work belong to different research areas—LLM systems, statistical genomics, and complex differential geometry, respectively.