Sherry Framework for Efficient LLM Quantization

• Sherry Framework is a hardware-efficient quantization method that uses a distinctive 3:4 sparsification and 1.25-bit encoding to optimize LLM deployment on constrained hardware.
• Its 4-to-5 bit encoding scheme aligns with commodity hardware vector lanes, enabling efficient SIMD processing and eliminating costly bit manipulations.
• The Arenas annealing residual synapse mitigates representational collapse during quantization-aware training, preserving model accuracy while reducing memory footprint.

The Sherry framework is a hardware-efficient ternary quantization method targeting the deployment of LLMs on edge devices with stringent memory and computational constraints. Sherry employs a unique 3:4 fine-grained sparsification—packing four weights into five bits—yielding a regularized 1.25-bit wide encoding that closely aligns with commodity hardware vector widths. This approach addresses inefficiencies in conventional ternary quantization schemes, which are typically misaligned with standard hardware and thus suffer from wasted storage or degraded inference speed. Through its Arenas annealing residual synapse mechanism, Sherry also resolves the problem of representational collapse during sparse ternary quantization-aware training (QAT), ensuring both efficiency and accuracy in practical deployment scenarios (Huang et al., 12 Jan 2026).

1. Three-of-Four Fine-Grained Sparsity and 1.25-Bit Quantization

Sherry introduces an $N:M$ sparsity regime with $N=3$ nonzero elements in each $M=4$ consecutive weight block. Each 4-weight group is encoded such that exactly three positions are ternary ($\pm1$) and one is zero. This design supports the following properties:

• Pattern cardinality: There are $\binom{4}{3} \times 2^3 = 32$ valid ternary patterns for a 4-weight block, i.e., four ways to choose a zero placement and $2^3$ choices for the signs.
• Bit-width per weight: Packing these 32 patterns requires 5 bits per 4 weights, yielding an effective $\frac{5}{4} = 1.25$ bits per weight.
• Sparse-ternary reconstruction objective: For each output channel $j$ and block $b$,

$$\min_{T, \alpha} \sum_{j=1}^{d_\mathrm{out}} \| W_{:,j} - T_{:,j} \,\alpha_j \|_2^2$$

subject to $T_{i,j}\in\{-1,0,+1\}$ and $\|T_{b{:}b+3,j}\|_0=3$ for all relevant $b$.

• Closed-form block solution:

$$\begin{align*} T^*_{i,j} &= \begin{cases} 0 & i = \arg\min_{k\in\{b,\ldots,b+3\}} |W_{k,j}| \ \operatorname{sign}(W_{i,j}) & \text{otherwise}, \end{cases} \end{align*}$$

with shared scaling

$$\alpha_j^* = \frac{1}{3}\sum_{i\in\mathcal S_j}|W_{i,j}| \,.$$

This precise regularization both minimizes bit waste and enables high model compression while maintaining representation richness throughout the quantization process.

2. Bit-Aligned Weight Packing: 4-to-5 Bit Encoding Scheme

The 3:4 pattern is realized using a packing scheme that optimally exploits hardware vector widths:

• Per-block encoding: Each 4-weight block is mapped into a single 5-bit codeword: 1 bit for shared sign $s\in\{-1,+1\}$, and 4 bits for an index $I\in\{0,\dots,15\}$ that encodes the magnitude/sparsity configuration.
• Exploiting symmetry: The 32 patterns can be paired by sign, so indexing the unsigned pattern and storing just one sign bit suffices.
• Packing structure: Bits are organized as $[s|I]$, with 3 unused bits to align to the byte boundary.
• SIMD decoding: On hardware with AVX2 (Intel/AMD) or Neon (ARM), 4-bit indices can address a 16-byte LUT (e.g., via vpshufb), yielding fast per-block accumulations; a subsequent sign correction applies as needed.

Maintaining power-of-two block stride restores alignment lost in prior ternary designs (e.g., 1.67-bit irregular packing), avoiding costly bit manipulations and making the approach amenable to vectorized inference on standard edge-device CPUs.

3. Weight-Trapping and Representational Collapse in Sparse Ternary QAT

Sparse ternary quantization introduces a notable pathology: weight trapping. During training with hard 3:4 block constraints, the gradients driving the zeroed positions tend to decay, while surviving nonzeros push toward $\pm1$. This collapse is analytically characterized as follows:

• Low-rank gradient effect: With weight matrix $T\alpha$ sharing a blockwise uniform zero position, the loss gradient with respect to inputs,

$$\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y}(T\alpha)^\mathrm{T},$$

becomes low-rank—specifically, blockwise Hadamard-like—which sharply diminishes Effective Rank (ER) and reduces the diversity of representational updates.

• Empirical outcome: The model’s parameters collapse to a nearly binary regime, resulting in sub-optimal accuracy compared to full-precision or higher-entropy ternary models.

The necessity to mitigate such collapse is central to maintaining model expressivity under aggressive quantization.

4. Arenas: Annealing Residual Synapse for Trapping Avoidance

To resolve representational collapse, Sherry introduces Arenas, an annealing residual synapse. This mechanism temporarily augments the quantized model with a decaying full-precision bypass during QAT:

• Training forward pass: For layer input $X$ and quantized weights $T,\alpha$, with full-precision latent weights $W$ and annealing factor $\lambda_t$ at training step $t$,

$Y = X\,T\,\alpha + \lambda_t X W$

• Decay schedule: $\lambda_t \to 0$ via a cosine decay (with optional linear warmup),

$$\lambda_t = \begin{cases} p_t / p_\mathrm{warmup}, & p_t < p_\mathrm{warmup} \ \frac{1}{2}\left[1+\cos\left(\pi \frac{p_t-p_\mathrm{warmup}}{1-p_\mathrm{warmup}}\right)\right], & p_t \ge p_\mathrm{warmup} \end{cases}$$

where $p_t = \mathrm{step}/\mathrm{total\_steps}$.

• Backward pass: The joint pathway

$$\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y}(T\alpha + \lambda_t W)^\mathrm{T}$$

sustains high-rank gradients until annealing concludes.

• Training pseudocode:

for t in 1…T: p = t / T λ = compute_lambda(p) W_fp = latent_fp_weights T, α = ternary_quantize(W_fp) # 3:4 mask + STE Y_pred = X @ (T*α) + λ*(X @ W_fp) loss = criterion(Y_pred, target) loss.backward() optimizer.step() # At end: keep only T, α for inference

Arenas prevents premature trapping and allows weights to explore a richer solution space during QAT, ultimately yielding better quantized models.

5. Empirical Performance on LLaMA-3.2 Benchmarks

Sherry’s efficacy is validated on LLaMA-3.2 models across five standard zero-shot evaluation tasks (ARC-Easy, ARC-Challenge, HellaSwag, PIQA, WinoGrande):

Size	Method	Bits	Average Accuracy
1B	TequilaLLM	1.67	0.519
1B	Sherry	1.25	0.519
3B	TequilaLLM	1.67	0.576
3B	Sherry	1.25	0.567

• Storage and bit-rate advantages: Sherry reduces model size by approximately $25\%$ compared to 1.67-bit baselines.
• Inference speed: On an Intel i7-14700HX CPU, Sherry consistently outpaces 1.67-bit models (e.g., 1.25-bit Sherry at 148.27 tokens/sec vs. 116.83 for 1.67-bit on 0.7B LLaMA-3.2), corresponding to $\sim$10–18% speedup.
• Zero accuracy loss: For the 1B LLaMA-3.2 model on the referenced benchmarks, Sherry matches the baseline’s average accuracy at a reduced bit-width.

Scale	Method	Bits	Speed (t/s)	Size (MB)
0.7B	2-bit	2.0	132.13	256.6
0.7B	1.67-bit	1.67	116.83	233.4
0.7B	Sherry	1.25	148.27	205.5
3B	2-bit	2.0	41.87	873.7
3B	1.67-bit	1.67	38.80	846.0
3B	Sherry	1.25	45.55	712.4

The results establish that Sherry yields state-of-the-art ternary compression and efficiency with statistically indistinguishable accuracy from denser baselines (Huang et al., 12 Jan 2026).

6. Design, Implementation, and Deployment Considerations

Sherry is tailored for practical edge and mobile deployment:

• Inference kernels: Execution uses only integer additions and LUT lookups; all floating-point operations are eliminated post-training.
• Vectorization: Packing and unpacking is synchronized with power-of-two (4 weights per group) vector lanes, facilitating hardware-accelerated operations and avoiding expensive bit-shuffles.
• SIMD compatibility: Directly compatible with common CPU SIMD instructions (x86 AVX2 vpshufb, ARM Neon).
• Training overhead: Annealing residual synapse and latent FP32 weights are only instantiated during QAT. All auxiliary parameters are discarded for inference.
• Ecosystem integration: The framework is open-sourced in the BitNet.cpp ecosystem, enabling direct integration for edge and mobile computation scenarios.

This framework demonstrates that hardware-conscious quantization schemes, informed by the constraints and capabilities of real-world inference targets, can yield substantial improvements in memory and compute efficiency without loss of functional performance (Huang et al., 12 Jan 2026).