Breaking the 1.58-bit Barrier for Ternary LLMs
Abstract: Ternary LLMs (LLM) store every weight as one of three symbols , so the cost of a ternary model is conventionally referenced to the information-theoretic bits per weight. The prevailing deployment format packs five ternary weights into one byte (five-trit packing), and due to the power-of-two group sizes used in practice this rounds up to $1.625$ bits per weight. This effective storage bit-width treats the three symbols as equiprobable. We measure the actual symbol distribution of 29 ternary LLM models and find that zeros account for up to of all weights. Motivated by this finding, we introduce BITCOS, a simple distribution-adaptive layout comprised of a dense presence bitmap plus a compacted sign vector, and costs $2 - z$ bits per weight element given a zero density in the model's weights. BITCOS stores weights more compactly than the five-trit packing in 26 of the 29 tested models, and reaches $1.485$ bits per weight on the sparsest of them. BITCOS is amenable to efficient unpacking on modern processors and GPUs, and we present optimized unpacking sequences for AVX-512, AVX2 and Intel Xe2 GPUs. Measured against production state-of-the-art ternary matrix-vector multiplication kernels, at the zero densities real-world ternary models exhibit, the realized gain with our proposed layout is up to . Finally, we illustrate end-to-end LLM inference results on 5 different platforms (client and server CPUs, integrated and discrete Xe2 GPUs) where decode throughput improves by up to on CPUs and on GPUs.
Summary
- The paper introduces BITCOS, a bit-exact bitmap and compacted-sign representation that uses 2−z bits per weight and beats five-trit packing when zero density exceeds 37.5%.
- The authors find zero densities of 29.7%–51.5% across 29 checkpoints, with 26 models surpassing the five-trit crossover and the sparsest reaching 1.485 bits per weight before metadata.
- The paper shows that compression improves batch-one decode only on memory-bound hardware, producing up to 1.18× CPU and 1.27× Xe2 GPU speedups while losing to simpler 2-bit decoding on instruction-bound Lunar Lake.
Problem formulation and principal claim
“Breaking the 1.58-bit Barrier for Ternary LLMs” (2609.16338) challenges the conventional interpretation of ternary storage efficiency. Ternary LLMs constrain each weight to , motivating the customary -bit reference. In deployed systems, however, the relevant rate is determined not only by the alphabet cardinality but also by the packing scheme and the empirical frequency of the three symbols. The paper argues that the widely used five-trit-per-byte representation is distribution-oblivious: it implicitly treats zero, positive, and negative codes as equally likely even though existing ternary checkpoints contain substantial zero mass.
The authors measure the symbol distributions of 29 ternary LLM checkpoints spanning seven model families. Zero density ranges from to , with the sparsest model containing more than half zero-valued weights. This observation supports the paper’s central claim: the effective storage rate of many deployed ternary models can be reduced below both the practical five-trit rate of $1.625$ bits per weight and, for sufficiently sparse models, the information-theoretic equiprobable reference of $1.585$ bits per weight without modifying the model or retraining it.
The proposed method, BITCOS, stores a dense presence bitmap and a compacted sign stream. For zero density , the bitmap contributes one bit per weight and the sign stream contributes one bit per nonzero weight. Its nominal symbol rate is therefore
BITCOS becomes more compact than five-trit packing when $2-z<1.625$, namely when . It is strictly more compact than ordinary 2-bit packing for every nonzero zero density. Across the evaluated checkpoints, 26 of 29 models satisfy the five-trit crossover condition, and the most zero-heavy model reaches 0 bits per weight before accounting for scale metadata.
Figure 1: BITCOS decreases linearly with zero density, while five-trit packing remains at a fixed practical rate of 1 bits per weight.
The result should be distinguished from a claim that ternary symbols contain less than 2 bits of information under an equiprobable source. BITCOS exploits a nonuniform empirical distribution, particularly the elevated probability of zero. Its rate is a storage-layout rate for the observed model distribution, not a contradiction of the entropy bound for uniformly distributed ternary symbols.
BITCOS representation and decoding semantics
BITCOS separates the ternary code into two positional components. The presence bitmap contains one bit for every tensor element and identifies whether the corresponding weight is zero. The compacted sign vector contains one bit only for each nonzero weight, in tensor order. A sign bit is associated with a bitmap position by its rank among the set bits preceding it. Consequently, decoding requires both the bitmap and a running rank into the sign stream.
The layout preserves the original ternary tensor exactly. It does not impose a structured sparsity pattern, alter the zero distribution, or require additional training. This distinguishes it from methods that enforce 3 structure during QAT, such as Sparse-BitNet (Zhang et al., 5 Mar 2026), or methods that deliberately constrain the zero rate and local pattern, such as Sherry (Huang et al., 12 Jan 2026). BITCOS instead exploits the unstructured zeros already generated by existing QAT and PTQ procedures, including BitNet (Ma et al., 2024), ParetoQ (Liu et al., 4 Feb 2025), Spectra/TriLM (Kaushal et al., 2024), CAT-Q (Wang et al., 25 Jun 2026), and related ternary model families.
The representation is illustrated by an 4 example. With 30 zero weights, the bitmap consumes 64 bits and the compacted sign stream consumes 34 bits, yielding 98 bits total, or 5 bits per weight. The bitmap remains dense and positionally aligned, avoiding the irregular index streams and gather-heavy access patterns associated with sparse CSC-like representations.
Figure 2: A dense presence bitmap and rank-ordered compact sign stream reconstruct the original ternary tensor while charging no sign bit to zero weights.
The principal implementation challenge is rank recovery. Given a bitmap word, the decoder must scatter the next 6 sign bits into the positions marked present. On x86, the authors use pdep, whose semantics exactly match this operation: source bits are deposited sequentially into the set-bit positions of a destination mask. This permits direct reconstruction of lane-aligned signs without scalar iteration over individual weights.
CPU decoding kernels
The AVX-512 implementation reconstructs scaled FP16 weights using mask registers. Group scales are stored as nonnegative FP16 values. A second vector is generated by setting the IEEE FP16 sign bit, thereby producing the exact negative of each scale. For each 32-weight block, the kernel loads the bitmap, extracts a sign window, applies pdep, and uses masked moves to select positive, negative, or zero values. The authors report a 17-instruction inner sequence, of which three instructions perform the core unpacking: one pdep and two masked moves. The remaining instructions handle bitmap and sign-stream addressing, prefetching, population counting, cursor updates, and the fused multiply-add.
The AVX2 implementation targets client CPUs without AVX-512 mask registers and FP16 arithmetic. It reconstructs ternary int8 values and contracts them with int8 activations through AVX-VNNI. Presence and negativity are materialized as byte masks. The ternary value is formed from the presence bit 7 and negative-sign bit 8 as 9. This implementation incurs a more substantial unpacking cost because AVX2 must expand bit predicates into byte masks using broadcasts, shuffles, compares, and Boolean operations.
This distinction is important for interpreting the performance results. BITCOS reduces memory traffic, but it also introduces population-count, variable-offset sign-stream accesses, pdep, and predicate reconstruction. Whether the smaller payload dominates depends on the balance between memory bandwidth and decode instruction throughput.
Xe2 GPU implementation
Xe2 GPUs lack a direct equivalent of pdep, so the GPU implementation uses a 256-entry lookup table resident in shared local memory. Each table key contains a four-bit presence nibble and a four-bit window from the compact sign stream. The table entry contains four FP16 values corresponding to the four rows represented by the nibble. A zero presence bit maps to 0; a present bit selects either 1 or 2 according to the rank-indexed sign window.
Figure 3: The Xe2 lookup table maps a four-bit presence nibble and four compact sign bits to four FP16 ternary values.
The four-row granularity is a hardware-aware compromise. An eight-row table would require 3 entries and would no longer be suitably compact for SLM. Four rows yield 4 entries, each containing four FP16 values, for a total table size of 2 KB. The granularity also matches Xe2’s VNNI2 and DPAS operand organization: a lookup produces two dwords per SIMD lane, corresponding to four reduction rows.
The GPU kernel maintains a per-column sign cursor. Because a four-row group may contain fewer than four nonzeros, the cursor advances by 5 rather than by a fixed amount. Consecutive four-bit sign windows may therefore overlap in their loaded representation even though the consumed sign bits are disjoint. This is a nontrivial aspect of the layout: the bitmap provides the local sparsity pattern, but the decoder must preserve a variable-rate stream position independently for each SIMD lane.
Figure 4: Xe2 maps output columns to SIMD lanes and uses contiguous bitmap loads together with independently advancing sign-stream cursors.
Figure 5: Successive four-row lookups consume compact sign bits according to the population count of each presence nibble.
The mapping is designed to maintain regularity where possible. Bitmap words for adjacent columns are contiguous and can be loaded cooperatively. Sign streams are column-specific and require gathered loads, but a single three-word gather supplies the sign windows for two consecutive 32-row blocks. The lookup then reconstructs the four-row groups in the exact VNNI2 order required by the contraction pipeline.
Roofline analysis and the conditions for benefit
The paper’s CPU roofline model is central to its interpretation of the results. For a 32-weight iteration, the BITCOS kernel reads approximately
- 4 bytes of bitmap data,
- 6 bytes of compact sign data,
- 0.5 bytes of amortized FP16 scale data.
The total is therefore 7 bytes per 32 weights. If 8 is the per-core memory bandwidth and 9 is the measured L1-resident execution cost of one iteration, the sustained rate is bounded by the larger of the memory-transfer time and instruction-execution time. Equivalently, the achievable effective bandwidth is the smaller of 0 and 1.
Figure 6: The BITCOS CPU roofline separates memory-bound platforms, where reduced payload produces speedup, from instruction-bound platforms, where unpack overhead dominates.
The model explains the platform-dependent results. Emerald Rapids is memory-bound across the relevant density range, and Arrow Lake performance cores are also predominantly memory-bound. Arrow Lake efficiency cores operate close to the knee. Lunar Lake, despite having substantial aggregate memory bandwidth, is instruction-bound because only eight cores share approximately 2 GB/s. The per-core bandwidth available on Lunar Lake exceeds what the BITCOS decoder can consume, so reducing the payload does not reduce execution time.
This is a significant qualification to the paper’s storage argument: a lower bit-width does not guarantee faster inference. In bandwidth-bound decode, the format’s smaller payload translates into lower latency. In instruction-bound decode, the additional unpacking work can make BITCOS slower than a simpler fixed-width 2-bit kernel.
Figure 7: CPU GEMV sweeps show that BITCOS benefits memory-bound Emerald Rapids and Arrow Lake, but loses to the 2-bit reference on instruction-bound Lunar Lake.
On Emerald Rapids, BITCOS outperforms the LIBXSMM 2-bit GEMV throughout the deployed density range 3 by 4--5. On Arrow Lake, the corresponding speedup is 6--7. At very high density, such as 8, the payload becomes so small that decoding itself becomes the bottleneck, confirming the roofline prediction. Lunar Lake provides the contrasting result: BITCOS never exceeds 9 GB/s effective bandwidth, compared with $1.625$0 GB/s for the 2-bit kernel, and is slower at every tested density.
End-to-end CPU inference
The authors integrate BITCOS and the LIBXSMM 2-bit baseline into the vLLM CPU backend and evaluate batch-one decode over seven ternary LLM checkpoints. On the 64-core Emerald Rapids system, BITCOS improves end-to-end decode throughput over the state-of-the-art 2-bit kernel by $1.625$1--$1.625$2. On the 24-core Arrow Lake system, improvements range from $1.625$3 to $1.625$4. These gains are smaller than the theoretical payload ratios because the full decode path includes non-GEMV work, scale handling, scheduling, and other runtime overheads.
BITCOS also substantially exceeds the five-trit implementation available in the evaluated llama.cpp fork. Relative to that implementation, the reported speedups are $1.625$5--$1.625$6 on Emerald Rapids and $1.625$7--$1.625$8 on Arrow Lake. The comparison is informative because five-trit packing has a nominally lower fixed symbol rate than 2-bit packing, yet its unpacking and kernel integration are less competitive in the tested software stack.
On Lunar Lake, the end-to-end results reverse the ordering predicted by storage size alone. The 2-bit reference is best for every tested model, while BITCOS loses because its decode sequence is instruction-bound. This outcome supports the paper’s roofline methodology and limits the scope of its performance claims to platforms with sufficient memory pressure relative to unpacking cost.
Figure 8: End-to-end CPU decode confirms BITCOS gains on memory-bound server and client systems but losses on the bandwidth-rich, instruction-bound Lunar Lake CPU.
Xe2 microbenchmarks and GPU inference
The Xe2 GPU results are more uniformly positive than the CPU results. On the integrated Arc 140V, BITCOS exceeds the XeTLA int2 reference by $1.625$9--$1.585$0 over the observed model-density interval. On the discrete Arc Pro B70, the corresponding microbenchmark improvement is $1.585$1--$1.585$2. The gains are below the nominal byte ratios because the decoding work becomes increasingly exposed as the payload shrinks. For example, at $1.585$3 on the B70, a measured $1.585$4 speedup falls short of the $1.585$5 payload ratio.
Figure 9: Xe2 GEMV sweeps show consistent, though sublinear, BITCOS improvements over the tuned int2 reference on integrated and discrete GPUs.
The end-to-end vLLM XPU evaluation uses the same seven checkpoints and batch-one decode protocol. BITCOS improves throughput on every model and both Xe2 platforms. The Arc 140V achieves $1.585$6--$1.585$7 speedup over the XeTLA int2 implementation, while the Arc Pro B70 achieves $1.585$8--$1.585$9. Relative to the llama.cpp Vulkan Q2_0 implementation, BITCOS is 0--1 faster on Arc 140V and 2--3 faster on Arc Pro B70.
The discrete-GPU comparison also reveals a baseline effect. XeTLA int2 is substantially faster than the llama.cpp Vulkan Q2_0 path on Arc Pro B70, by 4--5. Thus, BITCOS’s reported advantage over Q2_0 is not attributable solely to the storage layout; it also reflects the use of a more optimized backend. The relevant scientific comparison remains BITCOS versus the tuned XeTLA int2 kernel, for which the improvements are smaller but still consistently positive.
Figure 10: End-to-end Xe2 decode demonstrates positive BITCOS speedups on both integrated and discrete GPUs, with the largest gains on selected sparse checkpoints.
Position relative to sparse ternary inference
The paper differentiates BITCOS from arithmetic-sparsity approaches. Ternary CSC methods can exploit zeros to reduce additions and subtractions, but their irregular index representation is poorly matched to bandwidth-bound batch-one decode. The paper cites results in which such methods require approximately 6--7 sparsity to surpass dense baselines at the layer level, whereas the evaluated ternary checkpoints contain only 8--9 zeros. Their principal benefit is therefore expected in compute-bound regimes such as prefill, not in the decode regime targeted by BITCOS.
BITCOS instead preserves dense positional traversal. Its bitmap is regular, its sign stream is compact, and its decoding operations are designed around SIMD masks, pdep, GPU lookup tables, and existing VNNI/DPAS contraction layouts. The approach consequently targets memory traffic rather than arithmetic elimination. This distinction explains why it can improve batch-one decode without requiring specialized sparse arithmetic hardware.
The method is also orthogonal to the quantization techniques that produce ternary models. QAT methods such as BitNet, Bonsai, TriLM, and Maple, and PTQ methods such as CAT-Q and TWLA, determine accuracy and symbol distributions. BITCOS changes only representation and execution. Since it is bit-exact, the paper’s performance claims do not depend on an accuracy tradeoff introduced by the storage format.
Limitations and open questions
The paper’s empirical scope is substantial but hardware-specific. All optimized kernels target Intel AVX2, AVX-512, or Xe2 architectures, and the experiments use Intel CPUs and GPUs. The results therefore establish the viability of BITCOS on the tested platforms rather than demonstrating architecture-independent superiority. In particular, the use of pdep is beneficial on the evaluated x86 CPUs but cannot be assumed to have equivalent throughput on other ISAs.
The storage-rate analysis also abstracts away some implementation details. The headline 0 rate describes ternary symbols alone. Group scales, alignment, block padding, metadata, and tensor-layout constraints can reduce the realized compression ratio. The paper reports scale-inclusive rates for its model comparison, but the exact overhead depends on the checkpoint’s grouping scheme and deployment format.
The performance advantage is conditional on the memory-to-instruction balance. Lunar Lake is an explicit counterexample: BITCOS stores fewer bits but runs more slowly than the simpler 2-bit kernel. The GPU gains are likewise smaller than the byte ratios because decode work becomes exposed as zero density increases. A remaining question is how the layout should be adapted dynamically across heterogeneous systems, particularly when the same checkpoint is served on platforms with materially different instruction throughput and memory bandwidth.
Finally, the evaluated workload is batch-one decode, where matrix-vector operations are bandwidth-sensitive. The paper does not establish comparable gains for large-batch decode, prefill, or compute-bound GEMM regimes. Those regimes may favor different representations, including arithmetic sparse kernels or fixed-width formats. The open technical issue is therefore not whether BITCOS reduces storage, which follows directly from its encoding, but how broadly its decoding strategy remains optimal across batch sizes, tensor shapes, activation precisions, and non-Intel architectures.
Conclusion
BITCOS reframes ternary LLM storage around measured symbol distributions rather than an equiprobable ternary alphabet. Its bitmap-plus-compacted-sign representation costs 1 bits per weight, beats practical five-trit packing above 2 zero density, and is more compact than 2-bit packing for all nonzero zero densities. In 29 evaluated checkpoints, it improves storage for 26 and reaches 3 bits per weight.
The implementation contribution is equally important: pdep-based CPU decoding and SLM lookup-based Xe2 decoding make the variable-rate representation compatible with high-throughput SIMD and matrix-contraction pipelines. The measured gains are strongest in bandwidth-bound batch-one inference, reaching 4 on the evaluated server CPU and 5 on the discrete Xe2 GPU in end-to-end decode. The Lunar Lake results establish the principal boundary condition: compression benefit becomes performance benefit only when memory traffic, rather than unpacking throughput, is the dominant bottleneck.
Whiteboard
Paper Prompts
Sign up for free to create and run prompts on this paper.