Breaking the 1.58-bit Barrier for Ternary LLMs
This presentation examines a novel approach to storing ternary large language models more efficiently than conventional methods. By exploiting the nonuniform distribution of zero-valued weights in deployed ternary checkpoints, the BITCOS representation achieves sub-1.58-bit storage rates without retraining or accuracy loss. The talk explores the bitmap-plus-compacted-sign encoding, its hardware-aware decoding implementations for Intel CPUs and GPUs, and the conditions under which reduced storage translates into faster inference.Script
Ternary large language models store each weight as minus one, zero, or plus one, conventionally assumed to require log base 2 of 3, or about 1.585 bits per weight. But deployed ternary checkpoints contain far more zeros than the other values, and that asymmetry opens a surprising door: by simply repacking the same weights without retraining, researchers have pushed storage below 1.5 bits per weight.
The authors measured 29 ternary checkpoints across seven model families and found zero densities ranging from 29.7 percent to 51.5 percent. Twenty six of those models exceed the 37.5 percent threshold where the proposed method becomes more compact than the standard five trit per byte packing scheme.
BITCOS separates each ternary tensor into a dense presence bitmap, with one bit per weight, and a compacted sign stream holding one bit only for each nonzero weight. The bitmap identifies which positions are zero; the sign stream supplies plus or minus one for the rest, in tensor order.
The AVX 512 decoding kernel uses the pdep instruction to scatter compact sign bits into their bitmap indicated positions in a single operation. Whether this unpacking step is faster than a simpler 2 bit layout depends on whether memory traffic or instruction throughput is the bottleneck.
On the memory bound Emerald Rapids server and Arrow Lake client CPUs, BITCOS improves end to end decode throughput by up to 1.18 times over the state of the art 2 bit kernel. But on Lunar Lake, where eight cores share abundant memory bandwidth, the added unpacking work makes BITCOS slower than the simpler format.
BITCOS demonstrates that ternary storage rates depend not only on alphabet size but on the empirical distribution of symbols, and that reduced bit width translates into speedup only when decoding remains bandwidth bound. To explore this work further and create your own research videos, visit EmergentMind.com.
