- The paper introduces Invariant Bit Packing (IBP), which removes frequently repeated floating-point bits using compact metadata and warp-parallel decompression while ensuring compressed data never exceeds its original size.
- IBP achieves up to 9.7× compressed-transfer throughput and delivers end-to-end improvements including 74% faster Legion GNN training, 180% faster DLRM lookups, and 22% lower FlexGen LLM latency.
- The method integrates with major ML systems using tens of lines of code, but its benefits depend on stable data distributions and are limited for uniformly random data or very small tensors with high transfer overhead.
Motivation and problem
ML workloads such as GNN training, DLRM inference, and LLM inference routinely operate on tensors that exceed GPU memory capacity, forcing on-demand transfers from CPU memory over PCIe. Because PCIe bandwidth is more than an order of magnitude below GPU memory bandwidth, these transfers become the dominant performance constraint even when caching and compute/transfer overlap are employed. The paper "Reducing the GPU Memory Bottleneck with Lossless Compression for ML" (2605.30728) by Kamath, Krishnamurthy, Canini, and Peter addresses this bottleneck with lossless compression, explicitly avoiding the accuracy risks of lossy approaches such as quantization, which are considered unacceptable in commercial deployments where even a 0.1% accuracy loss is disallowed.
The authors first demonstrate why existing GPU-accelerated lossless compressors (nvCOMP algorithms such as ANS, GDeflate, LZ4, zStd, Cascaded, and ndzip-gpu) fail in this setting. Many achieve high space savings but deliver worse transfer throughput than sending uncompressed data: for example, zStd achieves the best space savings on sparse GNN datasets (93.4%) yet one of the worst speedups (1.8× on sparse, 0.1× on dense), because dictionary-based decompression requires multiple PCIe reads per element. Code-table-free schemes like Cascaded perform better but use metadata proportional to input size, causing compressed dense tensors to exceed their original size. This motivates a design with minimal metadata, single-pass memory access, and warp-parallel decompression.
Invariant Bit Packing
The core observation is that ML tensors exhibit low entropy at the bit level even when numerical values appear diverse. Exponent and sign bits of floating-point features are frequently identical across tensors drawn from structured distributions; for the Reddit dataset, bits 2–5 of float32 features held the same value over 90% of the time. The authors define an "80p-invariant" bit as one whose value is identical across at least 80% of tensors, and show all evaluated datasets contain substantial invariant fractions—up to 99% for Citeseer and Cora, 13–15% for dense GNN datasets, and 27% for BF16 LLM KV-caches.
Invariant Bit Packing (IBP) exploits this by storing invariant bits once in a fixed-size Mask/Bitval metadata pair (kilobytes even for terabyte-scale datasets) resident in GPU memory, and removing those bits from each tensor. Compression proceeds chunk-wise: a chunk is compressed only if its masked values match Bitval, with a participation bit per chunk indicating compression status; tensors that do not benefit remain uncompressed, guaranteeing the compressed dataset never exceeds the original size. Decompression uses warp-parallel iterative decompression: a warp cooperatively reads contiguous segments into shared memory via asynchronous zero-copy accesses, threads determine their starting bit offsets through a warp-intrinsic prefix scan, and decompress independently—all CPU memory is scanned exactly once, communication stays within the warp, and PCIe traffic uses 128B-aligned bounded transfers, which improve copy throughput by 25% over naive GPU-initiated copying.
System integration
IBP integrates into ML pipelines through two mechanisms. Cache compression repacks compressed tensors contiguously into per-GPU data slabs, replacing 64-bit pointers in the cache hashmap with a 3-bit slab ID, 40-bit offset, and a compressed flag—supporting up to 1 TB across 8 GPUs without extra lookups. PCIe transfer compression performs in-place compression so fixed-offset indexing still works, with a 1-bit-per-tensor bitmask in GPU memory identifying uncompressed tensors (64 MB for a 512 GB dataset of 1 KB tensors). Integration effort is small: fewer than 35 lines of Python for DGL, ~15 for DLRM's CachedEmbeddingBag, ~20 for FlexGen, and ~40 for InfiniGen.
Evaluation results
On an A100 with PCIe Gen4 x16 (~25 GB/s observed), IBP delivers the best compressed-transfer throughput among nine algorithms across all six workload categories, reaching 9.7× on sparse GNN data while remaining above 1× everywhere—including dense cases where most competitors fall below 0.5×. It is also the only algorithm that compresses every dataset type, achieving 10–12% savings on dense GNN features, 8% on DLRM weights, and up to 27% on BF16 LLM weights. A notable finding is datatype sensitivity: OPT-30B FP16 KV-caches compress only 4–5%, whereas Gemma-7B BF16 KV-cache and weights compress 23% and 27% respectively, because BF16's 8 exponent bits create more invariant positions—an advantage obtained with no algorithmic changes.
End-to-end results are strong:
| Workload |
Framework |
Average improvement |
| GNN training |
Legion |
74% faster |
| GNN training |
DGL |
56% faster |
| DLRM embedding lookup |
CachedEmbeddingBag |
180% faster |
| LLM weight offloading |
FlexGen (Gemma-7B) |
22% lower latency |
| LLM KV offloading |
InfiniGen (OPT) |
up to 27% lower latency |
Latency breakdowns show IBP nearly eliminates sampler wait time for CoraSU and even reduces train time by 12% by freeing SMs earlier from transfer duty. Cache capacity increases track compression ratios, up to 25.88× for CoraSU. Decompression overhead is modest: for space savings under 50%, throughput exceeds 95% of ideal; at 90% savings it remains 78–86% for 1–4 KB tensors, though 256 B tensors suffer roughly 50% overhead due to 128B-transfer granularity. Sampling experiments show masks built from as little as 1–10% of a dataset generalize to full-dataset compression ratios, enabling streaming and distributed deployments; in InfiniGen, masks are regenerated from prompt tokens during prefill, costing a 10% prefill increase that is outweighed by decode gains.
The DLRM result deserves qualification: much of the 180% gain stems from replacing CPU-side table lookups with GPU-initiated aligned zero-copy transfers rather than compression itself, which contributed only 8–20%. Similarly, for InfiniGen's small 128–256 B FP16 K/V entries, compression improved transfers by only 2%; the speedup there comes primarily from GPU-initiated copying versus InfiniGen's CPU-driven path. The authors are transparent about this decomposition.
Sensitivity studies identify T/N≈ 80–85% and 4–8 B chunk sizes as effective operating points, and introduce clustered compression using bit-similarity k-means for datasets with only local invariance—for the asteroid.f32 benchmark, clustering raises net space savings from ~10% to 47%.
Limitations and open questions
Several constraints bound the results. First, IBP depends on stable, structured distributions: uniformly random data is essentially incompressible, and the authors note that if tensor statistics shift after preprocessing, stale Mask/Bitval metadata would cause mismatches and lost compression—the paper does not evaluate dynamic re-training of metadata under distribution drift beyond the per-prefill sampling used for InfiniGen. Second, compression time is moderate (50–70% spent on preprocessing; e.g., 84 s preprocessing plus 55 s compression for a 180 GB dataset), acceptable only because it is off the critical path; online or frequently-updated datasets would face this cost repeatedly. Third, small tensors (256 B) incur significant decompression overhead from 128B-aligned transfer granularity, limiting applicability to fine-grained entries such as per-head K/V slices. Fourth, the GNN evaluation relies on scaled-up synthetic variants of Pubmed/Citeseer/Cora (Products topology with reassigned features) and a cache restricted to 1% of feature size to emulate production hit rates, so absolute speedups may differ on naturally large graphs. Fifth, the InfiniGen experiment could not be run with BF16 models due to RoPE compatibility issues, leaving BF16 KV-offloading demonstrated only indirectly. Finally, the principles are argued to extend to disk and network transfers, but only PCIe paths are evaluated.
Conclusion
This paper establishes that lossless compression is practical for ML data movement when the algorithm is co-designed with GPU execution: invariant-bit elimination provides kilobyte-scale metadata, warp-parallel iterative decompression hides overhead behind PCIe latency, and GPU-initiated aligned transfers outperform CPU-side copying. With average speedups of 74% (GNN training), 180% (DLRM lookup), and 24% (LLM inference) at unchanged model accuracy, and drop-in integration requiring tens of lines of code, IBP offers a deployable alternative to lossy quantization for bandwidth-constrained ML systems.