Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Approaching Rate-Distortion Limits in Neural Compression with Lattice Transform Coding (2403.07320v1)

Published 12 Mar 2024 in cs.IT, cs.LG, eess.SP, and math.IT

Abstract: Neural compression has brought tremendous progress in designing lossy compressors with good rate-distortion (RD) performance at low complexity. Thus far, neural compression design involves transforming the source to a latent vector, which is then rounded to integers and entropy coded. While this approach has been shown to be optimal in a one-shot sense on certain sources, we show that it is highly sub-optimal on i.i.d. sequences, and in fact always recovers scalar quantization of the original source sequence. We demonstrate that the sub-optimality is due to the choice of quantization scheme in the latent space, and not the transform design. By employing lattice quantization instead of scalar quantization in the latent space, we demonstrate that Lattice Transform Coding (LTC) is able to recover optimal vector quantization at various dimensions and approach the asymptotically-achievable rate-distortion function at reasonable complexity. On general vector sources, LTC improves upon standard neural compressors in one-shot coding performance. LTC also enables neural compressors that perform block coding on i.i.d. vector sources, which yields coding gain over optimal one-shot coding.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (33)
  1. E. Agrell and B. Allen. On the best lattice quantizers. IEEE Transactions on Information Theory, 69(12):7650–7658, 2023. doi: 10.1109/TIT.2023.3291313.
  2. Closest point search in lattices. IEEE Transactions on Information Theory, 48(8):2201–2214, 2002. doi: 10.1109/TIT.2002.800499.
  3. Gradient-based optimization of lattice quantizers. arXiv preprint arXiv:2401.01799, 2024.
  4. S. Arimoto. An algorithm for computing the capacity of arbitrary discrete memoryless channels. IEEE Transactions on Information Theory, 18(1):14–20, 1972. doi: 10.1109/TIT.1972.1054753.
  5. Variational image compression with a scale hyperprior. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=rkcQFMZRb.
  6. Nonlinear transform coding. IEEE Journal of Selected Topics in Signal Processing, 15(2):339–353, 2021. doi: 10.1109/JSTSP.2020.3034501.
  7. TensorFlow Compression: Learned data compression, 2024. URL http://github.com/tensorflow/compression.
  8. W. R. Bennett. Spectra of quantized signals. The Bell System Technical Journal, 27(3):446–472, 1948. doi: 10.1002/j.1538-7305.1948.tb01340.x.
  9. Do neural networks compress manifolds optimally? In 2022 IEEE Information Theory Workshop (ITW), pages 582–587, 2022. doi: 10.1109/ITW54588.2022.9965938.
  10. R. Blahut. Computation of channel capacity and rate-distortion functions. IEEE Transactions on Information Theory, 18(4):460–473, 1972. doi: 10.1109/TIT.1972.1054855.
  11. J. Bucklew. Companding and random quantization in several dimensions. IEEE Transactions on Information Theory, 27(2):207–211, 1981. doi: 10.1109/TIT.1981.1056319.
  12. J. Bucklew. A note on optimal multidimensional companders (corresp.). IEEE Transactions on Information Theory, 29(2):279–279, 1983. doi: 10.1109/TIT.1983.1056643.
  13. Entropy-constrained vector quantization. IEEE Transactions on Acoustics, Speech, and Signal Processing, 37(1):31–42, 1989. doi: 10.1109/29.17498.
  14. J. Conway and N. Sloane. Soft decoding techniques for codes and lattices, including the golay code and the leech lattice. IEEE Transactions on Information Theory, 32(1):41–50, 1986. doi: 10.1109/TIT.1986.1057135.
  15. On the voronoi regions of certain lattices. SIAM Journal on Algebraic Discrete Methods, 5(3):294–305, 1984. doi: 10.1137/0605031. URL https://doi.org/10.1137/0605031.
  16. Sphere Packings, Lattices, and Groups. Grundlehren der mathematischen Wissenschaften. Springer, New York, NY, 1999. ISBN 978-0-387-98585-5.
  17. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA, 2006. ISBN 0471241954.
  18. Density estimation using real NVP. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=HkpbnH9lx.
  19. Image compression with product quantized masked image modeling. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=Z2L5d9ay4B.
  20. W. Equitz and T. Cover. Successive refinement of information. IEEE Transactions on Information Theory, 37(2):269–275, 1991. doi: 10.1109/18.75242.
  21. Nvtc: Nonlinear vector transform coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6101–6110, 2023.
  22. A. Gersho. Asymptotically optimal block quantization. IEEE Transactions on Information Theory, 25(4):373–380, 1979. doi: 10.1109/TIT.1979.1056067.
  23. Normalizing flows: An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence, 43(11):3964–3979, 2020.
  24. Neural estimation of the rate-distortion function with applications to operational source coding. IEEE Journal on Selected Areas in Information Theory, 3(4):674–686, 2022. doi: 10.1109/JSAIT.2023.3273467.
  25. Strong functional representation lemma and applications to coding theorems. In 2017 IEEE International Symposium on Information Theory (ISIT), pages 589–593, 2017. doi: 10.1109/ISIT.2017.8006596.
  26. High-resolution source coding for non-difference distortion measures: multidimensional companding. IEEE Transactions on Information Theory, 45(2):548–561, 1999. doi: 10.1109/18.749002.
  27. Better lattice quantizers constructed from complex integers. IEEE Transactions on Communications, 70(12):7932–7940, 2022. doi: 10.1109/TCOMM.2022.3215685.
  28. D. McAllester and K. Stratos. Formal limitations on the measurement of mutual information. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 875–884. PMLR, 26–28 Aug 2020. URL https://proceedings.mlr.press/v108/mcallester20a.html.
  29. Neural discrete representation learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6309–6318, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
  30. A. B. Wagner and J. Ballé. Neural networks optimally compress the sawbridge. In 2021 Data Compression Conference (DCC), pages 143–152. IEEE, 2021.
  31. Y. Wu and P. Yang. Minimax rates of entropy estimation on large alphabets via best polynomial approximation. IEEE Transactions on Information Theory, 62(6):3702–3720, 2016. doi: 10.1109/TIT.2016.2548468.
  32. Y. Yang and S. Mandt. Towards empirical sandwich bounds on the rate-distortion function. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=H4PmOqSZDY.
  33. An introduction to neural data compression. arXiv preprint arXiv:2202.06533, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Eric Lei (14 papers)
  2. Hamed Hassani (120 papers)
  3. Shirin Saeedi Bidokhti (31 papers)
Citations (1)
View on Semantic Scholar