Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Lossy Compression of Individual Sequences Revisited: Fundamental Limits of Finite-State Encoders (2401.01779v1)

Published 3 Jan 2024 in cs.IT and math.IT

Abstract: We extend Ziv and Lempel's model of finite-state encoders to the realm of lossy compression of individual sequences. In particular, the model of the encoder includes a finite-state reconstruction codebook followed by an information lossless finite-state encoder that compresses the reconstruction codeword with no additional distortion. We first derive two different lower bounds to the compression ratio that depend on the number of states of the lossless encoder. Both bounds are asymptotically achievable by conceptually simple coding schemes. We then show that when the number of states of the lossless encoder is large enough in terms of the reconstruction block-length, the performance can be improved, sometimes significantly so. In particular, the improved performance is achievable using a random-coding ensemble that is universal, not only in terms of the source sequence, but also in terms of the distortion measure.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Davisson, L. D. “Universal noiseless coding,” IEEE Trans. Inform. Theory 1973, vol. IT–19, no. 6, pp. 783–795.
  2. Gallager, R. G. “Source coding with side information and universal coding,” LIDS-P-937, M.I.T., 1976.
  3. Ryabko, B. “Coding of a source with unknown but ordered probabilities,” Problems of Information Transmission, 1979, vol. 15, pp. 134–138.
  4. Davisson, L. D.; Leon-Garcia, A. “A source matching approach to finding minimax codes,” IEEE Trans. Inform. Theory 1980, vol. 26, pp. 166–174.
  5. Krichevsky, R. E.; Trofimov, R. K. “The performance of universal encoding,” IEEE Trans. Inform. Theory 1981, vol. 27, no. 2, pp. 199–207.
  6. Shtar’kov, Yu M. “Universal sequential coding of single messages,” Problems of Information Transmission, 1987, vol. 23, no. 3, pp. 175–186.
  7. Barron, A. R.; Rissanen, J.; Yu, B.  “The minimum description length principle in coding and modeling,” IEEE Transactions on Information Theory, 1998, vol. 44, pp. 2734-2760.
  8. Yang, Y.; and Barron, A. R.  “Information-theoretic determination of minimax rates of convergence,” Annals of Statistics, 1999, vol. 27, pp. 1564-1599.
  9. Rissanen, J.  “Modeling by shortest data description,” Automatica, 1978, vol. 14, no. 5, pp. 465–471.
  10. Rissanen, J. “Universal coding, information, prediction, and estimation,” IEEE Transactions on Information Theory 1984, vol. IT–30, no. 4, pp. 629–636.
  11. Merhav, N.; Feder, M. “A strong version of the redundancy–capacity theorem of universal coding,” 1995, IEEE Trans. Inform. Theory, vol. 41, no. 3, pp. 714-722.
  12. Ornstein, D. S.; Shields, P. C. “Universal almost sure data compression,” Ann. Probab. 1990, vol. 18, no. 2, pp. 441–452.
  13. Zhang, Z.; Yang, E.-h.; Wei, V. “The redundancy of source coding with a fidelity criterion. I. known statistics,” IEEE Trans. Inform. Theory 1997, vol. 43, no. 1, pp. 71–91.
  14. Yu, B.; Speed, T. “A rate of convergence result for a universal d𝑑ditalic_d-semifaithful code,” IEEE Trans. Inform. Theory 1993, vol. 39, no. 3, pp. 813–820.
  15. Silva, J. F.; Piantanida, P.  “On universal d𝑑ditalic_d-semifaithful coding for memoryless sources with infinite alphabets,” IEEE Transactions on Information Theory 2022, vol. 68, no. 4, pp. 2782–2800.
  16. Kontoyiannis, I. “Pointwise redundancy in lossy data compression and universal lossy data compression,” IEEE Trans. Inform. Theory 2000, vol. 46, no. 1, pp. 136-152.
  17. Kontoyiannis, I.; Zhang, J. “Arbitrary source models and Bayesian codebooks in rate-distortion theory,” IEEE Trans. Inform. Theory 2002, vol. 48, no. 8, pp. 2276–2290.
  18. Mahmood, A.; Wagner, A. B. “Lossy compression with universal distortion,” IEEE Trans. Inform. Theory 2023, vol. 69, no. 6, pp. 3525–3543.
  19. Mahmood, A.; Wagner, A. B. “Minimax rate-distortion,” IEEE Trans. Inform. Theory 2023, vol. 69, no. 12, pp. 7712–7737.
  20. Sholomov, L. A.  “Measure of information in fuzzy and partially defined data,” Dokl. Math., 2006, vol. 74, no. 2, pp. 775–779.
  21. Ziv, J. “Coding theorems for individual sequences,” IEEE Trans. Inform. Theory 1978, vol. IT–24, no. 4, pp. 405–412.
  22. Ziv, J.; Lempel, A. “Compression of individual sequences via variable-rate coding,” IEEE Trans. Inform. Theory 1978, vol. IT–24, no. 5, pp. 530–536.
  23. Potapov, V. N. “Redundancy estimates for the Lempel-Ziv algorithm of data compression,” Discrete Appl. Math., 2004, vol. 135, no. 1–3, pp. 245-254.
  24. Merhav, N.; Ziv, J. “On the Wyner-Ziv problem for individual sequences,” IEEE Trans. Inform. Theory 2006, vol. 52, no. 3, pp. 867–873.
  25. Ziv, J. “Fixed-rate encoding of individual sequences with side information,” IEEE Transactions on Information Theory 1984, vol. IT–30, no. 2, pp. 348–452.
  26. Merhav, N. “Finite-state source-channel coding for individual source sequences with source side information at the decoder,” IEEE Trans. Inform. Theory 2022, vol. 68, no. 3, pp. 1532–1544.
  27. Ziv, J. “Distortion-rate theory for individual sequences,” IEEE Trans. Inform. Theory 1980, vol. IT–26, no. 2, pp. 137–143.
  28. Weinberger, M. J.; Merhav, N.; Feder, M. “Optimal sequential probability assignment for individual sequences,” IEEE Trans. Inform. Theory 1994, vol. 40, no. 2, pp. 384–396.
  29. Merhav, N. “D𝐷Ditalic_D-semifaithful codes that are universal over both memoryless sources and distortion measures,” IEEE Trans. Inform. Theory 2023, vol. 69, no. 7, pp. 4746-4757.
  30. Merhav, N. “A universal random coding ensemble for sample-wise lossy compression,” Entropy, 2023, 25, 1199 https://doi.org/10.3390/e25081199
  31. Neuhoff, D. L.; Gilbert, R. K. “Causal source codes,” IEEE Trans. Inform. Theory 1982, vol. IT–28, no. 5, pp. 701–713.
  32. Foster, J.; Gray, R. M.; Ostendorf Dunham, M. “Finite-state vector quantization for waveform coding,” IEEE Trans. Inform. Theory 1985, vol. IT-31, no. 3, pp. 348–359.
  33. Merhav, N. “On the data processing theorem in the semi-deterministic setting,” IEEE Trans. Inform. Theory 2014, vol. 60, no. 10, pp. 6032-6040.
  34. Cohen, A.; Merhav, N. “Universal randomized guessing subjected to distortion,” IEEE Trans. Inform. Theory 2022, vol. 68, no. 12, pp. 7714–7734.
  35. Merhav, N.; Cohen, A. “Universal randomized guessing with application to asynchronous decentralized brute–force attacks,” IEEE Trans. Inform. Theory 2020, vol. 66, no. 1, pp. 114–129.
  36. Merhav, N. “Guessing individual sequences: generating randomized guesses using finite-state machines,” IEEE Trans. Inform. Theory 2020, vol. 66, no. 5, pp. 2912–2920.
  37. Ziv, J. “Universal decoding for finite-state channels,” IEEE Trans. Inform. Theory 1985, vol. IT–31, no. 4, pp. 453–460.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets