PAC Learnability for Reliable Communication over Discrete Memoryless Channels
Abstract: In practical communication systems, knowledge of channel models is often absent, and consequently, transceivers need be designed based on empirical data. In this work, we study data-driven approaches to reliably choosing decoding metrics and code rates that facilitate reliable communication over unknown discrete memoryless channels (DMCs). Our analysis is inspired by the PAC (probably approximately correct) learning theory and does not rely on any assumptions on the statistical characteristics of DMCs. We show that a naive plug-in algorithm for choosing decoding metrics is likely to fail for finite training sets. We propose an alternative algorithm called the virtual sample algorithm and establish a non-asymptotic lower bound on its performance. The virtual sample algorithm is then used as a building block for constructing a learning algorithm that chooses a decoding metric and a code rate using which a transmitter and a receiver can reliably communicate at a rate arbitrarily close to the channel mutual information. Therefore, we conclude that DMCs are PAC learnable.
- T. Gruber, S. Cammerer, J. Hoydis, and S. t. Brink, “On deep learning-based channel decoding,” in Proc. 51st Annual Conf. Inf. Sci. Systems (CISS), 2017, pp. 1–6.
- O. Simeone, “A very brief introduction to machine learning with applications to communication systems,” IEEE Trans. Cogn. Commun. Netw., vol. 4, no. 4, pp. 648–664, 2018.
- W. Zhang, Y. Wang, C. Shen, and N. Liang, “A regression approach to certain information transmission problems,” IEEE J. Sel. Areas Commun., vol. 37, no. 11, pp. 2517–2531, Nov 2019.
- C. She, C. Sun, Z. Gu, Y. Li, C. Yang, H. V. Poor, and B. Vucetic, “A tutorial on ultrareliable and low-latency communications in 6G: Integrating domain knowledge into deep learning,” Proc. IEEE, vol. 109, no. 3, pp. 204–246, 2021.
- L. G. Valiant, “A theory of the learnable,” Commun. ACM, vol. 27, no. 11, p. 1134–1142, nov 1984.
- N. Weinberger, “Generalization bounds and algorithms for learning to communicate over additive noise channels,” IEEE Trans. Inf. Theory, vol. 68, no. 3, pp. 1886–1921, 2022.
- A. Tsvieli and N. Weinberger, “Learning maximum margin channel decoders,” IEEE Trans. Inf. Theory, vol. 69, no. 6, pp. 3597–3626, June 2023.
- N. I. Bernardo, J. Zhu, and J. Evans, “Learning channel codes from data: Performance guarantees in the finite blocklength regime,” in Proc. IEEE Int. Symp. Inf. Theory (ISIT), June 2023, pp. 2129–2134.
- M. A. Tope and J. M. Morris, “Near-optimal sampling to optimize communication over discrete memoryless channels,” in Proc. 57th Annual Conf. Inf. Sci. Systems (CISS), March 2023, pp. 1–6.
- N. Merhav, G. Kaplan, A. Lapidoth, and S. Shamai Shitz, “On information rates for mismatched decoders,” IEEE Trans. Inf. Theory, vol. 40, no. 6, pp. 1953–1967, 1994.
- I. Csiszár and P. Narayan, “Channel capacity for a given decoding metric,” IEEE Trans. Inf. Theory, vol. 41, no. 1, pp. 35–43, Jan 1995.
- J. Salz and E. Zehavi, “Decoding under integer metrics constraints,” IEEE Trans. Commun., vol. 43, no. 2/3/4, pp. 307–317, 1995.
- A. Lapidoth and P. Narayan, “Reliable communication under channel uncertainty,” IEEE Trans. Inf. Theory, vol. 44, no. 6, pp. 2148–2177, Oct 1998.
- J. Scarlett, A. G. i Fàbregas, A. Somekh-Baruch, and A. Martinez, “Information-theoretic foundations of mismatched decoding,” Found. Trends in Commun. Inf. Theory, vol. 17, no. 2–3, pp. 149–401, 2020.
- I. Csiszár and J. Körner, “Graph decomposition: A new key to coding theorems,” IEEE Trans. Inf. Theory, vol. 27, no. 1, pp. 5–12, January 1981.
- J. Y. N. Hui, “Fundamental issues of multiple accessing,” Ph.D. dissertation, Massachusetts Institute of Technology, 1983.
- J. Ziv, “Universal decoding for finite-state channels,” IEEE Trans. Inf. Theory, vol. 31, no. 4, pp. 453–460, 1985.
- R. Krichevskiy, “Laplace’s law of succession and universal encoding,” IEEE Trans. Inf. Theory, vol. 44, no. 1, pp. 296–303, 1998.
- A. Orlitsky, N. P. Santhanam, and J. Zhang, “Always Good Turing: Asymptotically optimal probability estimation,” Science, vol. 302, no. 5644, pp. 427–431, 2003.
- G. Valiant and P. Valiant, “Estimating the unseen: Improved estimators for entropy and other properties,” J. ACM, vol. 64, no. 6, oct 2017.
- G. Miller, “Note on the bias of information estimates,” Inf. Theory Psychol.: Probl. Methods, vol. 2, Jan 1955.
- L. Paninski, “Estimation of entropy and mutual information,” Neur. Comput., vol. 15, no. 6, pp. 1191–1253, 06 2003.
- J. Jiao, K. Venkat, Y. Han, and T. Weissman, “Minimax estimation of functionals of discrete distributions,” IEEE Trans. Inf. Theory, vol. 61, no. 5, pp. 2835–2885, 2015.
- Y. Wu and P. Yang, “Minimax rates of entropy estimation on large alphabets via best polynomial approximation,” IEEE Trans. Inf. Theory, vol. 62, no. 6, pp. 3702–3720, 2016.
- S. Verdú, “Empirical estimation of information measures: A literature guide,” Entropy, no. 8, 2019.
- W. Hoeffding, “Probability inequalities for sums of bounded random variables,” J. American Statistical Assoc., vol. 58, no. 301, pp. 13–30, 1963.
- A. Antos and I. Kontoyiannis, “Convergence properties of functional estimates for discrete distributions,” Random Struct. Algor., vol. 19, no. 3-4, pp. 163–193, 2001.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
