CrossMPT: Cross-attention Message-Passing Transformer for Error Correcting Codes (2405.01033v2)
Abstract: Error correcting codes (ECCs) are indispensable for reliable transmission in communication systems. The recent advancements in deep learning have catalyzed the exploration of ECC decoders based on neural networks. Among these, transformer-based neural decoders have achieved state-of-the-art decoding performance. In this paper, we propose a novel Cross-attention Message-Passing Transformer (CrossMPT), which shares key operational principles with conventional message-passing decoders. While conventional transformer-based decoders employ self-attention mechanism without distinguishing between the types of input vectors (i.e., magnitude and syndrome vectors), CrossMPT updates the two types of input vectors separately and iteratively using two masked cross-attention blocks. The mask matrices are determined by the code's parity-check matrix, which explicitly captures the irrelevant relationship between two input vectors. Our experimental results show that CrossMPT significantly outperforms existing neural network-based decoders for various code classes. Notably, CrossMPT achieves this decoding performance improvement, while significantly reducing the memory usage, complexity, inference time, and training time.
- Deep learning for decoding of linear codes-a syndrome-based approach. In Proceedings of 2018 IEEE International Symposium on Information Theory (ISIT), pp. 1595–1599. IEEE, 2018.
- Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
- Pruning and quantizing neural belief propagation decoders. IEEE Journal of Selected Areas in Communications, 39(7):1957–1966, 2021.
- End-to-end object detection with transformers. In Proceedings of the European conference on computer vision (ECCV), 2020.
- Error correction code transformer. In Advances in Neural Information Processing Systems (NeurIPS), 2022a.
- Error correction code transformer. https://github.com/yoniLc/ECCT, 2022b. Accessed: 2023-05-22.
- Denoising diffusion error correction codes. In Proceedings of International Conference on Learning Representations (ICLR), 2023.
- Learning to decode protograph ldpc codes. IEEE Journal of Selected Areas in Communications, 39(7):1983–1999, 2021.
- BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
- Reduced complexity iterative decoding of low-density parity check codes based on belief propagation. IEEE Transactions on Communications, 47(5):673–680, 1999.
- Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
- Mask R-CNN. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2015.
- Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
- Database of Channel Codes and ML Simulation Results. In https://rptu.de/en/channel-codes, 2019.
- Communication algorithms via deep learning. In International Conference on Learning Representations (ICLR), 2018.
- Physical layer communication via deep learning. IEEE Journal of Selected Topics in Information Theory, 1(1):5–18, 2020.
- Adam: A method for stochastic optimization. In arXiv preprint arXiv:1412.6980, 2014.
- ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2012.
- Neural min-sum decoding for generalized ldpc codes. IEEE Communications Letters, 26(12):2841–2845, 2022.
- Boosting Learning for LDPC Codes to Improve the Error-Floor Performance. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
- Neural offset min-sum decoding. In Proceedings of 2017 IEEE International Symposium on Information Theory (ISIT), pp. 1316–1365. IEEE, 2017.
- Hyper-graph-network decoders for block codes. In Advances in Neural Information Processing Systems (NeurIPS), pp. 2326–2336, 2019.
- Autoregressive belief propagation for decoding block codes. In arxiv preprint arXiv:2103.11780, 2021.
- Learning to decode linear codes using deep learning. In 2016 54th Annual Allerton Conference on Communications, Control, and Computing (Allerton), pp. 341–346. IEEE, 2016.
- Deep learning methods for improved decoding of linear codes. IEEE Journal of Selected Topics in Signal Processing, 12(1):119–131, 2018.
- How to mask in Error Correction Code Transformer: Systematic and double masking. In arXiv preprint arXiv:2308.08128, 2023.
- You only look once: Unified, real-time object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
- Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
- The capacity of low-density parity check codes under message-passing decoding. IEEE Transactions on Information Theory, 47(2):599–618, 2001.
- Very deep convolutional networks for large-scale image recognition. In Proceedings of International Conference on Learning Representations (ICLR), 2015.
- Going deeper with convolutions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015.