Neural Equalisers for Highly Compressed Faster-than-Nyquist Signalling: Design, Performance, Complexity and Robustness

Abstract: Faster-than-Nyquist (FTN) signalling has emerged as a compelling technique for enhancing spectral efficiency in bandwidth-constrained communication systems. By intentionally introducing controlled intersymbol interference (ISI), FTN allows transmission at rates exceeding the traditional Nyquist limit, unlocking new potential in high-speed data communication. However, its practical deployment remains challenged by the need for low-complexity detection strategies that can cope with the induced ISI while maintaining low latency and robust performance. We propose deep learning receivers that are resilient to non-idealities. In this paper, we present a deep learning-based framework for FTN signalling that addresses these challenges through several novel contributions. First, we propose a sliding window detection method that leverages temporal context while preserving computational efficiency. Second, we demonstrate the viability of FTN systems with very low packing factors, showing that reliable performance can be achieved even under aggressive spectral compression (up to 75\%). Our architecture is optimised for low latency and low complexity, making it suitable for real-time applications and scalable deployment. In addition, we assess the robustness of our models across varying channel conditions and noise profiles, providing insights into their generalisability and resilience.

• The paper introduces neural equalisers using sliding window detection with RNNs and Transformers to achieve near-optimal BER in highly compressed FTN channels.
• It demonstrates robust performance across diverse SNR conditions and severe ISI, significantly reducing computational complexity compared to trellis-based methods.
• Empirical results confirm that the proposed models maintain reliability despite training limitations and sampling offsets, making them viable for advanced communication systems.

Neural Equalisers for Highly Compressed Faster-than-Nyquist Signalling: Design, Performance, Complexity, and Robustness

Introduction and Context

The paper "Neural Equalisers for Highly Compressed Faster-than-Nyquist Signalling: Design, Performance, Complexity and Robustness" (2605.01570) addresses a fundamental challenge in modern digital communication: supporting higher spectral efficiency through Faster-than-Nyquist (FTN) signalling. FTN intentionally introduces controlled intersymbol interference (ISI) by reducing symbol intervals below the traditional Nyquist criterion, enabling transmission rates that surpass conventional limits. However, this approach results in elevated receiver complexity due to the severe ISI, particularly for aggressive compression, i.e., small packing factors $\tau$.

Traditional equalisation and detection techniques—such as Viterbi or MAP-based BCJR receivers—become computationally intractable under these conditions due to the exponential growth in trellis state space. This paper advances the field by proposing deep learning architectures, particularly RNNs and Transformers, as viable, robust, low-complexity receivers for highly compressed FTN channels, even in severe ISI scenarios.

Figure 1: Generic Faster-than-Nyquist (FTN) system model, illustrating the transmitter, channel with ISI, and receiver components.

System Model and Problem Formulation

The authors model the FTN system as a linear ISI channel with the severity of ISI determined by the packing ratio ($\tau$) and pulse shape (e.g., root raised cosine). For small $\tau$, the channel memory becomes substantial (e.g., ISI spans 18 symbols for $\tau=0.5$, $\beta=0.3$). This necessitates processing windows that are computationally impractical for classic MLSE.

To circumvent channel ill-conditioning and colored noise, the paper employs a super-minimum phase equivalent FTN channel, which enables tractable noise whitening and reduces the effective channel memory.

Deep Learning-Based Sliding Window Detection

The core proposal is a sliding window detection method using neural networks (NNs), particularly tailored for long-memory ISI. The sliding detector processes a window of received symbols and predicts the symbol at the window's center, progressively sliding through the received sequence.

The architectures evaluated include:

• Fully Connected NNs (FCNNs): Baseline, low-memory, incapable of capturing extended temporal dependencies in severe ISI.
• RNNs (LSTM, GRU, Simple RNN): Both unidirectional (processing past context) and bidirectional (incorporating future context).
• Transformer architectures: Leveraging attention to aggregate global context efficiently, proposed as an alternative to RNNs for sequence modeling in equalisation.

Figure 3: Training and validation loss curves for FTN-LSTM and FTN-BiLSTM, demonstrating effective convergence and the utility of early stopping to prevent overfitting.

Implementation and Training Methodology

The networks are intentionally shallow, per observations that symbol detection for physical layer channels does not require deep or large models. Training is supervised, using large synthetic datasets spanning ranges of SNRs to ensure robustness, and early stopping is applied based on validation loss.

Careful evaluation of dataset size and diversity indicates that models can generalise well, with little degradation up to moderate reductions in training sample diversity, though severe bias (e.g., run-length limited training) impairs generalisability.

Empirical Results: BER Performance and Sensitivity

BER Across Packing Ratios and Architectures

Extensive simulations are performed for uncoded BPSK (and higher modulations) in AWGN FTN channels across packing ratios as low as $\tau=0.25$.

Figure 5: BER performance of neural equalizers (RNNs, Transformer) across varying FTN packing ratios, demonstrating near-optimal performance even under aggressive ISI.

Key findings include:

• Unidirectional LSTM models nearly achieve the minimum Euclidean distance bound, within 0.25 dB for $\tau=0.5$, outperforming DFE, MAP, and Viterbi decoders at practical computational cost.
• FCNNs and classic DFEs are inadequate for severe ISI.
• Transformer models deliver marginal gains over Bi-LSTM for $\tau=0.5$, but significantly outperform in lower $\tau$ regimes (increased ISI), showcasing the strength of attention mechanisms for long-range dependencies.
• Bidirectional LSTM (Bi-LSTM) architectures provide gains at low $\tau$ (e.g., $\tau$0) due to greater ISI memory.

SNR-Specific and Robustness Evaluation

NN receiver BER degrades if trained on data at mismatched SNR relative to deployment, but models trained on a "sweet spot" or a range of SNRs provide robust performance across SNR variations.

Figure 7: BER performance comparison of SNR-specific models, showing the tradeoff between training SNR and robustness to SNR mismatch in deployment.

Sensitivity to Non-Idealities

• Sampling Offset: LSTM architectures are robust to moderate sampling offsets (up to 40% of symbol period), extending resilience to practical receiver misalignments.

Figure 9: BER performance of the LSTM-based equalizer under sampling offset, demonstrating high robustness to timing misalignment for $\tau$1.

• Dataset Generalisation: Limiting sequence types in training (e.g., enforcing run-length constraints) degrades performance, stressing the need for representative training diversity.

Figure 11: Impact of training data bias (run-length constraints) on BER, underscoring the importance of diverse training sequence patterns.

• Generalisation to Unseen Sequences: Performance loss is observed when testing on sequences disjoint from training, but sliding window detection somewhat mitigates this effect.

Figure 13: BER when training and testing datasets are orthogonal, reflecting manageable, though non-negligible, performance degradation.

• Dataset Size: Little performance degradation is observed when reducing the dataset size to about 50,000 sequences for $\tau$2, but further reduction impairs BER.

Figure 2: BER as a function of training dataset size, revealing that moderate reduction is tolerated but aggressive reduction degrades generalisation.

Higher Order Modulation

• QPSK: Neural equalizers extend smoothly to QPSK; performance parallels BPSK.
• 4-PAM: Performance is worse due to increased interference and model complexity demands; QPSK is thus practically preferable.

Figure 16: BER for higher modulation orders, confirming QPSK performance tracks BPSK, while 4-PAM introduces noticeable degradation.

Complexity Analysis

Unlike MAP and Viterbi/BCJR algorithms whose complexity scales exponentially with channel memory $\tau$3 (i.e., $\tau$4), the neural sliding window detectors scale linearly with sequence length and cubically with window size ($\tau$5 for LSTM). Transformers, with attention heads, warrant a modest multiplicative overhead ($\tau$6), but remain much less complex than trellis-based methods for aggressive packing.

Theoretical and Practical Implications

This work establishes that deep learning-based equalizers—particularly sliding-window LSTM and Transformer models—are effective, efficient, and robust solutions for FTN channels operating at highly compressed symbol rates ($\tau$7). They:

• Achieve near-optimal BER with feasible latency and significantly reduced computational complexity compared to classical optimal detectors.
• Demonstrate strong resilience to SNR mismatch, sampling errors, and moderate reductions in training sample diversity.
• Extend to higher modulation with network scaling, though care is needed for non-orthogonal constellations.

These findings imply that aggressive FTN signalling, previously impractical due to the computational cost of detection, becomes feasible for high-rate physical layer systems, optical, or wireless through DNN equalization. The sliding-window method and attention-based models also present a blueprint for other communications problems characterized by long-memory, nonlinear ISI, or colored noise.

Future Developments

Potential future directions include:

• Adapting architectures for coded FTN systems with iterative detection and decoding.
• On-line/continual learning mechanisms to cope with time-varying channels.
• Hardware implementations to exploit parallelism in Transformer models for latency minimization in real-time systems.
• Analytical exploration of DNN generalization bounds for channel equalization, relating required dataset sizes to ISI order, modulation, and channel statistics.

Conclusion

This paper provides a thorough technical demonstration that deep learning—specifically sliding-window LSTM and Transformer equalizers—enables practical, high-performing FTN receivers under severe ISI and compressed signalling regimes. These NN-based equalizers balance low computational complexity, low latency, and strong robustness, making them promising for deployment in future ultra-high spectral efficiency communication systems. The comparison against established trellis-based methods and analysis of model robustness underscores the maturity and reliability of this neural equalization paradigm for advanced physical layer design (2605.01570).