Compensation of Fiber Nonlinearities in Digital Coherent Systems Leveraging Long Short-Term Memory Neural Networks

Abstract: We introduce for the first time the utilization of Long short-term memory (LSTM) neural network architectures for the compensation of fiber nonlinearities in digital coherent systems. We conduct numerical simulations considering either C-band or O-band transmission systems for single channel and multi-channel 16-QAM modulation format with polarization multiplexing. A detailed analysis regarding the effect of the number of hidden units and the length of the word of symbols that trains the LSTM algorithm and corresponds to the considered channel memory is conducted in order to reveal the limits of LSTM based receiver with respect to performance and complexity. The numerical results show that LSTM Neural Networks can be very efficient as post processors of optical receivers which classify data that have undergone non-linear impairments in fiber and provide superior performance compared to digital back propagation, especially in the multi-channel transmission scenario. The complexity analysis shows that LSTM becomes more complex as the number of hidden units and the channel memory increase can be less complex than DBP in long distances (> 1000 km).