OTFS-Modulated FTN Transmission

• OTFS-modulated FTN transmission is a hybrid signaling paradigm that boosts spectral efficiency by combining delay-Doppler processing with intentional ISI/ICI via FTN signaling.
• It employs advanced precoding and equalization techniques, including EVD-based noise whitening and water-filling power allocation, to manage interference in doubly dispersive channels.
• The approach increases capacity and reliability in high-mobility, spectrum-constrained environments while balancing performance trade-offs in complexity and BER.

Orthogonal time frequency space (OTFS)-modulated Faster-Than-Nyquist (FTN) transmission is a hybrid signaling paradigm that integrates OTFS—an effective solution for reliable communication in doubly dispersive channels—with intentional violation of the time-frequency orthogonality constraints imposed by the Nyquist criterion via FTN signaling. This combination aims to substantially boost spectral efficiency and information rate in high-mobility and spectrum-constrained wireless environments. The fundamental principle is to deliberately introduce controlled inter-symbol interference (ISI) and inter-carrier interference (ICI) by compressing the time and frequency grid beyond the Nyquist limit, while exploiting advanced precoding and detection strategies in the delay-Doppler (DD) domain to suppress resulting interference and maximize mutual information. The following sections present a comprehensive technical account of system models, signal constructions, transceiver algorithms, analytical performance, and implementation trade-offs for OTFS-modulated FTN systems, including recent multi-carrier, MIMO, and RIS-empowered extensions.

1. OTFS and FTN Signaling: System and Signal Model

In Nyquist OTFS, an $M\times N$ block of DD-domain QAM symbols $x[l,k]$ ($l=0,\ldots,M-1,\;k=0,\ldots,N-1$) is mapped to the time-frequency domain via the inverse symplectic finite Fourier transform (ISFFT): $$X[m,n]\;=\;\frac{1}{\sqrt{MN}}\sum_{k=0}^{N-1}\sum_{l=0}^{M-1}x[l,k]e^{j2\pi(\frac{nk}{N}-\frac{ml}{M})}.$$ Each $X[m,n]$ is pulse-shaped (typically with a Nyquist or root-raised-cosine (RRC) pulse) and transmitted with symbol interval $T_0$ and subcarrier spacing $\Delta f_0$ such that $T_0\Delta f_0=1$, maintaining strict orthogonality.

In FTN signaling, symbol time and/or frequency grid is compressed by factors $\alpha\leq1$ (time) and $\beta\leq1$ (frequency):

$$T = \alpha T_0, \quad \Delta f = \beta \Delta f_0,\quad T_0\Delta f_0=1.$$

Non-orthogonal (e.g., RRC with roll-off $\theta$) pulses are used on this compressed grid. The transmit signal then becomes: $$s(t) = \sqrt{\alpha\beta E_0} \sum_{m=0}^{M-1} \sum_{n=0}^{N-1} X_P[m,n]\, g_{tx}(t-n\alpha T_0)\, e^{j2\pi m\beta\Delta f_0 (t-n\alpha T_0)},$$ where $X_P[m,n]$ may include DD-domain precoding. ISI and ICI are now present due to loss of orthogonality.

Upon passing through a delay-Doppler sparse channel $$h(\tau,\nu) = \sum_{i=1}^L h_i \delta(\tau-\tau_i)\delta(\nu-\nu_i)$$, and after matched filtering and sampling, the time-frequency (TF) domain input-output is: $$Y[m,n] = \sum_{m',n'} H_{m,n}[m',n'] X_P[m',n'] + Z[m,n],$$ with $H_{m,n}[m',n']$ capturing all ISI/ICI contributions via ambiguity integrals over transmit/receive pulse shapes.

The equivalent DD-domain input-output after SFFT is: $$y^{\mathrm{DD}} = H^{\mathrm{DD}} x_P^{\mathrm{DD}} + z^{\mathrm{DD}}, \quad x_P^{\mathrm{DD}} = P x^{\mathrm{DD}},$$ where $H^{\mathrm{DD}}\in\mathbb C^{MN\times MN}$ encapsulates ISI/ICI from both FTN grid compression and the physical channel, and $z^{\mathrm{DD}}$ is colored noise (Wang et al., 12 Jan 2025, Hong et al., 2024, Zhang et al., 23 Dec 2025, Hong et al., 17 Jan 2026).

2. Precoding, Equalization, and Power Allocation Strategies

To counter FTN-induced interference and correlated noise, advanced precoding and detection are implemented in the DD-domain.

SISO Case:

Mutual information under Gaussian DD-domain inputs $x^{\mathrm{DD}}\sim\mathcal{CN}(0, \sigma_x^2 I)$ is given as: $$I(x^{\mathrm{DD}} ; y^{\mathrm{DD}}) = \log_2 \det \left( I + \frac{\sigma_x^2}{N_0} P^H H^{\mathrm{DD},H} G^{-1} H^{\mathrm{DD}} P \right ),$$ where $G$ is the TF-domain ISI/ICI matrix. By whitening and eigen-decomposition (EVD),

$$C^{\mathrm{SISO}} = \sum_{k=0}^{MN-1} \log_2 \left(1 + \frac{\sigma_x^2}{N_0} \lambda_{P,k} \lambda_{D,k}\right).$$

Optimal power allocation per sub-channel is given by water-filling: $$\lambda_{P,k} = \max \left\{ \frac{1}{\xi\phi_k \ln 2} - \frac{N_0}{\sigma_x^2 \lambda_{D,k}}, 0 \right\}.$$

EVD-based precoding diagonalizes the effective channel and enables symbol-by-symbol detection. This two-stage process (noise whitening and EVD) removes both inter-symbol correlation and channel mixing (Wang et al., 12 Jan 2025, Hong et al., 2024).

MIMO Case and SIC Precoding:

For $N_T$ transmit and $N_R$ receive antennas, the input-output relation is (with stacked DD streams): $$y^{\mathrm{DD}}_{\rm MIMO} = H^{\mathrm{DD}}_{\rm MIMO} P_{\rm MIMO} x^{\mathrm{DD}}_{\rm MIMO} + z^{\mathrm{DD}}_{\rm MIMO}.$$ MIMO capacity and power allocation can be computed via full EVD (complexity $O((MNN_T)^3)$) or Successive Interference Cancellation (SIC)-based decomposition, which splits the high-dimensional optimization into $N_T$ subproblems of size $MN\times MN$ each, reducing complexity to $O(N_T (MN)^3)$ while incurring marginal SNR loss (Wang et al., 12 Jan 2025).

3. Channel Estimation and Detection in OTFS-FTN

OTFS-FTN with RRC pulses and compressed symbol interval $T_f = \alpha T_0$ requires specialized channel estimation and detection to handle FTN-induced ISI:

• The DD-domain input-output for FTN-OTFS, after matched filtering, is characterized by a dense kernel where ISI bandwidth is limited to $2c+1$ diagonals, with $c \approx \beta/(2\alpha)$, exploiting the decay of $g(nT_f)$.
• Channel estimation uses FTN-interval pilots with Doppler and delay guards, detected in the DD grid after whitening, and supports joint estimation of fractional Doppler and path gain per tap.
• Reduced-complexity LMMSE equalization leverages the banded ISI matrix for $O(MN\,c^2) + O(MN\log_2 N)$ complexity, replacing the $\mathcal{O}((MN)^3)$ cost of full matrix inversion (Hong et al., 17 Jan 2026).

4. Transceiver Algorithmic Flows

A typical FTN-OTFS transceiver consists of the following operations (Hong et al., 2024):

Transmitter:

1. Map input bits to DD-symbols $X[k,l]$.
2. Perform EVD-based DD-domain precoding: $x_P = U^H x$, where $U$ is from EVD of noise-whitened effective channel.
3. ISFFT to map to TF domain: $X_{\rm TF} = F_M X F_N^H$.
4. IFFT to generate time-domain vector.
5. FTN sequence generation: $s(t) = \sum_n s_n g(t-nT_f)$.
6. Transmission.

Receiver:

1. Matched filtering and FFT/SFFT.
2. Noise whitening using EVD of the colored noise covariance.
3. Channel diagonalization, yielding independent scalar subchannels.
4. Log-likelihood ratio computation and FEC decoding.

5. Performance and Trade-Offs

Simulation studies consistently demonstrate that OTFS-modulated FTN transmission provides tangible improvements in both spectral efficiency and robustness under high Doppler, at manageable complexity and BER cost:

• Spectral Efficiency: Normalized capacity increases by $\sim$10–20% (even up to 30% in RIS-assisted settings) with moderate FTN factors (e.g., $\alpha=0.8,0.9$). Spectral efficiency grows approximately as $1/\alpha$, with $\alpha$ the time compression parameter (Wang et al., 12 Jan 2025, Hong et al., 2024, Zhang et al., 23 Dec 2025, Hong et al., 17 Jan 2026).
• BER/FER: The BER penalty for moderate FTN ($\alpha \approx 0.8$) is minor—$< 0.5$ dB at $10^{-4}$ BER for multi-carrier FTN-OTFS even with MMSE equalization and LDPC coding. Performance remains comparable or even superior to Nyquist-OTFS when joint DD-domain processing and/or advanced coding are used (Wang et al., 12 Jan 2025, Hong et al., 2024, Zhang et al., 23 Dec 2025, Hong et al., 17 Jan 2026, Mattu et al., 4 Aug 2025).
• Complexity: EVD and full water-filling realize maximum achievable capacity at cubic complexity, but SIC-based schemes in MIMO settings retain near-optimal rates with linear complexity in $N_T$.
• PAPR/IBO: FTN packing leads to negligible PAPR increase, and RIS beamforming can compensate average SNR loss, increasing available input back-off (Zhang et al., 23 Dec 2025).
• Robustness: FTN-OTFS with DD-domain estimation consistently outperforms conventional FTN (including DMFTN or EVD-FTN) and matches or exceeds Nyquist-OTFS in high-mobility channels (Hong et al., 17 Jan 2026, Mattu et al., 4 Aug 2025).

6. Advanced Architectures: MC-FTN, RIS-OTFS-FTN, and Zak-OTFS-FTN

• MC-FTN-OTFS generalizes FTN to both time and frequency compression (parameters $\alpha$, $\beta$). It leverages DD-domain EVD or SIC-based precoding for high-capacity MIMO with efficient resource utilization (Wang et al., 12 Jan 2025).
• RIS-OTFS-FTN incorporates passive beamforming with a finite-phase RIS, maximizing DD-domain channel taps to further boost spectral efficiency and reliability. Practical closed-form quantized phase design algorithms maximize channel gain. FTN-induced ISI and RIS beamforming jointly govern the rate-reliability trade-off (Zhang et al., 23 Dec 2025).
• Zak-OTFS-FTN enables FTN by symbol superposition in the DD domain using two mutually unbiased bases. A channel-aware MMSE precoder transforms the channel to effectively identity, and the receiver exploits Gaussianity of inter-frame interference, greatly simplifying detection. With trellis-coded modulation, Zak-OTFS-FTN achieves superior performance at high SNR and can surpass Nyquist-based schemes (Mattu et al., 4 Aug 2025).

7. Limitations and Practical Considerations

• Aggressive FTN compression ($\alpha, \beta \ll 1$) can render the ISI/ICI matrix ill-conditioned or singular, necessitating subcarrier deactivation or elaborate equalization.
• BER degradation under strong compression mandates stronger channel coding (e.g., LDPC, TCM) or iterative DD-domain detection.
• All advanced transceiver methods presuppose accurate DD-domain CSI; pilot design for FTN intervals and high-mobility scenarios remains an active research area (Hong et al., 17 Jan 2026).
• Real-time approximations (random-beam precoding, message-passing) are essential to bring EVD/SVD complexity down for practical 6G deployment.
• Implementation of large RIS, joint optimization of FTN factor $\alpha$, and quantization of RIS phases, as well as careful selection of DD grid size, provide further performance/complexity trade-offs (Zhang et al., 23 Dec 2025).

---

In conclusion, OTFS-modulated FTN transmission and its variants systematically exploit time-frequency resources and DD-domain channel sparsity, enabling a substantive increase in capacity and information rate in doubly-selective, high-mobility environments. These systems, rigorously developed and analyzed across several recent works (Wang et al., 12 Jan 2025, Hong et al., 2024, Zhang et al., 23 Dec 2025, Hong et al., 17 Jan 2026, Mattu et al., 4 Aug 2025), delineate the technical path toward high-performance, spectrum-efficient 6G waveform design.