Layered 2-Step MLSE for Partial-CSI Detection

• L2S-MLSE is a two-phase dynamic-programming algorithm that estimates sequences under inter-symbol interference with imperfect channel state information.
• It employs a forward Viterbi pass to collect essential statistics followed by a backward penalty refinement to approximate near-ML detection.
• Optimizations like sliding block decomposition and state reduction enable low-latency, hardware-efficient detection in high-speed optical systems.

Layered 2-Step Maximum Likelihood Sequence Estimation (L2S-MLSE) is a two-phase dynamic-programming receiver algorithm for digital communication systems impaired by inter-symbol interference (ISI) and subject to imperfect or partial channel state information (CSI). Unlike conventional Maximum Likelihood Sequence Estimation (MLSE) which typically requires perfect knowledge of the channel, L2S-MLSE directly addresses the partial-CSI scenario, achieving performance close to the true maximum-likelihood solution with practical computational complexity. The technique has also been adapted and simplified for optical transmission links where strict low-latency and low-complexity constraints prevail, without degrading detection performance (Wu et al., 2013, Guo et al., 22 Dec 2025).

1. MLSE with Partial Channel State Information

Conventional MLSE, often implemented via the Viterbi Algorithm (VA), is optimal in the presence of perfect CSI but becomes suboptimal under imperfect channel knowledge. In practice, the receiver typically acquires only a least-squares (LS) estimate of the channel impulse response $\hat h$ based on a known training sequence. The formal observation model after partial CSI estimation is

$y = B h + n,$

where $B = [B_P \; B_D]$ partitions the convolutional matrix into known (training) and unknown (data-dependent) sections, and $h$ is the unknown channel vector. The LS channel estimate is $\hat h = (B_P^H B_P)^{-1} B_P^H y_P$. The joint-likelihood for symbol detection, marginalized over uncertain $h$, leads to a sequence metric coupling all data symbols, precluding the direct use of single-pass VA due to the loss of the finite-state, additivity property (Wu et al., 2013).

2. Two-Phase L2S-MLSE Structure

The L2S-MLSE algorithm comprises two recursive passes over the trellis representation:

Phase I. Forward Pass (Conventional VA)

The standard VA is applied using only the Euclidean (mismatched) metric

$$\phi_t = \sum_{\tau=1}^t |y_{\tau+T} - u_{\tau} \hat h|^2,$$

preserving for each trellis state both the minimum-metric path and auxiliary (side-information) arrays:

• $\eta_t^{(\ell)}$: partial correlation of metric residuals with hypothesized symbol sequences,
• $\rho_t^{(\ell)}$: symbol autocorrelation statistics required for the subsequent penalty computations.

Phase II. Backward Pass (Penalty Refinement)

The backward VA phase augments the classical path metric with a state-and-branch-dependent penalty term reconstructing the quadratic coupling induced by the marginalization over $h$: $$\Sigma_t(j,i) = \varphi_{t+1}(j) + |c_{t+1}(i,j)|^2 + \phi_t(i) - \lambda_t(j,i).$$ The $\lambda_t(j,i)$ term incorporates the accumulated correlation statistics from Phase I and Phase II, allowing accurate approximation of the near-ML metric for partial-CSI. The final sequence is reconstructed by a trace-back over the global survivor path.

3. Parallel and Low-Latency Variants

For high-throughput optical interconnects subject to stringent latency and resource requirements, L2S-MLSE is further optimized by block-based parallelism and computational simplification (Guo et al., 22 Dec 2025). The key mechanisms include:

• Sliding block decomposition: Data is partitioned into overlapping subblocks, permitting all branch-metric (BM) and initial path-metric computations to be performed fully in parallel.
• Layered (lookahead) recursion: BM are grouped in two-step units, enabling the first layer of Add-Compare-Select (ACS) logic to commute many operations simultaneously.
• Binary-tree reduction: The resulting survivor metrics are collapsed using log-depth binary-tree ACS, reducing end-to-end decision latency from $O(N)$ (sequential 1-step MLSE) to $O(\log N)$.

A summary of complexity and resource reduction for typical PAM4, channel-memory-2 scenarios is as follows:

Algorithm Variant	Multipliers	Adders	Comparators
Full-rate 1-step MLSE	512	1024	387
Simplified 1-step MLSE	33	520	65
Basic L2S-MLSE	512	2512	1483
Simplified L2S-MLSE	33	935	127

Resource reductions arise from common-term factorization and state-reduction (e.g., 4-to-2 state trellis), with negligible BER penalty for moderate ISI.

4. Interaction of Forward and Backward Layers

The two recursion passes in L2S-MLSE serve complementary purposes:

1. Forward VA: Accumulates the conventional Euclidean path metric and gathers all symbol-wise first- and second-order correlation statistics necessary for evaluating joint likelihoods in the presence of channel uncertainty.
2. Backward VA: Utilizes the collected statistics to introduce a state- and branch-dependent correction term, effectively decoupling the non-additive penalty induced by partial-CSI integration. The procedure maintains dynamic programming efficiency by reusing statistics, thereby avoiding $O(2^{MN})$ complexity of exhaustive sequence search (Wu et al., 2013).

This layered approach enables the recovery of nearly the exact near-ML solution, bridging the gap between computational tractability and statistical optimality.

5. Performance Evaluation and Comparative Results

Performance has been established in both wireless block-fading channels and high-speed optical systems. Key empirical findings include:

• In BPSK systems (block-fading, $P=2$, $N=15$, $T=5$), L2S-MLSE matches exhaustive near-ML performance and outperforms coherent MLSE with LS channel estimate by approximately 0.8 dB at $10^{-2}$ word error rate.
• For larger data blocks ($N=70$, $T=10$), L2S-MLSE maintains a 0.7 dB advantage over conventional MLSE+LS, with genie-aided perfect-CSI MLSE forming a lower bound.
• Under time-varying Gauss–Markov fading ($\alpha=0.9999$), the L2S-MLSE gain persists; for faster fading ($\alpha=0.999$), a minor error floor emerges at high SNR, but the algorithm remains superior in the operational regime (Wu et al., 2013).
• In 112 Gb/s PAM4 optical IM/DD links with adaptive equalization and $[1, \alpha]$ post-filter ($\alpha = 0.55$), simplified L2S-MLSE attains identical BER to 1-step MLSE and meets FEC requirements at drastically reduced hardware complexity and sublinear latency (Guo et al., 22 Dec 2025).

6. Design Trade-Offs and Practical Guidelines

The L2S-MLSE framework exposes critical design parameters:

• Complexity vs. Latency: Sequential (1-step) MLSE trades moderate complexity for high ($O(N)$) latency; standard L2S-MLSE offers low latency ($O(\log N)$) at the cost of increased per-block complexity; full simplification achieves both low latency and linear complexity, preserving MLSE-level BER.
• Block size and overlap: Sliding block sizes should capture the entire ISI memory ($O$ overlap length), with $N$ kept minimal for hardware efficiency.
• State reduction: Reducing the number of trellis states (e.g., from 4 to 2) yields substantial hardware savings at negligible BER penalty under moderate ISI.

Optimal deployment involves matched filtering, symbol-rate resampling, FFE, and post-filtering prior to L2S-MLSE detection.

7. Broader Impact and Limitations

L2S-MLSE establishes an efficient paradigm for near-optimal sequence detection under imperfect CSI, particularly when moderate training/data ratios are achievable. The algorithm retains the core benefits of MLSE—robust ISI mitigation and minimum BER—while drastically reducing complexity relative to brute-force sequence search. Its adaptation to high-speed, low-latency optical systems confirms its broader viability. However, for highly dynamic fading or extremely limited training overhead, error floors may emerge due to residual channel estimation uncertainty—a recognized limitation inherent to all partial-CSI sequence detectors (Wu et al., 2013, Guo et al., 22 Dec 2025).