Trellis-Extended Codebooks in Massive MIMO

• Trellis-Extended Codebooks (TEC) are structured CSI quantization techniques that reduce the exponential search complexity in FDD massive MIMO systems.
• TEC partitions high-dimensional channels into blocks and utilizes Viterbi-based trellis search to efficiently map trellis transitions to extended codewords.
• The differential extension TE-SPA exploits temporal channel correlation to further decrease feedback overhead and improve quantization performance by up to 2 dB.

Trellis-Extended Codebooks (TEC) are a class of channel state information (CSI) quantization techniques designed to enable scalable, low-complexity, and bandwidth-efficient feedback in frequency division duplexing (FDD) massive multiple-input multiple-output (MIMO) systems. TEC approaches leverage the structure of trellis-coded quantization, convolutional encoders, and Viterbi decoding to extend conventional codebooks, originally constructed for small numbers of antennas, for practical use in systems with tens to hundreds of transmit antennas. The framework also includes differential extensions, notably Trellis-Extended Successive Phase Adjustment (TE-SPA), to exploit temporal channel correlation for further feedback reduction. TEC solves both the exponential codeword search complexity and excessive feedback challenges of conventional vector-quantized codebooks while remaining compatible with standards such as LTE and LTE-Advanced (Choi et al., 2014). Closely related methods include Noncoherent Trellis Coded Quantization (NTCQ), which applies similar principles based on Grassmannian quantization and noncoherent sequence detection (Choi et al., 2013).

1. System Model and Motivation

TEC is formulated in the context of FDD downlink systems where a base station with $M_t$ transmit antennas communicates with a single-antenna user under block fading: $$y[k] = \sqrt{P}\cdot \mathbf{h}^\mathrm{H}[k]\mathbf{f}[k] s[k] + z[k],$$ with $\mathbf{h}[k] \in \mathbb{C}^{M_t}$ (expectation $\|\mathbf{h}[k]\|^2 = M_t$), $$\mathbf{f}[k] \in \mathbb{C}^{M_t}, \|\mathbf{f}[k]\| = 1$$, and complex AWGN $z[k]$. CSI must be quantized and fed back using a limited bit budget $B_{\text{tot}}$. To preserve performance as $M_t$ increases, $B_{\text{tot}}$ must scale linearly: $B_{\text{tot}} = B \cdot M_t$.

Conventional channel direction quantization with a codebook $$y[k] = \sqrt{P}\cdot \mathbf{h}^\mathrm{H}[k]\mathbf{f}[k] s[k] + z[k],$$0 of $$y[k] = \sqrt{P}\cdot \mathbf{h}^\mathrm{H}[k]\mathbf{f}[k] s[k] + z[k],$$1 unit-norm vectors requires selecting

$$y[k] = \sqrt{P}\cdot \mathbf{h}^\mathrm{H}[k]\mathbf{f}[k] s[k] + z[k],$$2

but this incurs prohibitive $$y[k] = \sqrt{P}\cdot \mathbf{h}^\mathrm{H}[k]\mathbf{f}[k] s[k] + z[k],$$3 complexity and feedback overhead as $$y[k] = \sqrt{P}\cdot \mathbf{h}^\mathrm{H}[k]\mathbf{f}[k] s[k] + z[k],$$4 grows. TEC addresses these scalability constraints through structured codebook extension and efficient search.

2. Construction of Trellis-Extended Codebooks

TEC constructs large codebooks by integrating:

• A small-dimensional vector-quantized codebook $$y[k] = \sqrt{P}\cdot \mathbf{h}^\mathrm{H}[k]\mathbf{f}[k] s[k] + z[k],$$5 with $$y[k] = \sqrt{P}\cdot \mathbf{h}^\mathrm{H}[k]\mathbf{f}[k] s[k] + z[k],$$6 elements in $$y[k] = \sqrt{P}\cdot \mathbf{h}^\mathrm{H}[k]\mathbf{f}[k] s[k] + z[k],$$7, where $$y[k] = \sqrt{P}\cdot \mathbf{h}^\mathrm{H}[k]\mathbf{f}[k] s[k] + z[k],$$8.
• A rate-$$y[k] = \sqrt{P}\cdot \mathbf{h}^\mathrm{H}[k]\mathbf{f}[k] s[k] + z[k],$$9 convolutional encoder, mapping $\mathbf{h}[k] \in \mathbb{C}^{M_t}$0 input bits per branch to $\mathbf{h}[k] \in \mathbb{C}^{M_t}$1 output bits/labels, with $\mathbf{h}[k] \in \mathbb{C}^{M_t}$2 branches per trellis state.

The $\mathbf{h}[k] \in \mathbb{C}^{M_t}$3-dimensional input is partitioned into $\mathbf{h}[k] \in \mathbb{C}^{M_t}$4 consecutive $\mathbf{h}[k] \in \mathbb{C}^{M_t}$5-dimensional blocks, and each block is quantized through a trellis transition, emitting an $\mathbf{h}[k] \in \mathbb{C}^{M_t}$6-D codeword label. Across $\mathbf{h}[k] \in \mathbb{C}^{M_t}$7 transitions, the path through the trellis selects a sequence of $\mathbf{h}[k] \in \mathbb{C}^{M_t}$8 codewords. The user searches over trellis paths using the Viterbi algorithm and feeds back the sequence of branch input bits, for a total of $\mathbf{h}[k] \in \mathbb{C}^{M_t}$9 bits. This enables fractional bits per antenna: $\|\mathbf{h}[k]\|^2 = M_t$0.

The codeword selection criterion minimizes the Euclidean distance to the (possibly phase-rotated) channel: $\|\mathbf{h}[k]\|^2 = M_t$1 where $\|\mathbf{h}[k]\|^2 = M_t$2 is the concatenation of $\|\mathbf{h}[k]\|^2 = M_t$3 codewords along path $\|\mathbf{h}[k]\|^2 = M_t$4 and $\|\mathbf{h}[k]\|^2 = M_t$5 is the set of all trellis paths. A scalar phase grid $\|\mathbf{h}[k]\|^2 = M_t$6 accounts for the unknown global phase.

At the receiver, CSI is reconstructed by passing the fed-back bit sequence through the same convolutional encoder to recover the codeword sequence, which is concatenated and normalized.

3. Complexity Analysis and Comparison

The primary advantage of TEC is a reduction in quantization complexity from exponential to linear in $\|\mathbf{h}[k]\|^2 = M_t$7. Conventional vector quantized codebook search requires $\|\mathbf{h}[k]\|^2 = M_t$8 operations, which is infeasible for $\|\mathbf{h}[k]\|^2 = M_t$9. TEC's complexity is

$$\mathbf{f}[k] \in \mathbb{C}^{M_t}, \|\mathbf{f}[k]\| = 1$$0

where $$\mathbf{f}[k] \in \mathbb{C}^{M_t}, \|\mathbf{f}[k]\| = 1$$1 is the number of global phase candidates. For fixed $$\mathbf{f}[k] \in \mathbb{C}^{M_t}, \|\mathbf{f}[k]\| = 1$$2, $$\mathbf{f}[k] \in \mathbb{C}^{M_t}, \|\mathbf{f}[k]\| = 1$$3, and $$\mathbf{f}[k] \in \mathbb{C}^{M_t}, \|\mathbf{f}[k]\| = 1$$4, complexity is $$\mathbf{f}[k] \in \mathbb{C}^{M_t}, \|\mathbf{f}[k]\| = 1$$5. This efficiency allows practical support of hundreds of antennas without loss of quantization fidelity relative to Random Vector Quantization (RVQ) baselines.

Similar scalability is achieved in NTCQ, which builds the codebook from a small per-antenna base constellation, a rate-$$\mathbf{f}[k] \in \mathbb{C}^{M_t}, \|\mathbf{f}[k]\| = 1$$6 convolutional code (e.g., Ungerboeck TCM), and trellis search over Viterbi paths, yielding overall codebook size $$\mathbf{f}[k] \in \mathbb{C}^{M_t}, \|\mathbf{f}[k]\| = 1$$7 and encoding cost $$\mathbf{f}[k] \in \mathbb{C}^{M_t}, \|\mathbf{f}[k]\| = 1$$8 (Choi et al., 2013).

4. Key Mathematical Formulations and Workflow

TEC's quantization performs a trellis-based search:

• The $$\mathbf{f}[k] \in \mathbb{C}^{M_t}, \|\mathbf{f}[k]\| = 1$$9-dimensional channel is partitioned into $z[k]$0 blocks of $z[k]$1 entries.
• For each block, candidate codewords from the short codebook are mapped to trellis branches.
• For each partial path $z[k]$2 at stage $z[k]$3 and phase $z[k]$4,

$z[k]$5

where $z[k]$6 is the codeword label at stage $z[k]$7.

• The Viterbi decoder runs independently for each candidate $z[k]$8, producing the trellis path and quantized codeword sequence minimizing total error.
• Feedback is the concatenation of $z[k]$9 bits per transition.
• The base station reconstructs CSI by passing the received bit sequence through the convolutional encoder and concatenating codewords.

This enables fractional bits/entry. For example, $B_{\text{tot}}$0 and $B_{\text{tot}}$1 yields $B_{\text{tot}}$2 bits per entry.

5. Differential Extension: Trellis-Extended Successive Phase Adjustment (TE-SPA)

TE-SPA augments TEC to exploit temporal or frequency correlation:

• Channel evolves as $B_{\text{tot}}$3, $B_{\text{tot}}$4.
• Prior quantized CSI $B_{\text{tot}}$5 is adjusted by a block-wise diagonal phase matrix:

$B_{\text{tot}}$6

• The phases $B_{\text{tot}}$7 are jointly optimized over a PSK codebook using a Viterbi search, with a global phase $B_{\text{tot}}$8, to minimize

$B_{\text{tot}}$9

• TE-SPA enables low-overhead differential feedback ($M_t$0 bits per entry, typically $M_t$1) while maintaining accuracy for highly correlated channels.
• A block-shifting (circular interleaving) mechanism ensures uniform quantization refinement.

In temporally correlated (e.g., $M_t$2) channels, TE-SPA reduces quantization error by 1–2 dB over one-shot TEC, even with $M_t$3 bits per entry.

6. Performance and Application to Massive MIMO

TEC and TE-SPA are validated through simulation:

• Under i.i.d. Rayleigh fading, TEC with $M_t$4 and $M_t$5 bits/entry achieves beamforming gains within approximately 1 dB of an ideal RVQ codebook at the same bit rate, for $M_t$6 up to 256.
• TE-SPA further improves quantization accuracy by 1–2 dB when channel time correlation is high.
• For spatially correlated channels, quantizing the dominant eigenvector using TEC+TE-SPA offers more than 1 dB improvement over direct instantaneous quantization.
• Encoding and decoding complexity remains manageable due to Viterbi algorithm efficiency ($M_t$7 for fixed parameters).

NTCQ yields comparable results by mapping the quantization problem to a Grassmannian manifold and employing noncoherent sequence detection, achieving near-optimal chordal distance and beamforming gain for moderate feedback (Choi et al., 2013).

7. Relation to Standards and Related Techniques

TEC is specifically designed for interoperability with LTE and LTE-Advanced, extending existing 2-, 4-, or 8-antenna codebooks to support massive MIMO deployments. The fractional bits/entry property and scalable design allow practical feedback rates and reconstruction accuracy within the constraints of current wireless standards.

Related frameworks, such as NTCQ, utilize similar trellis-coded modulation structures, mapping codebook quantization to source encoding over a manifold and exploiting Viterbi-based path search to achieve large codebooks with linear encoding complexity. Differential and spatially adaptive variants enable further refinement for temporally or spatially correlated channels. Both TEC and NTCQ circumvent the exponential growth of codebook storage and search, producing feasible solutions for FDD massive MIMO systems (Choi et al., 2014, Choi et al., 2013).