Trellis-Coded Quantization (TCQ)

• Trellis-Coded Quantization is a source coding method that integrates conventional quantization with finite-state trellis codes to exploit signal dependencies.
• It employs a convolutional code and the Viterbi algorithm to minimize total distortion, achieving shaping gains close to theoretical rate–distortion limits.
• TCQ underpins practical applications such as video coding, Massive MIMO feedback, and high-dimensional neural network quantization with efficient complexity-performance trade-offs.

Trellis-Coded Quantization (TCQ) is a class of source coding schemes that fuses traditional quantization with finite-state trellis codes to exploit structure and dependencies in source signals for improved rate–distortion performance. TCQ leverages a convolutional code to label quantization alternatives across source sequences, enabling shaping gain and complexity–performance trade-offs unattainable by scalar or conventional vector quantization. The paradigm is foundational in a wide range of research and standards, from optimal feedback in massive MIMO to transform coding in video standards and high-dimensional post-training quantization of neural networks.

1. Principle and Mathematical Structure

TCQ operates by replacing the memoryless quantization of independent samples with a sequence-dependent quantization rule governed by a finite-state trellis, typically derived from a binary convolutional code. At each stage, the choice of quantization output is constrained by the present trellis state, reflecting past quantization decisions. Formally, a TCQ encoder minimizes the total distortion over a block of source symbols by finding the trellis path $a^{L*}\in\mathcal C_{\rm conv}$ yielding

$$a^{L*} = \arg\min_{a^L\in\mathcal C_{\rm conv}} \sum_{n=0}^{L-1} \lambda_n(a_n),\quad \lambda_n(a_n) = \min_{s\in\mu(a_n)} \|x_n - s\|^2$$

where $x_n$ is the source sample, $\mathcal C_{\rm conv}$ is the set of paths specified by the convolutional code, and $\mu(a_n)$ is the set of allowed reconstruction values for label $a_n$ (0704.1411). The Viterbi algorithm efficiently finds the optimal path due to the additive structure of the metric.

Key elements:

• Trellis structure: Rate and memory control the number of states and transitions per stage.
• Distance-preserving labeling: Mapping between quantizer output cosets and binary labels is engineered to ensure that Hamming distance in the code approximates or preserves Euclidean distance in the source space (0704.1411).
• Codebook design: Reconstruction values for each path-labeling coset are chosen to minimize distortion, subject to trellis constraints and possibly other application-specific metrics.

2. Rate–Distortion Performance and Shaping Gain

Asymptotically, TCQ can approach the theoretical rate–distortion bound for high-rate quantization. The normalized granular gain $\gamma_g$ with optimal TCQ codebooks reaches $1.428$ dB in the scalar case for a 1024-state trellis, close to the known upper bound of $1.53$ dB (0704.1411). The improvement over uniform quantization is realized by shaping the quantizer cells and leveraging code-constrained sequence selection.

Empirically, for blocklength and state complexity $(L,2^m)$, granular gain for maximum-Hamming-distance convolutional codes matches or slightly outperforms classical designs (see Table 1):

States	Ungerboeck code $$a^{L*} = \arg\min_{a^L\in\mathcal C_{\rm conv}} \sum_{n=0}^{L-1} \lambda_n(a_n),\quad \lambda_n(a_n) = \min_{s\in\mu(a_n)} \|x_n - s\|^2$$0 (dB)	Max-Hamming code $$a^{L*} = \arg\min_{a^L\in\mathcal C_{\rm conv}} \sum_{n=0}^{L-1} \lambda_n(a_n),\quad \lambda_n(a_n) = \min_{s\in\mu(a_n)} \|x_n - s\|^2$$1 (dB)
4	1.005	1.003
8	1.097	1.098
1024	–	1.428

This supports the theoretical principle that distance-optimal binary codes, paired with isometric coset labeling, maximize the minimum Euclidean distance between codewords (0704.1411).

3. Trellis Design and Complexity Trade-offs

The choice of trellis rate and memory (number of states) directly impacts both shaping gain and computational complexity. Encoding and decoding are performed via the Viterbi algorithm with per-sample cost $$a^{L*} = \arg\min_{a^L\in\mathcal C_{\rm conv}} \sum_{n=0}^{L-1} \lambda_n(a_n),\quad \lambda_n(a_n) = \min_{s\in\mu(a_n)} \|x_n - s\|^2$$2 for $$a^{L*} = \arg\min_{a^L\in\mathcal C_{\rm conv}} \sum_{n=0}^{L-1} \lambda_n(a_n),\quad \lambda_n(a_n) = \min_{s\in\mu(a_n)} \|x_n - s\|^2$$3-memory $$a^{L*} = \arg\min_{a^L\in\mathcal C_{\rm conv}} \sum_{n=0}^{L-1} \lambda_n(a_n),\quad \lambda_n(a_n) = \min_{s\in\mu(a_n)} \|x_n - s\|^2$$4 binary convolutional codes, scaling linearly with sequence length and exponentially with trellis memory (0704.1411).

In high-dimensional applications—such as massive MIMO feedback (Mirza et al., 2014, Choi et al., 2013) or large neural network quantization (Tseng et al., 2024, Iyengar, 11 May 2026)—TCQ exploits the trellis to avoid exponential storage and search costs seen in vector quantization:

• Massive MIMO: Per-antenna overhead for finding beamformer codewords scales linearly in $$a^{L*} = \arg\min_{a^L\in\mathcal C_{\rm conv}} \sum_{n=0}^{L-1} \lambda_n(a_n),\quad \lambda_n(a_n) = \min_{s\in\mu(a_n)} \|x_n - s\|^2$$5, as opposed to codebook search scaling as $$a^{L*} = \arg\min_{a^L\in\mathcal C_{\rm conv}} \sum_{n=0}^{L-1} \lambda_n(a_n),\quad \lambda_n(a_n) = \min_{s\in\mu(a_n)} \|x_n - s\|^2$$6 (Mirza et al., 2014, Choi et al., 2013).
• LLM Weight Quantization: In QTIP, the trellis allows ultra-high-dimensional quantization without an exponential codebook; stateful decoding with $$a^{L*} = \arg\min_{a^L\in\mathcal C_{\rm conv}} \sum_{n=0}^{L-1} \lambda_n(a_n),\quad \lambda_n(a_n) = \min_{s\in\mu(a_n)} \|x_n - s\|^2$$7 states replaces the $$a^{L*} = \arg\min_{a^L\in\mathcal C_{\rm conv}} \sum_{n=0}^{L-1} \lambda_n(a_n),\quad \lambda_n(a_n) = \min_{s\in\mu(a_n)} \|x_n - s\|^2$$8 codebook for vector blocks, providing $$a^{L*} = \arg\min_{a^L\in\mathcal C_{\rm conv}} \sum_{n=0}^{L-1} \lambda_n(a_n),\quad \lambda_n(a_n) = \min_{s\in\mu(a_n)} \|x_n - s\|^2$$9 storage and permiting full parallelized decode (Tseng et al., 2024, Iyengar, 11 May 2026).

4. TCQ in Practical Systems and Standards

4.1 Video Coding: VVC/VVC and AV2

Both VVC and AV2 integrate TCQ into the luma transform block quantization pipeline (Wang et al., 2020, Nalci et al., 6 Jan 2026). The trellis states encode the choice of one of two quantizer codebooks (offset by $x_n$0) and enforce path constraints:

• VVC: TCQ is run on $x_n$1 blocks, with up to four trellis states per stage, Lagrangian $x_n$2 cost, and dynamic programming for the optimal path. Complexity is reduced by adaptive departure-point selection and branch-pruning, saving $x_n$3 quantization time at $x_n$4 BD-rate loss (Wang et al., 2020).
• AV2: The TCQ trellis (nine states) enables joint optimization of coefficient levels, Q0 versus Q1 selection, and end-of-block signaling via dynamic programming. Total natural-content BD-rate savings are $x_n$5–$x_n$6 (RA, AI) in PSNR and VMAF, with negligible decode-side complexity (Nalci et al., 6 Jan 2026).

4.2 Wireless Communications: Massive MIMO CSI Feedback

TCQ and its extensions are central to low-complexity, high-performance CSI quantization:

• Standard TCQ for beamformer codebooks: Employs a convolutional code-based trellis to quantize the normalized channel in Euclidean or Grassmannian metrics. The Viterbi search replaces exponential codebook search, enabling scaling to large numbers of antennas (Choi et al., 2013).
• Differential and correlated schemes: By applying temporal and spatial correlation models, TCQ tracks incremental channel evolution with dynamic constellation scaling and translation, attaining $x_n$7–$x_n$8 dB improved beamforming gain and $x_n$9–$\mathcal C_{\rm conv}$0 bps/Hz ZF spectral efficiency improvements over noncoherent TCQ (Mirza et al., 2014).

4.3 Neural Network Quantization

QTIP exploits high-dimensional TCQ, using hardware-efficient shift-register trellises with either lookup or polynomially computed codebooks to quantize LLM weights:

• Computation: Each V-vector at time $\mathcal C_{\rm conv}$1 is reconstructed by a single bitshift and computed code routine; no large VQ lookup.
• Performance: For 2-bit Gaussian sources, 256D TCQ approaches theoretical rate–distortion bounds; in large LLMs, it achieves perplexity comparable to (or better than) 8D VQ with an order-of-magnitude higher throughput on memory-bound hardware (Tseng et al., 2024).
• BCJR-QAT: Differentiable relaxation of TCQ quantization for quantization-aware training leverages the BCJR forward–backward algorithm to compute a Boltzmann expectation over legal paths, provably recovers the original hard TCQ code at $\mathcal C_{\rm conv}$2, and attains improved downstream metrics in LLMs (e.g., $\mathcal C_{\rm conv}$3 PPL on WikiText-2 over QTIP-PTQ at layer 4) (Iyengar, 11 May 2026).

4.4 Relay Quantization in Networks

In compress-and-forward relay channels, TCQ is tuned end-to-end for maximizing information-theoretic rates instead of minimizing distortion. The trellis enables tractable iterative multi-level coding and soft-information exchange with LDPC codes and BCJR decoding, notably outperforming scalar quantization by $\mathcal C_{\rm conv}$4 dB for PSK at high rates (Wan et al., 2023).

5. Code Design and Labeling Optimization

The performance of TCQ depends critically on code construction:

• Distance-preserving labeling: Binary codewords are mapped to output cosets so that Hamming and Euclidean distances track, maximizing minimum distance in the codebook and enhancing shaping gain. The canonical mapping for $\mathcal C_{\rm conv}$5 is $\mathcal C_{\rm conv}$6 (0704.1411).
• Hamming-Optimal Convolutional Codes: Selecting codes with maximum free Hamming distance (e.g., $\mathcal C_{\rm conv}$7 for 4-state, $\mathcal C_{\rm conv}$8 for 64-state) delivers rate–distortion curves at least as good as, and usually marginally superior to, Ungerboeck-labeled codes at identical complexity.

6. Implementation Considerations and Algorithmic Acceleration

Efficient practical deployment of TCQ encompasses several enhancements:

• Complexity reduction in coding standards: Adaptive state initialization (“departure point”) and branch-pruning strategies in VVC and AV2 remove unnecessary computations, reducing encoding time by up to $\mathcal C_{\rm conv}$9 with negligible fidelity loss (Wang et al., 2020).
• Explicit bitstream design: In AV2 and VVC, the TCQ state fully determines context-adaptive entropy decoding, requiring no extra signaling bits per coefficient for quantizer selection (Nalci et al., 6 Jan 2026).
• Memory and hardware efficiency: In QTIP, the “bitshift” trellis allows hardware-parallel decoding and eliminates large codebooks, enabling $\mu(a_n)$0 LLM inference memory bandwidth utilization (Tseng et al., 2024).

7. Extensions and Applications

TCQ’s adaptability spans multiple domains:

• Deep learning-based compression: Integration of TCQ into end-to-end autoencoder architectures for image compression demonstrates superior rate–distortion trade-offs at low bit rates versus scalar quantization (Li et al., 2020).
• Soft-to-hard path relaxation: BCJR-QAT generalizes the non-differentiable Viterbi selection with a differentiable, annealed sum-product marginalization, enabling path-based quantization-aware retraining and surpassing post-training quantization baselines in LLMs (Iyengar, 11 May 2026).
• Relay channel coding: Optimizing the TCQ boundaries for compress-forward rates, not MSE, closes implementation–theory gaps in multilevel-coded relay networks (Wan et al., 2023).

---

TCQ’s combination of convolutional code-based trellis structures, distance-preserving labelings, and path-metric minimization yields coding schemes that outperform scalar and conventional vector quantization in rate–distortion, complexity, and robustness across modern communications and compression applications. Its ongoing evolution in neural network quantization and differentiable algorithms demonstrates broad potential for future systems architectures.