---
title: High-Order Residual Quantization (HORQ)
url: https://www.emergentmind.com/topics/high-order-residual-quantization-horq
type: topic
---

# High-Order Residual Quantization (HORQ)

High-Order Residual Quantization (HORQ) is a quantization technique developed to improve the efficiency and fidelity of neural network inference, particularly for deployment on hardware with restricted numerical precision. HORQ generalizes traditional (order-one) quantization by recursively quantizing the residual error between the original tensor and its lower-precision approximation, thereby yielding a multi-term, high-order expansion. This technique underpins methods such as REx for data-free quantization and high-order binary filtering, enabling superior trade-offs between accuracy and computational efficiency across a range of deep neural architectures and bit-widths [2203.14645][1708.08687].

## 1. Formal Definition and Algorithmic Structure

Given a full-precision tensor (such as a weight matrix) $W \in \mathbb{R}^n$, a uniform quantizer $Q$ is first applied, producing a quantized representation and a corresponding dequantized value $Q^{-1}(Q(W))$. The residual error after this first quantization is $E^{(1)} = W - Q^{-1}(Q(W))$. HORQ recursively quantizes this residual up to order $K$, obtaining a sequence of approximants:
\[
R^{(1)} = Q^{-1}(Q(W)), \qquad R^{(k)} = Q^{-1}\left(Q\left(W - \sum_{j=1}^{k-1} R^{(j)}\right)\right)
\]
The original tensor is thus approximated as
\[
W \approx \sum_{k=1}^{K} R^{(k)}
\]
For the binary case (with $Q$ mapping to $\pm1$), as used in high-order binary neural networks, each term is formed as
\[
X^{(i)} = \alpha_i \operatorname{sign}(R^{(i-1)}), \qquad R^{(i)} = R^{(i-1)} - X^{(i)}
\]
where $\alpha_i = \frac{1}{n} \|R^{(i-1)}\|_1$ and $R^{(0)} = X$ [1708.08687].

## 2. Theoretical Properties and Error Analysis

The central theoretical result of HORQ is a provable, monotonic reduction in approximation error with each additional quantized residual. For the scalar case under symmetry assumptions, there is exponential convergence:
\[
|w - \sum_{j=1}^K w^{(j)}| \le \left(\frac{1}{2^{b-1}-1}\right)^{K-1} \frac{s_{R^{(K)}}}{2}
\]
where $b$ is the quantization bit-width and $s_{R^{(K)}}$ the dynamic range. The Euclidean error after $k$ orders for a vector $X$ satisfies
\[
\|X - \widehat X^{(k)}\|_2^2 \le \|X\|_2^2 - \sum_{i=1}^k \frac{\|R^{(i-1)}\|_1^2}{n}
\]
These error bounds demonstrate that, in practice, $K=2$ or $3$ suffices to nearly recover full-precision values for most weights [2203.14645][1708.08687]. At the network level, the worst-case output deviation is bounded by
\[
\max_{\|X\| = 1} \|F(X) - F^{(K)}(X)\|_{\infty} \le \prod_{l=1}^L (\sigma_l u_l^{(K)} + 1) - 1
\]
where $F$ is the full network, $F^{(K)}$ its $K$th-order quantized counterpart, $\sigma_l$ the spectral norm associated with the $l$th layer, and $u_l^{(K)}$ the per-layer residual bound [2203.14645].

## 3. Group-Sparse and Ensemble (Parallel) Expansions

To mitigate the $K\times$ increase in bit-operations associated with order-$K$ expansion, HORQ employs group-sparse regularization. Only a fraction $\gamma\in (0,1)$ of the most significant output channels are expanded at higher order. Specifically, the $\ell_1$ norm of each channel's $K$-th residual identifies priority channels for expansion:
\[
N^{(K)}_i = \|R^{(K)}_{i,*}\|_1,\quad (R^{(K)}_\gamma)_{i,*} = R^{(K)}_{i,*} \cdot \mathbf{1}\{N^{(K)}_i \ge \tau^{(K)}_\gamma\}
\]
where $\tau^{(K)}_\gamma$ is the $\gamma$-percentile threshold. This offers an explicit trade-off between fidelity and compute cost [2203.14645].

Additionally, ensemble (parallel) expansion fuses the expanded kernels into a single wide kernel to enable concurrent execution. For both quantized weights and activations, only terms with $k_1 + k_2 \leq K + 1$ are retained, and all relevant convolutions or matrix multiplications are batched [2203.14645].

## 4. High-Order Binary Filtering and Training Dynamics

In binarized networks, both filters and input patches are recursively quantized as $\{X^{(i)}, W^{(i)}\}$. The high-order binary filtering thus approximates the matrix product as a sum of scaled binary matrix multiplications:
\[
Y_r \approx \sum_{i=1}^k \sum_{j=1}^k \alpha_{i} \beta_{j} \left[\operatorname{sign}(R_W^{(i-1)}) \ast \operatorname{sign}(R_X^{(j-1)})\right]
\]
The backward propagation employs the straight-through estimator (STE) for the derivative of the sign function, propagating gradients through residual subtraction in the usual manner [1708.08687].

## 5. Empirical Performance and Accuracy/Latency Trade-offs

Empirical studies demonstrate that HORQ achieves superior accuracy-latency trade-offs compared to conventional one-shot quantization and binarization. On benchmarks like ResNet-50, MobileNet-V2, and EfficientNet-B0, REx (an instance of HORQ) at $K=2$ with $\gamma=50\%$ in W4/A6 format matches or surpasses prior data-free W6/A6 methods while using approximately $1.5\times$ fewer bit-ops. On NLP tasks such as GLUE using BERT-Base, a single 3-bit sparse residual ($\gamma=33\%$) with HORQ bridges the gap to full precision and outperforms uniform, logarithmic, SQuant, and SPIQ quantizers [2203.14645].

On smaller models and datasets, order-two HORQ reduces MNIST error from 1.96% (XNOR) to 1.25%, and CIFAR-10 accuracy improves from ~73% (XNOR) to ~75%, with speedup factors of $30\times$ over full precision. Increasing order $K$ reduces quantization error and recovers more accuracy, but each new binary term roughly halves the computational speedup [1708.08687].

## 6. Applications and Hardware Considerations

HORQ's structure makes it amenable to a variety of hardware-centric optimizations, including fused bit-shift and accumulate routines. The ensemble expansion enables near-theoretical throughput, adding $<1\%$ runtime overhead for parallel expansion [2203.14645]. For vision and NLP workloads, HORQ supports both per-channel and group-sparse expansions, allowing adaptation to device-specific constraints and delivering a smooth accuracy–latency frontier. Unlike one-shot quantization methods, HORQ enables fine-grained control over the accuracy-speed trade-off by tuning $K$ and $\gamma$.

## 7. Significance and Comparative Assessment

HORQ generalizes simple quantization by providing a framework that is simultaneously data-free, provably convergent in error, and highly configurable for hardware deployment. It outperforms classical binarization and fixed-bit quantization methods in both empirical accuracy and cost-efficiency, particularly when augmented with group-sparsity and ensemble parallelism [2203.14645][1708.08687]. A plausible implication is that HORQ-like strategies are likely to become a foundational building block for deploying large-scale models on resource-constrained accelerators, especially when privacy-preserving, data-free quantization is required.

Source: https://www.emergentmind.com/topics/high-order-residual-quantization-horq