Block-Circulant Matrix Compression

• Block-circulant matrix compression is a technique that imposes circulant structure on matrix blocks, reducing storage costs and enabling efficient FFT-based operations.
• It is applied in signal processing, neural network inference, and parameter-efficient fine-tuning, providing significant computational savings.
• Empirical studies report up to 32× fewer FLOPs with minimal accuracy loss, balancing compression benefits with model expressivity.

Block-circulant matrix compression exploits the algebraic and computational properties of block-circulant matrices to achieve parameter reduction and fast algebraic operations in high-dimensional linear systems. By constraining matrices to have (block-)circulant structure, both storage and computation cost scale more favorably than for general unstructured matrices, while retaining sufficient expressivity for a wide range of applications. Block-circulant compression underpins efficient signal processing, low-overhead neural network inference, parameter-efficient fine-tuning for large models, and fast linear algebra in quaternionic/tensor domains.

1. Mathematical Foundations

A block-circulant matrix is defined as a matrix that can be partitioned into blocks, each of which is itself circulant. Formally, let $n=b\cdot m$ and partition an $n\times n$ matrix $C$ into $b\times b$ grid of $m\times m$ blocks: $$C = [C_{ij}]_{i,j=0}^{b-1},~~C_{ij}\in\mathbb{R}^{m\times m}.$$ Each $C_{ij}$ is circulant, i.e., determined by a first-row vector $r^{(i,j)}\in\mathbb{R}^m$ such that

$[C_{ij}]_{p,q} = r^{(i,j)}_{(q - p) \bmod m}.$

This structure reduces the storage cost for each block from $m^2$ to $m$ parameters, yielding a global compression ratio of $b$ relative to an $n\times n$ dense matrix (Xu et al., 2024).

For a signal $x\in \mathbb{R}^n$ partitioned into $b$ subvectors $x^{(j)}\in\mathbb{R}^m$, block-circulant multiplication is

$$y^{(i)} = \sum_{j=0}^{b-1} C_{ij} x^{(j)},\,\,\,i=0,\dots,b-1,$$

with each $C_{ij} x^{(j)}$ implemented as a circular convolution. For neural networks or signal processing operators, this admits significant savings and enables direct exploitation of FFT-based algorithms (Ding et al., 1 May 2025, Valsesia et al., 2014).

2. Algorithmic Acceleration via FFT

Circulant and block-circulant matrices are diagonalizable through the (block) Discrete Fourier Transform (DFT). For a circulant block of size $k$ with kernel $c\in\mathbb{R}^k$, and any $x\in\mathbb{R}^k$: $$\mathrm{circ}(c)\, x = \mathrm{IDFT}\left(\mathrm{DFT}(c) \circ \mathrm{DFT}(x)\right)$$ where $\circ$ is elementwise multiplication. For a block-circulant $B$, matvec reduces to $O(n\log p)$ (with $p$ block size), versus the $O(n^2)$ cost of dense multiplication (Ding et al., 1 May 2025).

In compressive sensing, block-circulant sensing can be realized by stacking row-wise circularly-shifted versions of a seed vector, and measurement computation is conducted via FFT/IFFT with pointwise multiplication, providing $O(n\log n)$ acquisition (Valsesia et al., 2014). For quaternion and tensor settings, FFTs are performed independently on split complex parts and combined to recover quaternionic or higher-order results (Zheng et al., 2022).

3. Compression Metrics and Storage Benefits

Block-circulant compression achieves an overall parameter reduction by a factor equal to the block size. Explicitly, compressing an $n\times n$ weight to block size $b$ results in

$$\text{Compression ratio}=\frac{\text{number of dense parameters}}{\text{number of circulant parameters}} = b.$$

In neural models, this yields:

• $14\times$ fewer parameters than VeRA,
• $16\times$ fewer than LoRA,
• $32\times$ fewer floating point operations (FLOPs) than FourierFT, with comparable accuracy retained on standard benchmarks (Ding et al., 1 May 2025).

For quaternionic matrices, block-circulant-diagonalized storage demands $O(m n p)$ (only the block-diagonal frontal slices), compared to $O(m n p^2)$ entries for the uncompressed full matrix (Zheng et al., 2022).

4. Exact Transformations and Applications

Block-circulant structure enables mathematically exact commutation for signal transforms such as filtering, interpolation, or registration—up to a boundary of "corrupted" measurements dictated by the effective filter length (Valsesia et al., 2014). For neural inference under homomorphic encryption (HE), multiplying by block-circulant weight matrices is functionally equivalent to 1D circular convolution, minimizing expensive ciphertext rotations and multiplications (Xu et al., 2024).

In PEFT (Parameter-Efficient Fine-Tuning) for LLMs, block-circulant adapters (BCA) are trained as "delta-weight" modules and merged post-hoc into the base model. The FFT implementation provides both runtime and memory savings, and a stable-training heuristic rescales learning rates by block size to avoid optimizer divergence due to gradient scaling (Ding et al., 1 May 2025).

Quaternionic and tensor-valued applications exploit octonion-based diagonalization, using a unitary octonion DFT (e.g., $F_p$ with suitable octonion scalar) to entirely block-diagonalize block-circulant quaternion matrices. This allows decoupling of large tensor linear algebra operations into $p$ independent $m\times n$ quaternion block solves (Zheng et al., 2022).

5. Optimization and Implementation Strategies

Selecting block size ($b$ or $p$) is a primary design choice, mediating a tradeoff between compression (and acceleration) and expressivity or downstream task accuracy. Recent methods implement layerwise block-size assignment using second-order approximations (Hessian-based sensitivity analysis) to minimize overall task loss while meeting computational latency constraints (Xu et al., 2024).

In HE-based inference, customized encoding schemes (e.g., CirEncode) nest coefficient and SIMD encodings to further optimize the packing of circulant-block matrix multiplications within a single ciphertext operation (Xu et al., 2024). Layer fusion absorbs operations such as BatchNorm and residual connections into the block-circulant representation, preserving the compressed structure through end-to-end computation.

In PEFT, block-circulant adapters are integrated as lightweight modules, typically attached to projection matrices in attention or feedforward blocks. Optimizers must scale learning rates inversely with block-size—this is theoretically justified by gradient magnitude analysis and is sufficient for stable convergence without additional normalization or regularization (Ding et al., 1 May 2025).

6. Applications and Empirical Performance

Block-circulant compression is empirically validated in:

• Compressive sensing: Filtered, interpolated, or wavelet-transformed measurements can be performed exactly or with controlled boundary corruption, directly in the compressed domain (Valsesia et al., 2014).
• Neural network inference under HE: PrivCirNet achieves up to $5\times$ speedup in DNN private inference (TinyImageNet, ImageNet) and up to $12\%$ accuracy improvement over structured-pruning baselines at the same HE runtime (Xu et al., 2024).
• LLM fine-tuning: BCA achieves $14\times$ parameter and $32\times$ FLOP reductions compared to leading PEFT approaches while maintaining accuracy within $0.1-0.3\%$ of full fine-tuning (Ding et al., 1 May 2025).
• Quaternion tensors: Octonion-based diagonalization enables nearly $1/p$ compute reduction in quaternionic T-products, with Tensor contractions decomposing into independent block computations (Zheng et al., 2022).

Application Domain	Compression Ratio	Empirical Speedup	Accuracy Retention
LLM fine-tuning	$\sim 14-16\times$	$32\times$ fewer FLOPs	$>95\%$ of full-tune
HE-based DNN inference	$\sim 2-8\times$	$1.3-5.0\times$	$+4.1\%$ to $+12\%$
Quaternion tensors	$p\times$	$1/p$ compute	exact

A plausible implication is that block-circulant compression serves as a unifying technique for structure-aware optimization across both classical and modern high-dimensional signal and learning systems, with parallel advances in algorithms for real, complex, and quaternionic/tensor regimes.

7. Theoretical Guarantees and Limitations

The commutation properties of (block-)circulant matrices under convolution and their partial commutation when combined with non-square submatrices provide rigorous guarantees for filter-style transformations, up to well-characterized edge effects (Valsesia et al., 2014). Diagonalization is exact in all cases where the block-circulant form is satisfied, including for quaternionic matrices in the octonion domain (Zheng et al., 2022).

However, the benefits of maximal compression may come at a modest expressivity penalty, with error rates increasing for excessively large block sizes—reported accuracy drops remain $<0.5\%$ in LLM adaptation when $p$ is large, and parameter/FLOP savings asymptote (Ding et al., 1 May 2025). Block structure also imposes architectural constraints, requiring weights to be square-multiple partitionable for maximal benefit, though generalization to non-square or incomplete blockings is possible (Xu et al., 2024).

In summary, block-circulant matrix compression is foundational for scalable and efficient computation in high-dimensional linear systems, providing a rigorously justified tradeoff between compactness, algorithmic expediency, and model expressivity across signal processing, privacy-preserving inference, and large-model optimization (Valsesia et al., 2014, Zheng et al., 2022, Xu et al., 2024, Ding et al., 1 May 2025).