Block-Circulant Matrix Compression

• Block-circulant matrix compression is a technique that imposes circulant structure on matrix blocks, reducing storage costs and enabling efficient FFT-based operations.
• It is applied in signal processing, neural network inference, and parameter-efficient fine-tuning, providing significant computational savings.
• Empirical studies report up to 32× fewer FLOPs with minimal accuracy loss, balancing compression benefits with model expressivity.

Block-circulant matrix compression exploits the algebraic and computational properties of block-circulant matrices to achieve parameter reduction and fast algebraic operations in high-dimensional linear systems. By constraining matrices to have (block-)circulant structure, both storage and computation cost scale more favorably than for general unstructured matrices, while retaining sufficient expressivity for a wide range of applications. Block-circulant compression underpins efficient signal processing, low-overhead neural network inference, parameter-efficient fine-tuning for large models, and fast linear algebra in quaternionic/tensor domains.

1. Mathematical Foundations

A block-circulant matrix is defined as a matrix that can be partitioned into blocks, each of which is itself circulant. Formally, let $n=b\cdot m$ and partition an $n\times n$ matrix $C$ into $b\times b$ grid of $m\times m$ blocks: $$C = [C_{ij}]_{i,j=0}^{b-1},~~C_{ij}\in\mathbb{R}^{m\times m}.$$ Each $C_{ij}$ is circulant, i.e., determined by a first-row vector $r^{(i,j)}\in\mathbb{R}^m$ such that

$[C_{ij}]_{p,q} = r^{(i,j)}_{(q - p) \bmod m}.$

This structure reduces the storage cost for each block from $m^2$ to $n\times n$0 parameters, yielding a global compression ratio of $n\times n$1 relative to an $n\times n$2 dense matrix (Xu et al., 2024).

For a signal $n\times n$3 partitioned into $n\times n$4 subvectors $n\times n$5, block-circulant multiplication is

$n\times n$6

with each $n\times n$7 implemented as a circular convolution. For neural networks or signal processing operators, this admits significant savings and enables direct exploitation of FFT-based algorithms (Ding et al., 1 May 2025, Valsesia et al., 2014).

2. Algorithmic Acceleration via FFT

Circulant and block-circulant matrices are diagonalizable through the (block) Discrete Fourier Transform (DFT). For a circulant block of size $n\times n$8 with kernel $n\times n$9, and any $C$0: $C$1 where $C$2 is elementwise multiplication. For a block-circulant $C$3, matvec reduces to $C$4 (with $C$5 block size), versus the $C$6 cost of dense multiplication (Ding et al., 1 May 2025).

In compressive sensing, block-circulant sensing can be realized by stacking row-wise circularly-shifted versions of a seed vector, and measurement computation is conducted via FFT/IFFT with pointwise multiplication, providing $C$7 acquisition (Valsesia et al., 2014). For quaternion and tensor settings, FFTs are performed independently on split complex parts and combined to recover quaternionic or higher-order results (Zheng et al., 2022).

3. Compression Metrics and Storage Benefits

Block-circulant compression achieves an overall parameter reduction by a factor equal to the block size. Explicitly, compressing an $C$8 weight to block size $C$9 results in

$b\times b$0

In neural models, this yields:

• $b\times b$1 fewer parameters than VeRA,
• $b\times b$2 fewer than LoRA,
• $b\times b$3 fewer floating point operations (FLOPs) than FourierFT, with comparable accuracy retained on standard benchmarks (Ding et al., 1 May 2025).

For quaternionic matrices, block-circulant-diagonalized storage demands $b\times b$4 (only the block-diagonal frontal slices), compared to $b\times b$5 entries for the uncompressed full matrix (Zheng et al., 2022).

4. Exact Transformations and Applications

Block-circulant structure enables mathematically exact commutation for signal transforms such as filtering, interpolation, or registration—up to a boundary of "corrupted" measurements dictated by the effective filter length (Valsesia et al., 2014). For neural inference under homomorphic encryption (HE), multiplying by block-circulant weight matrices is functionally equivalent to 1D circular convolution, minimizing expensive ciphertext rotations and multiplications (Xu et al., 2024).

In PEFT (Parameter-Efficient Fine-Tuning) for LLMs, block-circulant adapters (BCA) are trained as "delta-weight" modules and merged post-hoc into the base model. The FFT implementation provides both runtime and memory savings, and a stable-training heuristic rescales learning rates by block size to avoid optimizer divergence due to gradient scaling (Ding et al., 1 May 2025).

Quaternionic and tensor-valued applications exploit octonion-based diagonalization, using a unitary octonion DFT (e.g., $b\times b$6 with suitable octonion scalar) to entirely block-diagonalize block-circulant quaternion matrices. This allows decoupling of large tensor linear algebra operations into $b\times b$7 independent $b\times b$8 quaternion block solves (Zheng et al., 2022).

5. Optimization and Implementation Strategies

Selecting block size ($b\times b$9 or $m\times m$0) is a primary design choice, mediating a tradeoff between compression (and acceleration) and expressivity or downstream task accuracy. Recent methods implement layerwise block-size assignment using second-order approximations (Hessian-based sensitivity analysis) to minimize overall task loss while meeting computational latency constraints (Xu et al., 2024).

In HE-based inference, customized encoding schemes (e.g., CirEncode) nest coefficient and SIMD encodings to further optimize the packing of circulant-block matrix multiplications within a single ciphertext operation (Xu et al., 2024). Layer fusion absorbs operations such as BatchNorm and residual connections into the block-circulant representation, preserving the compressed structure through end-to-end computation.

In PEFT, block-circulant adapters are integrated as lightweight modules, typically attached to projection matrices in attention or feedforward blocks. Optimizers must scale learning rates inversely with block-size—this is theoretically justified by gradient magnitude analysis and is sufficient for stable convergence without additional normalization or regularization (Ding et al., 1 May 2025).

6. Applications and Empirical Performance

Block-circulant compression is empirically validated in:

• Compressive sensing: Filtered, interpolated, or wavelet-transformed measurements can be performed exactly or with controlled boundary corruption, directly in the compressed domain (Valsesia et al., 2014).
• Neural network inference under HE: PrivCirNet achieves up to $m\times m$1 speedup in DNN private inference (TinyImageNet, ImageNet) and up to $m\times m$2 accuracy improvement over structured-pruning baselines at the same HE runtime (Xu et al., 2024).
• LLM fine-tuning: BCA achieves $m\times m$3 parameter and $m\times m$4 FLOP reductions compared to leading PEFT approaches while maintaining accuracy within $m\times m$5 of full fine-tuning (Ding et al., 1 May 2025).
• Quaternion tensors: Octonion-based diagonalization enables nearly $m\times m$6 compute reduction in quaternionic T-products, with Tensor contractions decomposing into independent block computations (Zheng et al., 2022).

Application Domain	Compression Ratio	Empirical Speedup	Accuracy Retention
LLM fine-tuning	$m\times m$7	$m\times m$8 fewer FLOPs	$m\times m$9 of full-tune
HE-based DNN inference	$$C = [C_{ij}]_{i,j=0}^{b-1},~~C_{ij}\in\mathbb{R}^{m\times m}.$$0	$$C = [C_{ij}]_{i,j=0}^{b-1},~~C_{ij}\in\mathbb{R}^{m\times m}.$$1	$$C = [C_{ij}]_{i,j=0}^{b-1},~~C_{ij}\in\mathbb{R}^{m\times m}.$$2 to $$C = [C_{ij}]_{i,j=0}^{b-1},~~C_{ij}\in\mathbb{R}^{m\times m}.$$3
Quaternion tensors	$$C = [C_{ij}]_{i,j=0}^{b-1},~~C_{ij}\in\mathbb{R}^{m\times m}.$$4	$$C = [C_{ij}]_{i,j=0}^{b-1},~~C_{ij}\in\mathbb{R}^{m\times m}.$$5 compute	exact

A plausible implication is that block-circulant compression serves as a unifying technique for structure-aware optimization across both classical and modern high-dimensional signal and learning systems, with parallel advances in algorithms for real, complex, and quaternionic/tensor regimes.

7. Theoretical Guarantees and Limitations

The commutation properties of (block-)circulant matrices under convolution and their partial commutation when combined with non-square submatrices provide rigorous guarantees for filter-style transformations, up to well-characterized edge effects (Valsesia et al., 2014). Diagonalization is exact in all cases where the block-circulant form is satisfied, including for quaternionic matrices in the octonion domain (Zheng et al., 2022).

However, the benefits of maximal compression may come at a modest expressivity penalty, with error rates increasing for excessively large block sizes—reported accuracy drops remain $$C = [C_{ij}]_{i,j=0}^{b-1},~~C_{ij}\in\mathbb{R}^{m\times m}.$$6 in LLM adaptation when $$C = [C_{ij}]_{i,j=0}^{b-1},~~C_{ij}\in\mathbb{R}^{m\times m}.$$7 is large, and parameter/FLOP savings asymptote (Ding et al., 1 May 2025). Block structure also imposes architectural constraints, requiring weights to be square-multiple partitionable for maximal benefit, though generalization to non-square or incomplete blockings is possible (Xu et al., 2024).

In summary, block-circulant matrix compression is foundational for scalable and efficient computation in high-dimensional linear systems, providing a rigorously justified tradeoff between compactness, algorithmic expediency, and model expressivity across signal processing, privacy-preserving inference, and large-model optimization (Valsesia et al., 2014, Zheng et al., 2022, Xu et al., 2024, Ding et al., 1 May 2025).