DiBA: Diagonal and Binary Matrix Approximation for Neural Network Weight Compression

Abstract: In this paper, we propose DiBA (Diagonal and Binary Matrix Approximation), a compact matrix factorization for neural network weight compression. Many components of modern networks, including linear layers, $1\times1$ convolutions, attention projections, and embedding layers, have dense matrix weights. DiBA approximates $A\in\mathbb{R}^{m\times n}$ by $\widehat A=D_1B_1D_2B_2D_3$, where $D_1,D_2,D_3$ are diagonal matrices and $B_1,B_2$ are $0/1$ binary matrices. The intermediate dimension $k$ controls the trade-off between theoretical storage and approximation accuracy. For matrix-vector products, DiBA decomposes dense multiplication into three element-wise scaling operations and two binary mixing operations, reducing the floating-point multiplication count from $mn$ to $m+k+n$. For optimization, we introduce DiBA-Greedy, an alternating solver that combines closed-form least-squares updates for the diagonal factors with exact one-bit improvement tests for the binary factors. We also introduce DiBARD (DiBA with Retuning only Diagonal factors), which replaces dense-matrix layers by DiBA factors, freezes the binary matrices, and retunes only the diagonal entries on downstream data. This preserves compact binary mixing without discrete search during adaptation. On 40 dense weight matrices extracted from public pretrained models, DiBA-Greedy yields consistent SNR improvements as the theoretical storage ratio increases. After DiBA replacement in two component-replacement studies, DiBARD improves DistilBERT/WikiText masked-token accuracy from 0.4447 to 0.5210 and Speech Commands test accuracy for an Audio Spectrogram Transformer from 0.7684 to 0.9781 without reoptimizing the binary factors.

• The paper introduces a novel DiBA method that approximates weight matrices using interleaved diagonal and binary factors to achieve high compression.
• The methodology employs a DiBA-Greedy solver that alternates between closed-form diagonal updates and greedy binary flips to enhance approximation quality.
• Empirical results show significant storage reduction with near-original or improved accuracy through a lightweight diagonal retuning process.

DiBA: Diagonal and Binary Matrix Approximation for Neural Network Weight Compression

Overview

"DiBA: Diagonal and Binary Matrix Approximation for Neural Network Weight Compression" (2605.05994) introduces a structured, highly parameter-efficient matrix factorization method for compressing neural network weights. The DiBA method approximates a dense weight matrix $A \in \mathbb{R}^{m \times n}$ as $\widehat{A} = D_1 B_1 D_2 B_2 D_3$, where $D_1, D_2, D_3$ are diagonal matrices and $B_1, B_2$ are binary (0/1) matrices. The central hyperparameter, the intermediate dimension $k$, tunes the compression-accuracy trade-off and theoretical storage.

The paper provides:

• A derivation of the DiBA parameterization and a theoretical storage analysis,
• The DiBA-Greedy solver, which alternates between closed-form diagonal updates and greedy one-bit improvements of binary terms,
• DiBARD, a protocol for task adaptation that retunes only the diagonal factors while freezing the binary support,
• Extensive empirical analysis on matrices from contemporary models and practical downstream evaluations, notably showing large performance recovery from diagonal-only retuning after extreme storage reduction.

DiBA Formulation and Theoretical Analysis

DiBA factorizes $A$ as a product of three real diagonal matrices interleaved with two binary mixing matrices:

Figure 1: DiBA provides a structured approximation to a dense matrix using three real diagonal factors and two 0/1 binary factors; $k$ controls the storage–accuracy trade-off.

This design induces a highly-structured low-rank approximation. The binary matrices create shared mixing patterns, while all magnitude and sign information is delegated to the diagonals. For $A \approx D_1 B_1 D_2 B_2 D_3$, the storage—the bits required to represent $\widehat{A}$—is dominated by $m+k+n$ real diagonal entries (each requiring $\widehat{A} = D_1 B_1 D_2 B_2 D_3$0 bits, typically $\widehat{A} = D_1 B_1 D_2 B_2 D_3$1 or $\widehat{A} = D_1 B_1 D_2 B_2 D_3$2), and $\widehat{A} = D_1 B_1 D_2 B_2 D_3$3 bits for the binary factors.

The storage ratio relative to a dense $\widehat{A} = D_1 B_1 D_2 B_2 D_3$4-bit matrix is:

$\widehat{A} = D_1 B_1 D_2 B_2 D_3$5

$\widehat{A} = D_1 B_1 D_2 B_2 D_3$6 thus directly interpolates between accuracy and compression.

On matrix–vector products, DiBA reduces floating-point multiplications from $\widehat{A} = D_1 B_1 D_2 B_2 D_3$7 (dense) to $\widehat{A} = D_1 B_1 D_2 B_2 D_3$8 (DiBA), moving the complexity into selection/summation with binary factors—well-suited for modern hardware.

Optimization: DiBA-Greedy

Fitting DiBA is a mixed discrete–continuous optimization. DiBA-Greedy alternates:

• Diagonal refits: Each $\widehat{A} = D_1 B_1 D_2 B_2 D_3$9 is updated in closed-form by solving least-squares subproblems.
• Binary updates: Each element of $D_1, D_2, D_3$0, $D_1, D_2, D_3$1 is tested for one-bit flip improvement using exact computation of the Frobenius norm difference. Only flips improving the objective beyond a threshold are applied. Monotonicity is preserved for the local subproblems.

Global optimality is not guaranteed, but each flip precisely improves the corresponding local binary subproblem.

Downstream Adaptation: DiBARD

Replacing dense weights with DiBA factors can degrade end-task (e.g., prediction) accuracy due to the proxy nature of matrix reconstruction losses. DiBARD circumvents the highly nonconvex mixed optimization by only retuning the diagonal matrices $D_1, D_2, D_3$2 for each replaced component, freezing $D_1, D_2, D_3$3. This makes adaptation a lightweight, fully continuous problem, feasible with standard gradient-based optimizers and compatible with arbitrary downstream losses/architectures.

This approach preserves maximal storage reduction (no new binary search), while endowing a degree of expressivity for per-row/column/channel scaling, yielding substantial empirical recovery in downstream accuracy.

Empirical Results

Matrix Approximation on Pretrained Model Weights

A large-scale sweep across matrix types (attention, FFN/projection, embedding, $D_1, D_2, D_3$4 conv) and models demonstrates that SNR (in dB) increases monotonically with storage, controlled by $D_1, D_2, D_3$5, across all examined instances.

Figure 2: SNR versus realized DiBA storage ratio for the 40 selected matrices in Experiment 1, showing monotonic improvement with more storage; color groups reflect functional categories.

For most weights, even aggressive compression ($D_1, D_2, D_3$6) yields SNR > 10 dB. Embedding matrices tend to be more compressible than projection or attention matrices at equivalent storage.

DistilBERT/WikiText and Audio Spectrogram Transformer

DistilBERT Embedding/Projection

With $D_1, D_2, D_3$7, DiBA reduces the tied embedding matrix's storage to $D_1, D_2, D_3$8. Before retuning, slight degradation is evident, but retuning only the diagonals (DiBARD) recovers and even surpasses the original test accuracy (0.5210 > 0.4906). Compared to symmetric Int4 quantization (PTQ), DiBA operates at lower storage and achieves higher accuracy after diagonal retuning.

AST Speech Commands

On the AST model, DiBA is applied to all Q/K/V/output projections ($D_1, D_2, D_3$9 layers, $B_1, B_2$0). At an aggressive $B_1, B_2$1 (nearly $B_1, B_2$2 compression), naive DiBA degrades accuracy. Diagonal-only retuning (DiBARD) recovers nearly all of the lost accuracy, reaching 0.9781 test accuracy ($B_1, B_2$3 below original).

Figure 3: Held-out accuracy versus target-component theoretical storage ratio for DiBA and quantization baselines; arrows indicate improvement from diagonal-only retuning.

Per-layer SNR

Layerwise analysis on AST Q/K/V/out projections reveals that Q/K projections are generally more compressible (higher SNR) than value or output layers at fixed compression, consistent across layers.

Figure 4: Per-layer reconstruction SNR for AST attention projections, with Q/K layers showing higher SNR than Value/Output at fixed $B_1, B_2$4 and layer shape.

Implications and Future Directions

DiBA's core innovation is the extreme modularity between mixing structure (binary) and scaling (diagonal), enabling highly efficient, interpretable, and hardware-synergistic compression. The results indicate that large classes of pretrained model matrices can be compressed far beyond conventional quantization regimes, with recovery of task accuracy by only retuning $B_1, B_2$5 scalars, even when $B_1, B_2$6.

Practical implications include:

• Model distribution and deployment: Significant reduction in storage and memory footprint, which supports edge and resource-constrained inference without retraining.
• Transfer and adaptation: The fixed binary structure is compatible with fast, lightweight adaptation to diverse tasks by retuning only diagonals.
• Potential for custom inference kernels: While no hardware kernel is implemented, the structure is highly favorable for efficient implementations on CPU/FPGA/ASIC, given the predominance of selection/summation and drastically reduced floating-point multiplications.

Theoretical considerations involve the representation power of DiBA relative to classical low-rank, binary, or quantized factorization. The ability to explicitly decompose and store binary mixing separately from continuous scaling gives rise to new perspectives on matrix redundancy and structured pruning.

Conclusion

DiBA presents a highly flexible, binary–diagonal parameterization for neural weight compression. Empirical results demonstrate that, given an appropriate intermediate dimension and downstream diagonal-only retuning, dramatic storage reduction is feasible with minimal-to-no degradation, and sometimes improvement, in downstream performance. The DiBA method generalizes over prior quantization and low-rank schemes and invites future research on optimized inference kernels, integration with broader compression pipelines, and theoretical analyses of expressivity and optimization.