---
title: 'DiBA-Greedy: Structured Matrix Compression'
url: https://www.emergentmind.com/topics/diba-greedy
type: topic
---

# DiBA-Greedy: Structured Matrix Compression

Searching arXiv for the cited DiBA paper to ground the article in the current record.
DiBA-Greedy is the alternating solver introduced for fitting the **Diagonal and Binary Matrix Approximation (DiBA)** to a dense matrix \(A \in \mathbb{R}^{m\times n}\). In that formulation, a dense weight matrix is approximated by a structured product of three real diagonal matrices and two binary \(0/1\) matrices, and DiBA-Greedy optimizes the resulting mixed continuous-discrete reconstruction problem by combining closed-form least-squares updates for the diagonal factors with exact one-bit improvement tests for the binary factors [2605.05994].

## 1. Definition and placement within DiBA

DiBA-Greedy belongs to the compression framework in which a dense matrix is replaced by the factorization
\[
\widehat A = D_1 B_1 D_2 B_2 D_3,
\]
with
\[
D_1\in\mathcal{D}_m,\quad B_1\in\{0,1\}^{m\times k},\quad D_2\in\mathcal{D}_k,\quad B_2\in\{0,1\}^{k\times n},\quad D_3\in\mathcal{D}_n.
\]
Here \(\mathcal{D}_p\) denotes the set of real \(p\times p\) diagonal matrices, and the diagonal entries are parameterized as
\[
(d_1)_i=(D_1)_{ii},\qquad (d_2)_\ell=(D_2)_{\ell\ell},\qquad (d_3)_j=(D_3)_{jj}.
\]
The representation is intended for compressing dense neural-network weights such as linear layers, embeddings, attention projections, and \(1\times1\) convolutions [2605.05994].

A useful expansion is
\[
B_1D_2B_2 =\sum_{\ell=1}^{k}(d_2)_\ell (B_1)_{:,\ell}(B_2)_{\ell,:},
\]
so that
\[
\widehat A_{ij} =(d_1)_i(d_3)_j \sum_{\ell=1}^{k}(d_2)_\ell (B_1)_{i\ell}(B_2)_{\ell j}.
\]
Accordingly, \(\widehat A\) is a sum of \(k\) scaled rank-one binary patterns, with row and column scaling corrections, and
\[
\operatorname{rank}(\widehat A)\le k.
\]

Within this framework, DiBA-Greedy is the fitting procedure rather than the factorization family itself. The factorization family is **DiBA**; the downstream retuning procedure that freezes the binary matrices and updates only the diagonal entries is **DiBARD**; and DiBA-Greedy is the solver that searches over both the binary and diagonal factors under the Frobenius reconstruction objective.

## 2. Factorization model, storage accounting, and arithmetic interpretation

The intermediate dimension \(k\) is the main control parameter governing the storage-accuracy trade-off. Larger \(k\) yields more expressive approximations but increases storage [2605.05994].

The theoretical storage accounting assumes that each dense scalar uses \(Q\) bits, each binary entry in \(B_1,B_2\) uses 1 bit, and each diagonal entry in \(D_1,D_2,D_3\) uses \(Q\) bits. Under that model, a dense matrix requires
\[
S_{\mathrm{dense}}(Q)=Qmn \ \text{bits},
\]
while DiBA requires
\[
S_{\mathrm{DiBA}}(k;Q)=k(m+n)+Q(m+k+n)\ \text{bits}.
\]
The theoretical storage ratio is therefore
\[
\rho_{\mathrm{DiBA}}(k;Q)=\frac{k(m+n)+Q(m+k+n)}{Qmn}.
\]

The same factorization induces a specific arithmetic decomposition for matrix-vector multiplication. A dense product \(y=Ax\) uses \(mn\) floating-point multiplications, whereas DiBA computes
\[
y=D_1B_1D_2B_2D_3x
\]
from right to left. If multiplication by \(B_1,B_2\) is implemented as selection-and-summation rather than floating-point multiplication, the floating-point multiplies arise only from the three diagonal scalings, for a total of
\[
m+k+n
\]
floating-point multiplications per input vector [2605.05994].

This arithmetic interpretation is central to the method’s compression rationale. DiBA does not discretize each entry independently. Instead, it uses shared binary mixing patterns together with diagonal scalings, so the optimization target is a structured low-storage approximation rather than ordinary scalar quantization.

## 3. Optimization objective and alternating solver structure

DiBA-Greedy addresses the mixed continuous-discrete optimization problem
\[
\min_{D_1,D_2,D_3,B_1,B_2} \left\|A-D_1B_1D_2B_2D_3\right\|_F^2,
\]
subject to
\[
D_1\in\mathcal{D}_m,\quad B_1\in\{0,1\}^{m\times k},\quad D_2\in\mathcal{D}_k,\quad B_2\in\{0,1\}^{k\times n},\quad D_3\in\mathcal{D}_n.
\]
The problem is nonconvex because of the product of factors and mixed continuous-discrete because \(D_1,D_2,D_3\) are real-valued whereas \(B_1,B_2\) are binary [2605.05994].

The solver is described as an alternating scheme with the following outer loop:

1. Initialize \(D_1,D_2,D_3\) and random \(B_1,B_2\).
2. Refit \(D_1,D_2,D_3\).
3. Repeat:
   - update \(B_1\) using a greedy bit-flip routine,
   - refit \(D_1\),
   - update \(B_2\) via the same routine applied to \(B_2^\top\),
   - refit \(D_3\),
   - refit \(D_2\).
4. Stop when
   \[
   c_1+c_2=0,
   \]
   where \(c_1,c_2\) are the numbers of accepted flips in the \(B_1\) and \(B_2\) updates.

The solver inputs include the matrix \(A\), the intermediate dimension \(k\), a flip tolerance \(\tau\ge 0\), and a row-batch size \(b\ge 1\).

When the binary factors are fixed, the diagonal factors are updated by least squares. For \(D_1\), letting
\[
G=B_1D_2B_2D_3,
\]
the subproblem is
\[
\min_{D_1\in\mathcal{D}_m}\left\|A-D_1G\right\|_F^2,
\]
which decomposes row-wise as
\[
(d_1)_i =\frac{\langle A_{i,:},G_{i,:}\rangle}{\left\|G_{i,:}\right\|_2^2}.
\]
If \(\|G_{i,:}\|_2^2=0\), then \((d_1)_i\) is set to zero.

For \(D_3\), letting
\[
G=D_1B_1D_2B_2,
\]
the corresponding update is
\[
(d_3)_j =\frac{\langle A_{:,j},G_{:,j}\rangle}{\left\|G_{:,j}\right\|_2^2}.
\]

For the middle diagonal factor \(D_2\), with
\[
G_L=D_1B_1,\qquad G_R=B_2D_3,
\]
the subproblem is
\[
\min_{D_2\in\mathcal{D}_k}\left\|A-G_LD_2G_R\right\|_F^2.
\]
Its normal equations are
\[
Fd_2=b,
\]
with
\[
F=(G_L^\top G_L)\odot(G_RG_R^\top), \qquad b_r=(G_L)_{:,r}^\top A (G_R)_{r,:}^\top,
\]
where \(\odot\) denotes the Hadamard product. The implementation uses a regularized solve,
\[
(F+\lambda I)d_2=b.
\]

## 4. Binary-factor optimization and the RowGreedy mechanism

The binary updates are cast into a common subproblem of the form
\[
\min_{B\in\{0,1\}^{p\times q}} \left\|\widetilde A-D_LBG_R\right\|_F^2,
\]
where \(\widetilde A\in\mathbb{R}^{p\times t}\), \(D_L\in\mathcal{D}_p\), \(B\in\{0,1\}^{p\times q}\), and \(G_R\in\mathbb{R}^{q\times t}\) [2605.05994].

The two DiBA binary factors fit this template as follows. Updating \(B_1\) uses
\[
\widetilde A=A,\quad B=B_1,\quad D_L=D_1,\quad G_R=D_2B_2D_3,
\]
while updating \(B_2\) is handled through \(B_2^\top\), using
\[
\left\|A^\top-D_3B_2^\top D_2B_1^\top D_1\right\|_F^2,
\]
with
\[
\widetilde A=A^\top,\quad B=B_2^\top,\quad D_L=D_3,\quad G_R=(D_1B_1D_2)^\top.
\]

To flip a single binary entry \(B_{ij}\), the method defines
\[
\delta_{ij}=1-2B_{ij}\in\{-1,+1\},
\]
so the flipped matrix is
\[
B^{(ij)}=B+\delta_{ij}e_ie_j^\top.
\]
The exact change in squared Frobenius error is
\[
\Delta_{ij} = \left\|\widetilde A-D_LB^{(ij)}G_R\right\|_F^2 -\left\|\widetilde A-D_LBG_R\right\|_F^2
=2(1-2B_{ij})(Y_{ij}-Z_{ij})+h_ir_j .
\]
This is the exact one-bit improvement test. The auxiliary quantities are
\[
H=G_RG_R^\top\in\mathbb{R}^{q\times q},\qquad h=d\odot d\in\mathbb{R}^p,\qquad r=\operatorname{diag}(H)\in\mathbb{R}^q,
\]
\[
Y=(h\odot_{\mathrm{row}}B)H\in\mathbb{R}^{p\times q},\qquad Z=d\odot_{\mathrm{row}}(\widetilde A G_R^\top)\in\mathbb{R}^{p\times q}.
\]
A flip is accepted only if
\[
\Delta_{ij}<-\tau.
\]

The paper exploits the row-wise decomposition
\[
\sum_{i=1}^{p}\left\|\widetilde A_{i,:}-d_iB_{i,:}G_R\right\|_2^2.
\]
Because of this decomposition, flips from distinct rows, with at most one flip per row, do not affect one another’s objective differences. This yields the `RowGreedy` routine. The workspace is initialized by computing \(H\), \(h\), \(r\), \(Y\), and \(Z\); then, for each row \(i\), the routine finds
\[
j_i\leftarrow\arg\min_{1\le j\le q}\Delta_{ij},
\qquad
v_i\leftarrow \Delta_{i j_i}.
\]
While
\[
\min_i v_i<-\tau,
\]
the routine forms the eligible set
\[
\mathcal E=\{i\in\{1,\ldots,p\}: v_i<-\tau\},
\]
selects up to \(b\) rows with the most negative \(v_i\), flips the corresponding bits, and updates only the affected row of \(Y\):
\[
Y_{i,:}\leftarrow Y_{i,:}+s h_i H_{j,:},
\]
where \(s=1-2B_{ij}\). This row-local update is the main implementation device for making greedy bit updates practical. For Experiment 1, the reported setting is
\[
b=1024.
\]

## 5. Theoretical properties, reconstruction behavior, and downstream use

The paper states a monotonicity property for the reconstruction objective: assuming exact least-squares refits and exact flip-difference evaluations, every primitive update in DiBA-Greedy is monotone non-increasing for the reconstruction objective [2605.05994]. Least-squares diagonal refits cannot increase the objective, and each accepted one-bit flip satisfies \(\Delta_{ij}<-\tau\), so it strictly decreases the current quadratic subproblem. The stated caveats are numerical roundoff and the regularization used in the \(D_2\) solve. The method does not provide global convergence to a stationary point or optimality guarantees for the full mixed problem.

The reconstruction metric used in the experiments is
\[
SNR = 10\log_{10} \left(\frac{\|A\|_F^2}{\|A-\widehat A\|_F^2}\right).
\]
Experiment 1 evaluates **40 dense matrices** extracted from public pretrained models, grouped into attention-related, FFN/projection, embedding, and conv1x1. DiBA-Greedy was run with
\[
k\in\{8,16,32,64,128,256,512,1024\},
\]
using \(Q=16\) for storage accounting, float32 arithmetic in the solver, flip tolerance
\[
\tau=10^{-6},
\]
random seed 0, and storage-ratio cap
\[
\rho_{\mathrm{DiBA}}(k;16)\le 0.75.
\]
Of 320 planned runs, 317 were completed.

The main reported finding is that **SNR is monotone nondecreasing in \(k\) for all 40 matrix curves over completed points**. Average SNR across matrices increased from **0.70 dB at \(k=8\)** to **16.35 dB at \(k=1024\)**, with the latter averaged over the 37 matrices not excluded by the storage-ratio cap. At \(k=1024\), category-wise mean SNRs were **21.3 dB** for embedding, **19.6 dB** for \(1\times1\) convolutions, **13.6 dB** for attention-related matrices, and **11.5 dB** for FFN/projection matrices.

DiBA-Greedy also supplies the binary structure later used by **DiBARD**. In DiBARD, \(B_1\) and \(B_2\) are frozen and only \(d_1,d_2,d_3\) are updated on downstream data. In the two component-replacement studies reported in the paper, DiBARD improves DistilBERT/WikiText masked-token accuracy from **0.4447 to 0.5210** and Speech Commands test accuracy for an Audio Spectrogram Transformer from **0.7684 to 0.9781** without reoptimizing the binary factors [2605.05994].

## 6. Limitations, scope, and methodological significance

DiBA-Greedy is a local greedy solver rather than a global optimizer. The paper is explicit that its storage ratios are theoretical, assuming bit-packed binary matrices, and that it does not provide measured runtime or packed-kernel benchmarks [2605.05994]. It also notes that approximation quality can vary across matrix types, that performance may depend on random initialization of the binary factors, and that downstream evaluations are limited in seeds and tasks.

These limitations delimit the method’s scope. DiBA-Greedy is most naturally interpreted as a post hoc compression procedure for dense weight matrices when the objective is a quadratic reconstruction loss and when one wants a structured approximation with a tunable parameter \(k\), exact local binary decisions, and closed-form continuous refits. Its exact bit-flip machinery depends on the simple reconstruction objective and does not transfer directly to arbitrary downstream losses; that is why DiBARD freezes the binary structure found by DiBA-Greedy and retunes only the diagonal factors during adaptation.

Within that scope, the method is notable for the transparency of its optimization primitives. The diagonal subproblems are solved by least squares, the binary steps are accepted only when the exact one-bit objective difference is negative beyond tolerance, and the factorization itself has direct interpretations in terms of storage ratio, rank bound, and arithmetic decomposition. This suggests a compression strategy centered on structured binary mixing and diagonal scaling rather than per-entry discretization, with DiBA-Greedy serving as the fitting mechanism for that representation.

Source: https://www.emergentmind.com/topics/diba-greedy