---
title: Frequency-Aware Multi-Parameter Intra-Row Grouping
url: https://www.emergentmind.com/topics/frequency-aware-multi-parameter-intra-row-grouping
type: topic
---

# Frequency-Aware Multi-Parameter Intra-Row Grouping

Frequency-aware multi-parameter intra-row grouping is a structure-aware quantization strategy introduced in HBLLM, a wavelet-based high-fidelity 1-bit quantization method for large language models. This method combines frequency decomposition via the Haar transform with adaptive, band-specific grouping and aggregation to increase the capacity and accuracy of ultra-low-bit quantization while incurring negligible storage overhead [2512.00862].

## 1. Formal Definition and Notation

Given a full-precision weight matrix $W \in \mathbb{R}^{d \times m}$ of a linear layer, the method operates row-wise. For a selected row $w \in \mathbb{R}^m$, a one-dimensional Haar wavelet transform is applied to obtain the spectral coefficients:
$$
\hat{h} = \mathcal{H}(w) \in \mathbb{R}^m
$$
$\hat{h}$ is partitioned into low- and high-frequency components:
$$
\hat{h} = [\hat{h}^{(L)}, \hat{h}^{(H)}]
$$
with $\hat{h}^{(L)} = \mathcal{H}_{\text{low-pass}}(w) \in \mathbb{R}^{m/2}$ and $\hat{h}^{(H)} = \mathcal{H}_{\text{high-pass}}(w) \in \mathbb{R}^{m/2}$. Each frequency band $f \in \{L, H\}$ is independently partitioned into two groups (dense/sparse) via a threshold $t^{(f)}$ selected from a discrete candidate set, yielding four final subgroups per row. Thresholds are set per-row and per-band by minimizing local quantization error, reflecting diverse spectral patterns across rows and bands.

## 2. Mathematical Formulation and Quantization Workflow

Within each frequency band $f$, the absolute values of coefficients $|\hat{h}^{(f)}|$ are sorted, and a set of $C$ candidate percentiles $P = \{p_1, ..., p_C\}$ is chosen. For each candidate $p_k$, threshold $t_k$ is defined as the $p_k$-th percentile, forming two groups:
- $G_k^{(f,1)} = \{i : |\hat{h}_i^{(f)}| \leq t_k\}$
- $G_k^{(f,2)} = \{i : |\hat{h}_i^{(f)}| > t_k\}$

Group-wise means are computed:
$$
\mu^{(f,g)} = \frac{1}{|G^{(f,g)}|}\sum_{i\in G^{(f,g)}} \hat{h}_i^{(f)},\quad g\in\{1,2\}
$$
A single rowwise scale $\alpha$ (or optionally per-group) is used. 1-bit quantization is applied:
$$
\hat{h}_i^{(f)}(B) = \alpha \cdot \text{sign}(\hat{h}_i^{(f)} - \mu^{(f,g)})
$$
The optimal threshold $k^*$ is selected to minimize reconstruction error $E^{(f)}_k$, aggregating within-group squared errors. The final grouping is $G_{k^*}^{(f,1)}, G_{k^*}^{(f,2)}$ with means $\mu_{k^*}^{(f,1)}, \mu_{k^*}^{(f,2)}$.

To reduce mean-storage overhead, HBLLM can employ mean sharing within each band:
$$
\mu^{(f)}_{\text{shared}} = \frac{|G^{(f,1)}|\mu^{(f,1)} + |G^{(f,2)}|\mu^{(f,2)}}{|G^{(f,1)}| + |G^{(f,2)}|}
$$
This reduces per-weight storage by approximately $0.25$ bits/weight without negatively affecting, and sometimes slightly improving, perplexity (see Table 3c in [2512.00862]).

## 3. Algorithm Workflow and Pseudocode

A single-row grouping and quantization pass is implemented via the following routine, omitting salient columns (which are handled separately):

```python
def RowFreqGroup(w, C, share_means=True):
    # 1. Haar transform
    h = Haar1D(w)      # h splits into h^(L) and h^(H)
    for f in {L, H}:   # two frequency bands
        c_abs = sorted(abs(h_f))
        best_err = float('inf')
        for p in linspace(0, 100, C):
            t = percentile(c_abs, p)
            G1 = [i for i in range(len(h_f)) if abs(h_f[i]) <= t]
            G2 = [i for i in range(len(h_f)) if abs(h_f[i]) > t]
            mu1 = mean([h_f[i] for i in G1])
            mu2 = mean([h_f[i] for i in G2])
            # alpha can be precomputed (row or group); assume rowwise
            B1 = [alpha * sign(h_f[i] - mu1) for i in G1]
            B2 = [alpha * sign(h_f[i] - mu2) for i in G2]
            err = sum((h_f[i] - B1[i] - mu1)**2 for i in G1) + \
                  sum((h_f[i] - B2[i] - mu2)**2 for i in G2)
            if err < best_err:
                best_err, best_groups = err, (G1, G2, mu1, mu2)
        G1_star, G2_star, mu1_star, mu2_star = best_groups
        if share_means:
            mu_shared = (len(G1_star)*mu1_star + len(G2_star)*mu2_star)/(len(G1_star)+len(G2_star))
            mu1_star = mu2_star = mu_shared
        # store groups and means
        for i in G1_star: h_f_B[i] = alpha * sign(h_f[i] - mu1_star)
        for i in G2_star: h_f_B[i] = alpha * sign(h_f[i] - mu2_star)
    return inverse_Haar([h_L_B, h_H_B])
```

In practice, vectorized implementations batch multiple rows, and salient columns are skipped or assigned via FillAvg prior to the Haar step (see Algorithm 1 and Fig. 2 in [2512.00862]).

## 4. Computational and Storage Complexity

The overall complexity per row is $O(m + Cm)$, dominated by $O(Cm)$ due to scanning $C$ thresholds per frequency band ($C \approx 40$ is constant):
- Haar 1D transform per row: $O(m)$
- Threshold enumeration: $O(Cm)$ per row, $O(dCm)$ for $d$ rows

The storage overhead per row is minimal, requiring either $4$ floats/row (two means per band) or $2$ floats/row (if mean sharing is used). For typical $m$, extra per-weight storage is negligible, e.g., $(4 \times 32)/m \approx O(1/m)$ bits/weight. Table 1 reports an average $W$-bits of $1.09$ for the HBLLM-row scheme and $1.00$ for HBLLM-col [2512.00862].

## 5. Comparative Analysis with Prior Grouping Schemes

Several baselines are contrasted:
- **BiLLM (intra-row global):** Employs a single split per row, independent of frequency.
- **ARB-LLM$_x$ (channel-wise):** Groups along columns with uniform, value-agnostic partitions.
- **Mixture-of-Scales, OneBitGPT:** Use data-driven groupings on raw (time-domain) weights, lacking frequency selectivity.

HBLLM’s method, by applying localized Haar transforms and optimizing two independent splits per row (one per band), produces four intra-row subgroups and increases the cardinality of the inverse quantization set (CIQ) from $\leq 10$ (ARB-LLM$_x$) to up to $1024$, directly improving representation fidelity under 1-bit quantization (see Sec. 3.1, Appendix B/C in [2512.00862]).

| Method                | #Subgroups/Row | Frequencies Used | CIQ Upper Bound |
|-----------------------|:--------------:|:---------------:|:---------------:|
| BiLLM                 |       2        |      No         |       8         |
| ARB-LLM$_x$           |   Block-wise   |      No         |      10         |
| HBLLM (proposed)      |       4        |      Yes        |     1024        |

## 6. Empirical and Theoretical Impact on Quantization Fidelity

HBLLM’s frequency-aware multi-parameter intra-row grouping achieves substantial practical gains:
- **Ablation (Table 3b):** Switching to frequency-aware grouping reduces LLaMA2-7B perplexity from $16.32 \to 11.08$ (Wiki2) and $1990 \to 95.58$ (PTB).
- **Final Model Results (Table 1):** HBLLM-row reports $7.62/6.68/34.94$ perplexities on C4/Wiki2/PTB for LLaMA1-13B, versus $13.93/14.99/69.75$ for BiLLM.
- **Relative Distance to FP16 (Fig. 1):** HBLLM reduces the gap to full-precision by $33$–$66\%$ compared to previous 1-bit methods.

The theoretical CIQ bound for HBLLM reaches $1024$, versus $8$ (BiLLM) and $10$ (ARB-LLM$_x$) [2512.00862].

## 7. Figures, Tables, and Implementation Reference

- **Fig. 2:** Shows the pipeline including FillAvg handling, Haar transform, and differentiation of salient/non-salient paths.
- **Eq. 3–4 (Sec. 3.3):** Provide the explicit formulations for the Haar transform and sign-based quantization.
- **Table 3(b):** Details the ablation comparing grouping granularities.
- **Table 1:** Summarizes perplexity and accuracy of HBLLM against baselines.
- **Sec. 3.1, Appendix B/C:** Discuss CIQ analysis and the theoretical expressiveness implications.

Taken collectively, frequency-aware multi-parameter intra-row grouping significantly enriches the discrete quantization set, achieving $O(N)$ runtime and negligible storage cost. It provides marked improvements in quantization error and downstream perplexity relative to both global intra-row and channel-wise quantization schemes [2512.00862].

Source: https://www.emergentmind.com/topics/frequency-aware-multi-parameter-intra-row-grouping