---
title: 'Amber Pruner: Training-Free N:M Activation Sparsity'
url: https://www.emergentmind.com/topics/amber-pruner
type: topic
---

# Amber Pruner: Training-Free N:M Activation Sparsity

Amber Pruner is a training-free N:M activation sparsity method designed to accelerate the prefill stage of large language model (LLM) inference by imposing semi-structured activation sparsity without model retraining. It addresses the inefficiencies and accuracy challenges associated with traditional weight sparsity and training-dependent activation sparsity approaches, providing a unified framework for performance-oriented model compression while preserving accuracy and offering seamless integration with post-training quantization [2508.02128].

## 1. Background and Motivation

The prefill stage in autoregressive LLM inference is dominated by dense linear projections—specifically, key, query, value, and MLP gating operations—which are highly compute-intensive when batched. Structured N:M activation sparsity enforces, within every block of M consecutive activation values, that only the top N (by a specified importance criterion) are preserved while the rest are zeroed. Empirically, activations in LLM linear layers tend to have a large proportion (>50%) of values near zero, presenting an opportunity for computational savings far greater than with weight sparsity, which typically causes substantial (>20%) accuracy loss at comparable sparsity ratios when applied without retraining.

Prior activation sparsity approaches (e.g., Q-Sparse, TEAL, Squared-ReLU) require retraining, leverage nonstandard activation functions, or mainly accelerate the decoding phase. In contrast, Amber Pruner focuses on prefill, where most inference time is spent during multi-batch prompt encoding. The method exploits pre-existing activation sparsity and maps efficiently to sparsity-aware hardware via semi-structured N:M masking, minimizing additional indexing overhead.

## 2. Methodology

### 2.1 N:M Activation Sparsity Mechanism

Amber Pruner operates by generating an N:M activation mask per layer in the inference graph. Let $X \in \mathbb{R}^{B \times d_{in}}$ denote the activation matrix for a linear projection with batch size $B$ and input dimension $d_{in}$, partitioned into $G = d_{in}/M$ groups of size $M$. For each entry $X_{i,j}$ within group $g$, the robust scoring function is:

$$
S^*_{i,j} = |X_{i,j}| \cdot f(\widetilde{W}_{:,j})
$$

where $\widetilde{W}_{:,j}$ is the weight vector for column $j$ standardized by percentile-based outlier removal and normalization, and $f(\widetilde{W}_{:,j}) = \|\widetilde{W}_{:,j}\|_2 / \min_k \|\widetilde{W}_{:,k}\|_2$. This “Wanda-style” norm-sensitive scoring preferentially preserves activations linked to more salient weight channels.

Within each group $g$, the top $N$ activations by $S^*_{i,j}$ are retained; the rest are zeroed:

$$
\text{Mask}_{i,j} =
\begin{cases}
1, & \text{if } S^*_{i,j} \geq \tau_g \\
0, & \text{otherwise}
\end{cases}
$$

where $\tau_g$ ensures $\sum_{j \in g} \text{Mask}_{i,j} = N$. Sparse masking is carried out before the weight multiplication to enforce the desired sparsity pattern.

### 2.2 Sensitivity-Driven Layer Skipping

Not all layers are equally robust to activation sparsity. Amber Pruner computes the relative perturbation for each projection:

$$
e_L = \frac{\|Y - Y'\|_2}{\|Y\|_2 + \epsilon}
$$

where $Y$ is the dense output and $Y'$ the output under N:M masked activations. Layers with large $e_L$ are considered sensitive (notably o_proj, up_proj) and exempted from pruning; q_proj and gate_proj are skipped in 5–6 layers, and k_proj/v_proj are pruned selectively, all determined via a fast layerwise scan.

### 2.3 Algorithmic Workflow

The core mask generation process in the prefill stage is:

```python
for each layer ℓ in model.preFillLayers:
    if ℓ in skipped_layers:
        continue
    X ← activation_in(ℓ)           # B×d_in
    S ← compute_scores(X, precomputed_norms[ℓ])
    for each sample i in 1…B:
        for each group g in 1…(d_in/M):
            idx = topk(S[i, g*M:(g+1)*M], N)
            mask[i, g*M:(g+1)*M] = 0
            mask[i, idx] = 1
    X_sparse ← X ⊙ mask
    Y ← X_sparse · Wℓ + bℓ      # SpMM operation
    feed Y to next module
```

The outlined approach is entirely training-free and is realized by lightweight per-tensor statistics and efficient top-k selection.

## 3. Integration with Quantization: Outstanding-sparse Framework

Amber Pruner is integrated within Outstanding-sparse, a pipeline for post-training compression that synergizes N:M activation sparsity with 8-bit quantization (W8A8). The core sequence is:

1. Start with a SmoothQuant-quantized model, which scales each channel via $s_j = \max|X_{:,j}|^\alpha \cdot \max|W_{:,j}|^{1-\alpha}$ for some $\alpha\in[0,1]$.
2. Invert the scale for activations to amplify outliers: $\hat{s}_j = 1/s_j$ (Outstanding-sparse scaling).
3. Scale activations as $X_{\text{scaled}} = X \odot \hat{s}$; quantize to 8-bit.
4. Generate N:M masks on the scaled activations using the Amber Pruner criterion.
5. Apply sparse-dense multiplication in 8-bit on W8A8 weights and activations:

$$
Y = (X_{q8} \odot \text{mask})_{\text{spmm}} \cdot W_{q8}
$$

This unified pipeline boosts the efficacy of activation sparsity (by making outliers more prominent under masking) and ensures minimal loss from quantization.

## 4. Experimental Results

Empirical benchmarks demonstrate the following:

- **Linear Compute Reduction**: Amber Pruner achieves $\sim$56% pruned linear compute in LLaMA3.1-8B, Qwen2-7B, and Qwen3-30B-A3B (MoE), consistently after layer skipping.
- **Accuracy Impact (zero-shot, BFloat16 baseline $\to$ Amber Pruner)**:
  - 2:4: $-2.3\%$ to $-2.7\%$
  - 4:8: $-1.8\%$
  - 8:16: $-0.7\%$ to $-1.4\%$
- **Outstanding-sparse integration** (W8A8 quantization, zero-shot drop vs BFloat16):
  - 2:4: $-2.6\%$ to $-2.9\%$
  - 4:8: $-1.2\%$ to $-1.7\%$
  - 8:16: $-1.1\%$ to $-1.7\%$
- **Generative and Long-context Tasks**: GSM8K (5-shot) drop $<1\%$ at 4:8 and 8:16, $<2.5\%$ at 2:4; LongBench average drop $<1.2\%$ (4:8, 8:16), $<2\%$ (2:4); MoE mode often yields improvements or $<1\%$ drop at moderate sparsity.
- **Throughput and Latency (theoretical speed-up for prefill stage)**:
  - 2:4: $\sim1.25\times$
  - 4:8: $\sim1.35\times$
  - 8:16: $\sim1.4\times$
  - On 8$\times$ Ascend 910B hardware, prompt encoding times reduce proportionally in multi-batch regimes.

## 5. Key Observations and Practical Implications

- Activations naturally exhibit block-structured sparsity that is not present in weights, allowing substantial, hardware-efficient pruning without model retraining or notable accuracy loss ($<1\%$ at 8:16).
- Score computation is both robust and extensible; channel-wise weight norm boosts accuracy retention by favoring more semantically critical neurons.
- The pruning process and associated mask generation can be applied to both dense and MoE architectures with minimal tuning.
- Outstanding-sparse scaling amplifies high-magnitude activations, synergizing with top-N masking for effective quantization and sparsity without additional accuracy penalty.
- Selective layer skipping, governed by simple sensitivity metrics, is essential for limiting deleterious effects in specific projection layers.

## 6. Prospective Developments

Recommendations for next-generation AI systems emerging from this work include:

- Development of 16-bit and 8-bit N:M sparse-dense matrix multiplication kernels in hardware and software stacks to maximize real-world speed-up.
- Systematic integration of channel norm and sensitivity estimation routines into inference software for automated, model-agnostic mask scheduling.
- Co-design of LLM architectures (projection group sizes, dimensions) to further exploit intrinsic activation sparsity.
- Extension of similar mask generation strategies to attention pruning and dynamic decoding optimizations for broader inference acceleration.

Amber Pruner and its Outstanding-sparse extension collectively establish training-free N:M activation pruning as a high-utility, low-risk strategy for large language model prefill acceleration, offering compute reductions $>55\%$ at negligible loss and setting the stage for hardware–software co-design in emerging LLM pipelines [2508.02128].

Source: https://www.emergentmind.com/topics/amber-pruner