---
title: 'GEM: Gradient-to-Weight Ratio & Entropy Masking'
url: https://www.emergentmind.com/topics/gradient-to-weight-ratio-and-entropy-guided-masking-gem
type: topic
---

# GEM: Gradient-to-Weight Ratio & Entropy Masking

to=arxiv_search.query ＿奇米影视  天天中彩票网  ปมถวายสัตย์  菲娱 code=json
{"query":"arXiv:2508.16191 OR title:\"GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"}
to=arxiv_search.query  彩票天天 code=json
{"query":"title:\"GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation\"","max_results":10,"sort_by":"relevance","sort_order":"descending"}
to=arxiv_search.query  菲律宾申博 code=json
{"query":"all:\"Gradient-to-Weight Ratio and Entropy-guided Masking\"","max_results":10,"sort_by":"relevance","sort_order":"descending"}
to=arxiv_search.query 官网群 code=json
{"query":"title:\"Rectifying Mask via Entropy for Distractor-Free 3DGS in Ambiguous Scenarios\"","max_results":10,"sort_by":"relevance","sort_order":"descending"}
Gradient-to-Weight Ratio and Entropy-guided Masking (GEM) is a sparse parameter-efficient fine-tuning framework for downstream adaptation of large pre-trained models that prioritizes parameter updates by their magnitude relative to the scale of the underlying pre-trained weights and allocates the tuning budget across layers using entropy over normalized selection scores. In its canonical formulation, GEM is introduced as a scale-aware, distribution-sensitive masking method for PEFT, with static masks computed once from initial gradients and then used throughout training. The reported empirical result is that GEM can achieve up to a 1.6% improvement in fine-tuning accuracy over full fine-tuning while updating only 0.1% of model parameters on evaluated benchmarks [2508.16191].

## 1. Conceptual basis and motivation

GEM is motivated by a limitation of masking-based PEFT methods that select trainable parameters by maximizing absolute gradient magnitude or related sensitivity measures while freezing the remainder of the model. Under that regime, a gradient of fixed magnitude does not imply a comparable behavioral change for all parameters: if a parameter already has large absolute value, the same update can be small relative to its pre-trained scale. GEM therefore treats *relative* update size, rather than absolute update size alone, as the primary selection signal.

For a standard first-order update with learning rate $\eta$,
$$
w_{t+1} = w_t - \eta \nabla_w \mathcal{L},
$$
the per-parameter relative update is
$$
\frac{|\Delta w|}{|w|} = \eta \cdot \frac{|\nabla_w \mathcal{L}|}{|w|}.
$$
This formulation makes explicit why absolute gradient ranking can be misleading under tight sparsity budgets: a parameter may exhibit a large gradient yet still undergo only a minor relative perturbation. GEM’s central principle is to select parameters for which the induced change is significant in proportion to the initial pre-trained value, thereby making sparse updates more behaviorally consequential [2508.16191].

This also clarifies a common misunderstanding. GEM is not simply a top-gradient mask with a different budget allocator. Its first distinguishing feature is parameter-scale awareness, and its second is the use of entropy to determine how the global budget should be distributed across layers.

## 2. Gradient-to-Weight Ratio and masked optimization

The core scalar prioritization signal in GEM is the Gradient-to-Weight Ratio (GWR). For a scalar parameter $w^{(i)}$, GEM defines
$$
\rho^{(i)} = \left| \frac{\partial \mathcal{L}}{\partial w^{(i)}} \Big/ w^{(i)} \right|.
$$
In practical implementations, a small $\varepsilon$ may be added for numerical stability,
$$
\rho^{(i)} = \frac{|g_i|}{|w_i| + \varepsilon},
$$
although the main formula is the ratio without $\varepsilon$.

GWR is intended to be scale-invariant in the sense relevant to PEFT selection: it ranks parameters by the expected update magnitude relative to the pre-trained weight magnitude. Under sparse adaptation, GEM therefore prioritizes parameters with high $\rho^{(i)}$ rather than parameters with merely high $|g_i|$.

Once the selected set is determined, optimization proceeds through a binary mask. Let $m_i \in \{0,1\}$ denote whether parameter $i$ is trainable. The masked update is
$$
\Delta w_i = -\eta \, m_i \, g_i,
$$
with $g_i = \frac{\partial \mathcal{L}}{\partial w_i}$. At layer level,
$$
W_\ell \leftarrow W_\ell - \eta \cdot \nabla_{W_\ell}\mathcal{L} \odot M_\ell,
$$
where $M_\ell$ has the same shape as $W_\ell$ and $\odot$ denotes element-wise multiplication.

The original GEM framework does not introduce extra regularization terms or auxiliary loss functions. Stability is attributed to the scale-aware selection rule and entropy-guided budget allocation rather than to new regularizers, per-layer rescaling factors, or trainable reparameterization modules [2508.16191].

## 3. Entropy-guided masking and layer-wise budget allocation

GEM’s second defining component is entropy-guided masking, which makes the method distribution-sensitive at the layer level. For each tunable layer $\ell$, the GWR scores are normalized into a discrete distribution
$$
p_\ell^{(i)} = \frac{\rho_\ell^{(i)}}{\sum_j \rho_\ell^{(j)}}.
$$
The layer entropy is then
$$
\mathcal{H}(\boldsymbol{p}_\ell) = -\sum_i p_\ell^{(i)} \log p_\ell^{(i)}.
$$

In this formulation, entropy measures how concentrated the learning signal is within a layer. Low entropy indicates that a small subset of parameters dominates the relative-update signal, whereas high entropy indicates that the signal is spread across many parameters. GEM combines signal strength and distribution through the layer importance score
$$
\alpha_\ell = \left\| \boldsymbol{\rho}_\ell \right\|_2 \cdot \mathcal{H}(\boldsymbol{p}_\ell).
$$
The normalized layer allocation coefficient is
$$
\gamma_\ell = \frac{\alpha_\ell}{\sum_j \alpha_j}.
$$
Given total parameter count $N$ and global tuning ratio $r$, the per-layer budget is
$$
k_\ell = \left\lfloor r \cdot N \cdot \gamma_\ell \right\rfloor.
$$
Within layer $\ell$, GEM then selects the top-$k_\ell$ parameters ranked by $\rho_\ell^{(i)}$.

The significance of this construction is that GEM does not allocate trainable parameters uniformly across layers. Instead, it allocates more capacity to layers whose relative-update signals are both strong and broadly distributed. The reported ablations treat this as essential rather than incidental: uniform allocation, norm-only allocation, and entropy-only allocation are all weaker than the combined $\|\rho\|_2 \times \mathcal{H}$ strategy on the reported tasks [2508.16191].

## 4. Algorithmic procedure, implementation, and computational profile

In the reported experiments, GEM is applied to the query and value projection matrices in transformer layers, while the BitFit baseline trains only biases. The mask is constructed once at the pre-trained initialization $W_0$ by running a single forward/backward pass on a calibration batch or the first training batch, computing GWR scores for all tunable parameters, estimating per-layer entropy and importance, allocating per-layer budgets, and forming a static binary mask. During subsequent fine-tuning, masked updates are applied while all unselected parameters remain frozen [2508.16191].

This static-mask design is a notable property of the method. GEM does not recompute entropy scores or GWR rankings during training in the reported formulation. That makes it closer to a one-shot selection framework than to dynamic sparsification or learned-mask approaches.

The computational overhead is correspondingly modest. Once gradients are available, GWR computation is an $O(N)$ elementwise operation. Entropy computation for each layer is $O(n_\ell)$, and per-layer top-$k$ selection is $O(n_\ell \log n_\ell)$ with sorting or $O(n_\ell)$ with linear-time selection. GEM introduces no extra train-time or inference-time parameters compared with reparameterization methods such as LoRA or adapters, and optimizer state can be restricted to the selected parameters by disabling gradients for frozen weights.

The reported implementation details are specific. The optimizer is AdamW with $\beta_1 = 0.9$, $\beta_2 = 0.999$, and $\epsilon = 10^{-8}$, together with a constant learning rate. Learning rates are selected by grid search over $\{10^{-2}, 10^{-3}, 10^{-4}, 10^{-5}, 10^{-6}\}$ with scale factors of 1 and 5. Batch size is 8. Training typically uses 10 epochs, with 7 epochs for larger datasets such as SST-2 and SQuAD. Training and evaluation samples per task are capped, including 6k train and 1k eval for SST-2, 8k train and 1k eval for SQuAD, and 1k/1k for most other tasks. Reproducibility is assessed with three random seeds per experiment on two 40GB NVIDIA A100 GPUs [2508.16191].

## 5. Empirical performance and analytical findings

The principal evaluation covers seven GLUE/SuperGLUE-style tasks—RTE, SST-2, WiC, BoolQ, MultiRC, COPA, and SQuAD v2.0—on OPT-125M, OPT-1.3B, and Microsoft Phi-2 (2.7B). On OPT-125M, GEM at 0.1% trainable parameters achieves an average score of 68.85, compared with 67.23 for full fine-tuning, and outperforms LoRA, BitFit, Adapter, AdaLoRA, Random Mask, and Top-Grad Mask. On OPT-1.3B, GEM at 0.1% averages 77.47 versus 76.96 for full fine-tuning, surpassing Top Gradient Mask by 6.05% and Random Mask by 2.45%. On Phi-2, GEM at 0.1% reaches 85.52 versus 85.27 for full fine-tuning and is better than the same PEFT baselines across the seven tasks [2508.16191].

The domain-specific evaluation covers GSM8k and MBPP under sparse 0.1% tuning with Phi-2. On GSM8k, measured as zero-shot exact match, full fine-tuning attains 53.41, GEM 48.78, Random 47.96, LoRA 42.82, and AdaLoRA 41.47. On MBPP, measured as 5-shot pass@3, full fine-tuning attains 47.83, GEM 43.58, AdaLoRA 42.94, LoRA 40.24, and Random 40.18. These results indicate that under strong distribution shift all PEFT methods trail full fine-tuning, while GEM leads among the sparse baselines.

The ablation studies are especially important for interpreting the method. Entropy-guided allocation improves both coverage and validation performance: with OPT-1.3B on representative tasks, the captured GWR share of selected parameters is approximately 49–48% for uniform allocation, approximately 68–71% for norm-only allocation, and approximately 72–75% for the combined Norm$\times$Entropy strategy used in GEM. The same section reports that GEM beats Uniform and Norm-only on SST-2, BoolQ, and MultiRC.

Budget sensitivity is also explicit. On Phi-2 across the seven tasks, 0.5% trainable parameters gives the best average score at approximately 85.71, followed by 0.1% at approximately 85.52, 0.05% at approximately 83.45, and 0.01% at approximately 81.11. The paper therefore identifies a marked degradation below 0.1%, suggesting that excessively sparse updates can remove too many degrees of freedom for effective adaptation. Additional analysis reports that GWR-driven masks produce the largest relative weight changes and the greatest loss reduction compared with random and top-gradient masking, which the authors align with the view that relative change is a more meaningful proxy for adaptation than absolute gradient magnitude alone [2508.16191].

## 6. Positioning, limitations, and later extensions of the terminology

Within PEFT, GEM is positioned against reparameterization methods such as LoRA, adapters, and prompt tuning; against bias-only updating as in BitFit; and against mask-learning or saliency-based selection schemes such as DiffPruning, movement pruning, gradient-magnitude masking, Fisher/Hessian-based selection, SNIP, and GraSP. The defining claim is not merely sparsity, but the combination of parameter-wise scale awareness and entropy-guided layer allocation under a static-mask regime. A concise way to state the distinction is that GEM does not add modules and does not learn masks during training; it computes a one-time mask using relative-update statistics and layer-wise entropy, then fine-tunes only the selected parameters [2508.16191].

Several limitations follow directly from this design. Because the mask is computed once from initial gradients, noisy gradient snapshots—such as those arising from tiny datasets or unstable batches—can produce suboptimal selection. Very small $|w_i|$ can inflate $\rho_i$, which is why practical use may require an $\varepsilon$ stabilizer or gradient accumulation before mask construction. Performance drops markedly below approximately 0.1% tuning ratio on several tasks. Under strong distribution shift, as in GSM8k and MBPP, full fine-tuning still outperforms sparse PEFT. The reported scale range also stops at 2.7B parameters, so 7B+ behavior is not reported.

A further source of confusion is terminological. In a later technical note associated with RefineSplat for distractor-free 3D Gaussian Splatting, entropy-guided masking is used in a different context, and a *GEM-style* Gradient-to-Weight Ratio is proposed as a complementary criterion rather than as a component originally defined by RefineSplat. There, local masks are built from per-instance semantic entropy, global masks from the ratio of feature-level uncertainty to entropy, and a per-Gaussian ratio
$$
\rho_i = \frac{\|\nabla_{\theta_i} L_{\text{total}}\|}{w_i + \varepsilon}
$$
is introduced using alpha-blending contribution weights $w_i$ to identify low-weight, high-gradient artifacts. The note explicitly states that RefineSplat itself does not define a Gradient-to-Weight Ratio; the GEM criterion is presented as a consistent extension for ambiguous transient suppression in 3DGS rather than as the original meaning of GEM in PEFT [2606.29496].

Taken together, these uses of the term share a common intuition: entropy is employed to characterize the distribution of uncertainty or learning signal, while a gradient-to-weight quantity identifies components whose update significance is large relative to a scale variable. In the original PEFT setting, that scale variable is the pre-trained parameter magnitude; in the 3DGS extension, it is a Gaussian’s contribution weight. This suggests a broader methodological pattern in which sparse or selective adaptation benefits from ranking updates by relative, rather than absolute, effect.

Source: https://www.emergentmind.com/topics/gradient-to-weight-ratio-and-entropy-guided-masking-gem