Information-Gain Weighting: Concepts & Applications

• Information-gain weighting is a statistical method that adjusts data, feature, and model contributions by scaling entropy and divergence measures to reflect relative importance.
• It employs weighted entropy functions and exponential updating to control posterior contraction and refine the informativeness of Bayesian models.
• Applications span decision tree induction, bias-corrected feature selection, and robust embedding models in lexical semantics to improve predictive performance.

Information-gain weighting systematically alters the contribution of data, features, or model components by scaling their empirical or theoretical information content—typically Shannon entropy or Kullback–Leibler (KL) divergence—within probabilistic estimators, model updates, or downstream metrics. This concept appears in statistical modeling, machine learning, and information theory as a means to encode domain knowledge about the relative informativeness of events, features, or observations, thereby refining model calibration, learning dynamics, or interpretability.

1. Theoretical Foundations and Core Definitions

Information-gain weighting leverages weighted entropy or KL divergence as central quantities. Given a probability distribution $P = \{p_i\}$ for $i=1,...,n$, and a set of positive utility weights $U = \{u_i\}$, the Belis–Guiasu weighted entropy is defined as

$H(P,U) = -\sum_{i=1}^n u_i\, p_i \ln p_i,$

which simultaneously encodes stochastic uncertainty through $p_i$ and application-specific “importance” via $u_i$ (Srivastava et al., 2015). This generalizes classical entropy—recovered by setting all $u_i = 1$—and admits alternative interpretations for feature relevance, event cost, or other priorities.

The weighted information-generating function (IGF) further unifies multiple entropy and divergence notions:

$I(P,U; t) = \sum_{i=1}^n p_i^{\,1-u_i(1-t)},$

with derivatives at $t=1$ yielding the (weighted) entropy and higher moments of the self-information (Srivastava et al., 2015). Weighted IGFs facilitate the construction of weighted Rényi and Tsallis entropies, extending the scope of information-gain weighting to non-Shannon settings.

In the context of Bayesian inference, exponential weighting of measures underpins the weighted updating framework:

$$\tilde{\pi}(\theta \mid x) = \frac{L(x\mid\theta)^\beta \, p(\theta)^\alpha}{\int L(x\mid\theta)^\beta \, p(\theta)^\alpha\, d\theta},$$

where $(\alpha, \beta)$ act as information scaling coefficients that distort the relative impact of the likelihood and the prior (Zinn, 2016). These exponents directly manipulate the entropy of the updated distribution, as explicated by the entropy monotonicity theorem (below).

2. Entropy, Information Gain, and Exponential Weighting

Weighted updating and its entropy-theoretic impact are formalized as transformations of base distributions by positive powers. For any $\gamma>0$, the transformation $$g_\gamma(\omega) = g(\omega)^\gamma / \int g(\omega)^\gamma d\mu(\omega)$$ is strictly increasing and mode-preserving, but systematically contracts ($\gamma>1$) or disperses ($\gamma<1$) the density (Zinn, 2016). This relationship is captured by the following results:

• If $0 < \gamma < 1$, the weighted distribution is a monotone dispersion: increased entropy, more uniform, less “informative.”
• If $\gamma > 1$, it is a monotone concentration: decreased entropy, more peaked, more “informative.”

When applied to Bayesian updating, the Kullback–Leibler divergence between prior and posterior distributions,

$$D_{KL}[\pi(\cdot|x) \,\|\, p] = H(p) - H(\pi(\cdot|x)),$$

measures information gain from data $x$. Under weighted updating,

• $\alpha > 1$ (resp. $< 1$) causes the prior’s contribution to become over-concentrated (resp. under-dispersed).
• $\beta > 1$ (resp. $< 1$) similarly reflects an “over-reaction” (resp. “under-reaction”) to the likelihood (Zinn, 2016).

Information-gain weighting thus directly governs posterior contraction and can quantitatively model deviations from ideal Bayesian updating due to behavioral biases or application needs.

3. Estimation of Information Gain in Empirical Machine Learning

The practical application of information-gain weighting is prominent in decision tree induction and feature selection. Information gain for a discrete split variable $B$ and target $Y$ is estimated as

$$I(Y;B) = H(Y) - \Pr[B=L]\, H(Y|B=L) - \Pr[B=R]\, H(Y|B=R),$$

where $H(\cdot)$ is typically the plug-in estimator (empirical entropy) (Nowozin, 2012). However, standard plug-in estimators are negatively biased by $\frac{K-1}{2n}$ for $K$ classes and $n$ samples.

To correct this, advanced estimators are used:

• Grassberger’s estimator for discrete entropy employs digamma corrections, significantly reducing bias (Nowozin, 2012).
• UMVU estimator for continuous (Normal) entropy, as well as Kozachenko–Leonenko 1-NN estimators, allow nonparametric and unbiased estimation in the regression context.

By utilizing these corrected estimators in the information gain criterion, both split selection and feature importance calculations reflect more accurate underlying mutual information.

Table: Plug-in vs. Bias-Corrected Estimators in Decision Trees

Estimator	Setting	Bias Properties
Plug-in	Discrete/Continuous	Negatively biased
Grassberger	Discrete	Low bias
UMVU, 1-NN	Continuous	Unbiased/Nonparametric

Empirical results show that trees built with bias-corrected information gain estimators yield statistically significant improvements in predictive metrics, especially on high-class-count problems or low-sample regimes, with minimal runtime or code complexity overhead (Nowozin, 2012).

4. Information Gain in Embedding Models and Lexical Semantics

Recent work demonstrates that in Skip-Gram with Negative Sampling (SGNS) and modern LLMs, the squared norm of word embeddings encodes the information gain—defined as the KL divergence between the word’s empirical co-occurrence distribution and the unigram distribution (Oyama et al., 2022). Precisely,

$$I(w) = KL(p(\cdot|w) \,\|\, p(\cdot)) = \sum_{w'} p(w'|w) \log \frac{p(w'|w)}{p(w')}$$

is approximated (after whitening) by $\|\tilde{u}_w\|^2$, where $u_w$ is the raw embedding and the whitening is with respect to the Fisher information matrix of the exponential family underlying the embedding model.

To remove the influence of word frequency on both $KL(w)$ and embedding norm,

• Either shuffle-based noise baselines (for $KL$) or lower-percentile subtraction (for $\|u_w\|^2$) are used.
• The frequency-bias-corrected information gains ($\Delta KL$, $\Delta\|u_w\|^2$) are then robust informativeness measures.

These bias-corrected metrics have been shown empirically to outperform frequency and baseline methods for keyword extraction, proper noun discrimination, and hypernym detection (Oyama et al., 2022).

5. Applications to Feature Selection, Ensemble Models, and Informativeness Metrics

Information-gain weighting provides a principled mechanism for ranking and selecting features in high-dimensional models, especially within tree ensembles. The standard approach is to aggregate the information gain at each internal node across the ensemble for every feature and to normalize the resulting scores (Nowozin, 2012). When using bias-corrected (weighted) information gain, these scores more faithfully represent the true mutual information between features and targets.

In lexical semantics, information-gain-based measures derived from KL divergence or embedding norms yield superior unsupervised importance and discriminability measures compared to frequency-based heuristics (Oyama et al., 2022). Enhanced keyword extraction and type identification in large corpora are among the demonstrable downstream benefits.

6. Calibration, Interpretability, and Connections with Weighted Information Measures

The explicit interpretation of weighting parameters as information scalars enables calibration against observed behavioral updating or domain-knowledge-driven priors (Zinn, 2016). Parameters ($\alpha$, $\beta$) in weighted updating or ($u_i$) in weighted entropy are typically selected either by moment matching—matching observed summary statistics such as mean shifts or variance contraction—or via entropy-based fitting, using confidence intervals or distributional estimates.

Weighted information measures generalize to generating functions for non-Shannon entropies, supporting the derivation and comparison of weighted Rényi and Tsallis entropies:

$$H_\alpha(P,U) = \frac{1}{1-\alpha} \ln \left[ \sum_i p_i^{1-u_i(1-\alpha)} \right],$$

with the standard weighted Shannon entropy recovered as $\alpha \to 1$ (Srivastava et al., 2015).

The additivity and analytic properties of these weighted information measures under independent composition preserve interpretability and ensure the stability of informativeness metrics across composite systems or hierarchies.

---

Information-gain weighting thus provides a rigorous and flexible apparatus for manipulating and interpreting the informational structure of probabilistic models, feature importance scores, and embedding-based representations, with direct applications in statistical learning, decision theory, and natural language processing. Quantitative insights from this framework enable more accurate model calibration, improved predictive performance, and robust interpretability metrics across diverse domains.