Mean Information Gain (MIG) in ML

• Mean Information Gain (MIG) is an information-theoretic measure that quantifies the average reduction in uncertainty across multivalued attribute splits.
• It leverages adaptive simulated annealing to efficiently search through exponentially many candidate partitions, ensuring robust split selection.
• MIG aids feature selection by identifying midsize attribute subsets that balance classification performance with computational efficiency.

Mean Information Gain (MIG) is an information-theoretic quantity central to assessing the discriminative power and feature selection capacity in rule-based machine learning, especially decision tree induction. Its conceptual extension within the multivalued subset approach incorporates not only the most informative split but the average information gain across candidate partitions generated by grouping attribute values—a procedure rigorously explored in "Multivalued Subsets Under Information Theory" (Dabhade, 2011). This perspective offers a more robust statistical framework for evaluating feature sets and optimizing classification models.

1. Mathematical Definition

Mean Information Gain (MIG) generalizes the classic information gain metric. Traditional information gain for an attribute $X$ is given by:

$\text{Gain}(X) = H(T) - H(T|X)$

where $H(T)$ is the entropy of the target class distribution, and $H(T|X)$ is the conditional entropy after splitting on $X$. Extending this to MIG, suppose there are $n$ candidate partitions—subsets $S_1, S_2, ..., S_n$ formed by combining attribute-values. The MIG is defined as:

$$\text{MIG} = \frac{1}{n} \sum_{i=1}^n \text{Gain}(S_i)$$

Each gain term is computed as:

$\text{Gain}(S_i) = H(T) - H(T|S_i)$

where $S_i$ is a (potentially multivalued) subset of attribute groupings. MIG thus quantifies the average reduction in uncertainty achievable by considered splits, not just the optimal split.

2. Multivalued Subset (MVS) Heuristic and Search

Unlike classic decision tree algorithms (e.g., ID3), which evaluate all singleton attribute partitions, the MVS approach generates binary partitions by grouping attribute values. For an attribute $A$ with $r$ distinct values, there are up to $2^{r-1}-1$ binary partitions. The combinatorial search space grows exponentially, so heuristic search is employed.

The paper utilizes Adaptive Simulated Annealing (ASA) to effectively sample and rank candidate subsets:

• The search identifies $n$ promising multivalued divides, rather than merely the globally maximal gain.
• For each candidate subset $S_i$, compute $H(T|S_i)$ using:

$H(T|S) = \sum_j \frac{|S_j|}{|T|} H(S_j)$

where $S_j$ is the $j$-th partition and $H(S_j)$ is its class entropy.

• MIG is subsequently the mean over the selected candidate gains.

3. Statistical Evidence and Results

Experimental validation is conducted using the Iris and Vehicle Silhouettes datasets:

• MVS-selected attribute sets achieve statistically lower classification errors (e.g., 22.22% error on Iris) versus errors from single-attribute splits.
• Comparative plots (e.g., Figures 6 and 8) show MIG correlated with lower errors for same-sized feature sets, indicating that maximizing mean rather than maximal gain leads to improved generalization.
• The distribution of subset sizes selected via ASA is found to be (approximately) normal, supporting the averaging process’s stability; individual gain values are non-normal due to their bounded and skewed nature.

4. Feature Selection Implications

MIG extends its utility to feature subset selection:

• Rather than selecting features based solely on the highest information gain, using MIG facilitates the selection of feature subsets that are collectively informative.
• Empirical data shows that "midsize" subset selections (not minimal or maximal dimensionality) often achieve superior classification performance, as per MIG-guided selection.
• MIG thus serves as a more global criterion, balancing the trade-off between underfitting (too few features) and overfitting/noise (too many).

5. Computational Considerations

The multivalued subset search has exponential complexity in the number of attribute values, necessitating heuristic methods:

• Adaptive Simulated Annealing efficiently traverses the search space and identifies high-quality partitions.
• MIG calculation is robust to the number of candidate subsets $n$ when the search heuristic identifies a consistent range of effective splits.
• The arithmetic mean of gain values remains valid where the subset selection process stabilizes with respect to size, as indicated by normality testing in subset size distributions.

6. Broader Context and Extensions

The approach outlined in "Multivalued Subsets Under Information Theory" (Dabhade, 2011) situates MIG within a wider context of tree-based classifiers, feature selection, and ensemble learning:

• MIG directly connects to classical information-theoretic principles (entropy and uncertainty reduction).
• The ranking and averaging of information gain over multivalued splits provides a basis for algorithms that seek not just locally optimal splits but robust, ensemble-level performance measures.
• The methodology accommodates extensions to larger datasets and higher-dimensional spaces, provided computational efficiency in subset search is managed.

Summary Table: MIG Formulas

Concept	Formula	Description
Entropy	$H(T) = - \sum_i p_i \log_2 p_i$	Uncertainty of target distribution
Conditional Entropy	$H(T|S) = \sum_j \frac{|S_j|}{|T|} H(S_j)$	Weighted sum over partitions
Information Gain	$Gain(S) = H(T) - H(T|S)$	Reduction in uncertainty per split
Mean Information Gain	$MIG = \frac{1}{n} \sum_{i=1}^n Gain(S_i)$	Average over multivalued splits

In summary, MIG as formulated in the MVS framework provides a robust average measure of discriminative power across decision splits, reflecting both the complexity and the stability of information gain-based classifiers. Its computation through adaptive search and subsequent use in feature selection and model optimization establishes MIG as a foundational metric for rule-based machine learning and predictive analytics.