--- title: Contextual Max-Value Expected Information Gain url: https://www.emergentmind.com/topics/contextual-max-value-expected-information-gain-cmv-eig type: topic --- # Contextual Max-Value Expected Information Gain Contextual Max-Value Expected Information Gain (CMV-EIG) refers to a principled criterion for selecting informative examples in few-shot in-context learning (ICL) with large language models (LLMs). CMV-EIG quantifies the informativeness of candidate demonstration examples by estimating their effect on reducing predictive uncertainty, employing an entropy-minimization approach, and explicitly mitigating adverse effects of template-induced bias through calibration procedures. The approach introduces robust selection mechanisms for constructing few-shot prompts, yielding significant improvements in ICL performance across diverse classification benchmarks [2310.08923]. ## 1. Formal Definition and Min-Entropy Reduction Expected Information Gain (EIG) in ICL measures, for a candidate input $x$, the expected reduction in model uncertainty about the output labels $Y$ given a context $C$. In the paradigm instance considered, $C$ comprises only an empty or templated prompt $T$. The general EIG expression is \[ \mathrm{EIG}_C(x) = H\bigl[p(y \mid C)\bigr] - \mathbb{E}_{y' \sim p(y \mid C, x)}\bigl[ H\bigl[p(y \mid C, x, y')\bigr] \bigr] \] where $H[\cdot]$ denotes Shannon entropy. In practice, $H[p(y \mid C)]$ is context-constant, and the expectation over unknown true labels is intractable for black-box LLMs. Employing black-box access yields an operational “min-entropy” criterion where utility is given by $-\mathrm{H}\left[p_\theta(y \mid x, T)\right]$, with $p_\theta$ the model’s zero-shot output conditional on $x$ and $T$. Thus, the practical utility of an example $x$ is defined as \[ \mathrm{IG}(x) := - \sum_{y} p_\theta(y \mid x, T) \log p_\theta(y \mid x, T) \] The top-$K$ examples minimizing conditional entropy $H(Y \mid x)$ are selected for prompt assembly [2310.08923]. ## 2. Contextual Max-Value EIG Sampling Criterion The CMV-EIG criterion formalizes informativeness-driven selection via: \[ x^* = \underset{x \in \mathcal{D}_{\mathrm{unlab}}}{\arg\max} \mathrm{IG}(x) = \underset{x \in \mathcal{D}_{\mathrm{unlab}}}{\arg\min} H\left[p_\theta(y \mid x, T)\right] \] where $\mathcal{D}_{\mathrm{unlab}}$ denotes the candidate pool. The reduction from “expected” to “observed” entropy is justified by invariance of the baseline entropy term and infeasibility of true label marginals when only black-box evaluations are possible. The information theoretic interpretation aligns with principles of active learning, though the candidate pool and selection dynamics are uniquely adapted to few-shot prompt engineering. ## 3. Template Bias and Calibration Before Sampling Raw application of the min-entropy criterion is sensitive to template bias: non-uniform prior distributions $p_\theta(y \mid T)$ arising from the prompt template alone. This bias causes certain candidate examples to “correct” for systemic over-prediction of specific labels, yielding low entropy but trivial informativeness (e.g., contentless samples appearing more informative than they are). Empirically, even empty templates induce pronounced skew. To remove this bias, content-free calibration (CBS) is employed [2310.08923]. The procedure consists of: 1. **Content-free prompt pool**: Assemble $D_{\mathrm{cf}} = \{ "", "N/A", "[MASK]" \}$. 2. **Template bias vector computation**: Average zero-shot outputs on content-free strings, \[ p_{\mathrm{cf}} = \mathrm{softmax}\!\left( \frac{1}{|D_{\mathrm{cf}}|} \sum_{x_{\mathrm{cf}} \in D_{\mathrm{cf}}} p_\theta(y \mid x_{\mathrm{cf}}, T) \right) \] 3. **Vector-scale calibration**: Given model output $p$, scale by $W = \mathrm{diag}(p_{\mathrm{cf}})^{-1}$ and apply softmax: \[ q_\theta(y \mid x, T) = \mathrm{softmax}( W p ) \] 4. **Calibrated entropy criterion**: Compute $H_{\mathrm{cal}}(Y \mid x) = -\sum_y q_\theta(y \mid x, T) \log q_\theta(y \mid x, T)$, and select examples minimizing $H_{\mathrm{cal}}$. This adjustment ensures selection uniquely reflects example informativeness rather than template-induced label preferences. ## 4. Algorithmic Implementation The full CMV-EIG with calibration procedure is as follows: ```plaintext 1. // Precompute template bias via content-free inputs 2. for each x_cf in D_cf: 3. prompt_cf ← fill_template(T, x_cf) 4. p_cf_sum += LLM( prompt_cf ) // vector of size |Y| 5. p_cf ← softmax( p_cf_sum / |D_cf| ) 6. for each x in D_unlab: 7. prompt_x ← fill_template(T, x) 8. p ← LLM( prompt_x ) // zero-shot output probs 9. // Calibration (vector scaling) 10. W ← diag( 1.0 / p_cf ) 11. q ← softmax( W · p ) // calibrated probs 12. H_cal[x] ← −∑_{y} q[y] * log q[y] // conditional entropy 13. // Rank and select 14. Sort D_unlab by ascending H_cal[x] 15. return top-K examples ``` Practical recommendations include drawing $N=100$ candidate samples, use of three content-free calibration strings, default greedy decoding (temperature $0.0$), and K-shot prompt construction with true gold labels for downstream evaluation [2310.08923]. ## 5. Experimental Context and Benchmarking Evaluations were conducted on six classification tasks: SST-2, AGNews, TREC, CB, RTE, and DBPedia. Three LLMs were considered (GPT-2 XL, GPT-J, GPT-3 davinci) with $N=100$ randomly subsampled train candidates per task. Each LLM was assessed over five random seeds (two for GPT-3), and evaluated on 300 test samples per task. Comparative baselines included: - Random selection, - MaxEntropy (highest raw $H[p_\theta(y|x,T)]$), - MaxIG (min raw $H$, no calibration), - CBS MaxIG (calibrated information gain). Empirical results demonstrate that CBS MaxIG achieves a $12$–$19$\% relative accuracy gain on average over random selection and consistently outperforms both MaxEntropy and uncalibrated MaxIG. This confirms IG as a robust informativeness proxy in ICL and indicates the necessity of pre-sampling calibration to neutralize template bias [2310.08923]. ## 6. Significance and Related Methodologies The CMV-EIG approach generalizes conventional entropy-based active learning techniques for use with LLM in-context learning via black-box access. By centering entropy minimization and incorporating a formal calibration step, CMV-EIG addresses both the variance induced by demonstration selection and systemic pitfalls of template and prompt-based evaluation. The technique is directly extensible to other language models and is robust under settings where gold labels are not initially accessible for all candidates—a scenario typical in prompt selection for ICL. A plausible implication is that further improvements in in-context demonstration selection may be achievable by refining calibration strategies or entropy estimation techniques. Source: https://www.emergentmind.com/topics/contextual-max-value-expected-information-gain-cmv-eig