Document Information Gain (DIG)

• Document Information Gain (DIG) is a model-aware measure that quantifies how a candidate document increases or decreases a language model’s confidence in producing a correct answer.
• It reframes document selection by focusing on downstream contribution, using metrics like confidence change and entropy reduction rather than just semantic relevance.
• DIG underpins methods for reranking, filtering, and set selection in retrieval-augmented generation and few-shot learning, enhancing model performance and reducing redundancy.

Document Information Gain (DIG) is a model-aware measure of document utility that quantifies how much observing a candidate document changes uncertainty about a target output or increases confidence in generating the correct answer. In its most explicit recent formulation for retrieval-augmented generation (RAG), DIG is the difference between an LLM’s generation confidence for the ground-truth answer with and without a candidate document. Closely related work applies the same informational principle to other settings: entropy reduction over label distributions for few-shot in-context learning, and relevant information gain (RIG), a probabilistic set-coverage objective for selecting diverse retrieved passages. Across these formulations, DIG shifts the selection criterion from semantic relatedness alone to downstream contribution to correct prediction or generation (Wang et al., 16 Sep 2025, Liu et al., 2023, Pickett et al., 2024).

1. Conceptual motivation and scope

DIG arises from a recurrent limitation in retrieval and prompting pipelines: topical relevance is only a proxy for usefulness. In RAG, a passage can be semantically similar yet misleading, or only marginally similar yet contain a decisive fact. Long contexts also exhibit “lost-in-the-middle” effects, and passage order affects generation. Semantic-similarity rerankers therefore do not directly answer the operational question that matters at inference time: whether a document helps the model produce the correct output. The DIG framework introduced in "InfoGain-RAG: Boosting Retrieval-Augmented Generation via Document Information Gain-based Reranking and Filtering" formalizes this question as document-level contribution to answer generation, rather than document-query affinity (Wang et al., 16 Sep 2025).

A related motivation appears in few-shot in-context learning. "Towards Informative Few-Shot Prompt with Maximum Information Gain for In-Context Learning" reports that even when input distribution, ordering, and prompt format are held constant, random example selection still induces high variance. This motivates ranking candidate examples by the information they provide to the model’s predictive distribution, rather than by surface similarity or random sampling (Liu et al., 2023).

In retrieval-set selection, "Better RAG using Relevant Information Gain" frames the same problem at set level. Because context windows are limited, redundant passages consume budget without increasing answer coverage. The proposed RIG objective rewards query-relevant coverage of latent target passages, so diversity emerges from diminishing marginal gains rather than from an explicit relevance-diversity trade-off (Pickett et al., 2024).

Taken together, these formulations establish DIG as a family of utility measures that are conditioned on the model, the task, and the target output. A plausible implication is that DIG is best understood not as a single metric family tied to one architecture, but as a design principle: rank documents by their marginal effect on task-relevant uncertainty.

2. Formal definitions and operational variants

The most direct DIG definition is document-level confidence gain for answer generation. Given an LLM $\phi$, a query $x$, ground-truth answer $y$, and candidate document $d$, the 2025 RAG formulation defines

$DIG(d|x) = p_{\phi}(y|x,d) - p_{\phi}(y|x)$

where $p_{\phi}(y|\cdot)$ is a probability-like confidence score that the model would generate $y$ given its inputs. The paper specifies that this confidence is not a raw log-likelihood; it is a calibrated sequence probability proxy computed from token probabilities with sliding-window smoothing and token importance weighting. The smoothing term is

$$p_{\text{smooth}}(t_i) = \frac{1}{W}\sum_{j=i-\lfloor W/2 \rfloor}^{i+\lfloor W/2 \rfloor} p(t_j)$$

and the sequence confidence is

$$p_{\phi}(y|x) = \left[\prod_{i=1}^{k} p_{\text{smooth}}(t_i)^{\omega_i \cdot \alpha}\right] \cdot \left[\prod_{j=k+1}^{|y|} p_{\text{smooth}}(t_j)^{1-\alpha}\right]$$

with the same computation applied to $p_{\phi}(y|x,d)$ under the augmented prompt (Wang et al., 16 Sep 2025).

The few-shot ICL formulation is entropy-reduction based. For a candidate observation $x$0 and label space $x$1, information gain is defined as

$x$2

and, because $x$3 is treated as constant for a given task, maximizing IG is equivalent to minimizing conditional entropy:

$x$4

In practice, the paper approximates $x$5 using a zero-shot template $x$6, and after calibration ranks candidates by the entropy of the calibrated distribution $x$7 (Liu et al., 2023).

The set-selection formulation in RAG uses relevant information gain. Under a “one true target passage” premise, the objective for a set $x$8 is

$x$9

where $y$0 is a Gaussian-kernel similarity proxy. This operationalizes document information gain as a coverage functional: each selected document contributes insofar as it covers likely latent target passages conditioned on the query (Pickett et al., 2024).

Setting	Operational quantity	Selection principle
RAG document reranking	$y$1	Prefer positive-contribution documents
Few-shot ICL	$y$2	Prefer lower calibrated conditional entropy
Retrieval-set selection	$y$3	Prefer query-relevant coverage with diminishing returns

This comparison shows that DIG has both pointwise and setwise instantiations. The pointwise forms evaluate a single document/example relative to a baseline prompt, whereas the setwise form directly models redundancy and coverage.

3. DIG in RAG: confidence estimation, reranking, and filtering

In "InfoGain-RAG" (Wang et al., 16 Sep 2025), DIG is used to create supervision for a specialized reranker. The offline DIG data collection pipeline retrieves a candidate set $y$4 for each query $y$5, computes $y$6 once, then computes $y$7 for each candidate document $y$8, and stores triples $y$9. DIG is computed independently per document as a marginal contribution relative to the query-only baseline; interactions among documents are not modeled during label creation. The pipeline also stratifies queries by model proficiency: queries already answered well by the LLM surface harmful or noisy documents with $d$0, while challenging queries surface helpful documents with $d$1.

The resulting DIG labels train a RoBERTa-large cross-encoder reranker that maps $d$2 pairs to scalar scores $d$3. Training is multi-task. First, a CE objective performs binary classification of document utility, with labels derived from DIG thresholds. Second, a margin-based ranking loss uses a LogSumExp-approximated extremal margin to force the maximum negative score below the minimum positive score. The combined objective is

$d$4

with $d$5 yielding strong results. During training-data creation, the implementation uses $d$6 and $d$7 as positive and negative thresholds, with negligible contributions defined in the range $d$8 to $d$9.

The experimental setup computes DIG offline on 110K TriviaQA queries with Qwen2.5-7B, yielding approximately 70K positive, 150K negative, and 1.2M negligible examples. The reranker training set contains 88K samples. The confidence hyperparameters are $DIG(d|x) = p_{\phi}(y|x,d) - p_{\phi}(y|x)$0 for the first $DIG(d|x) = p_{\phi}(y|x,d) - p_{\phi}(y|x)$1 tokens and $DIG(d|x) = p_{\phi}(y|x,d) - p_{\phi}(y|x)$2; the reranker uses Adam with learning rate $DIG(d|x) = p_{\phi}(y|x,d) - p_{\phi}(y|x)$3. At inference, the system retrieves top-$DIG(d|x) = p_{\phi}(y|x,d) - p_{\phi}(y|x)$4 documents, scores each once with the reranker, filters by threshold $DIG(d|x) = p_{\phi}(y|x,d) - p_{\phi}(y|x)$5 while retaining at least two documents, selects top-$DIG(d|x) = p_{\phi}(y|x,d) - p_{\phi}(y|x)$6, and then makes a single LLM call on the reranked context.

The evaluation covers TriviaQA, NaturalQA, PopQA, and FM2, using the December 2018 Wikipedia dump as retrieval corpus. LLMs include Qwen2.5, Qwen3-8B, LLaMA3.1, Gemma-2, DeepSeek-V3, DeepSeek-R1, GPT-4o-20241120, ChatGPT-20240125, and Claude-3.5-Sonnet-20241022; retrievers include Contriever and a multi-retriever combination of Contriever + BM25 + DPR. Exact Match is the primary metric. The reported gains are substantial: on NaturalQA EM, InfoGain-RAG improves over naive RAG by 17.9%, over self-reflective RAG by 4.5%, and over modern ranking-based RAG by 12.5%; GPT-4o shows an average 15.3% improvement across datasets. The paper also reports that InfoGain-RAG, at approximately 335M parameters, often exceeds larger reranker baselines such as GTE-7B. In a multi-retriever setting, average improvement from single to multi-retriever is about 3.8% for InfoGain-RAG. Ablations show that the multi-task loss outperforms CE-only and margin-only variants, DIG labels collected with Qwen or LLaMA produce similar final gains, and filtering with $DIG(d|x) = p_{\phi}(y|x,d) - p_{\phi}(y|x)$7 improves over no filtering.

The principal significance of this formulation is architectural. DIG itself is not computed online at inference time; instead, DIG supervision trains a reusable reranker that approximates document utility with one cross-encoder pass per candidate. This makes the method markedly cheaper than self-reflection pipelines that call the LLM repeatedly for passage critique.

4. DIG as entropy reduction in few-shot in-context learning

In "Towards Informative Few-Shot Prompt with Maximum Information Gain for In-Context Learning" (Liu et al., 2023), DIG is interpreted as IG computed for a candidate document or example in a closed-set classification task. Each candidate is inserted into a zero-shot template $DIG(d|x) = p_{\phi}(y|x,d) - p_{\phi}(y|x)$8 as the test input, and the model’s normalized label distribution is used to estimate the candidate’s conditional entropy. Lower conditional entropy implies higher information gain and therefore higher priority for few-shot selection.

A central technical issue in this formulation is template bias. The paper observes that content-free inputs such as the empty string, “N/A”, or “[MASK]” can induce a biased label distribution through the template alone. To mitigate this, it introduces Calibration Before Sampling (CBS), a vector-scaling calibration step applied before entropy computation:

$DIG(d|x) = p_{\phi}(y|x,d) - p_{\phi}(y|x)$9

with diagonal $p_{\phi}(y|\cdot)$0, zero bias $p_{\phi}(y|\cdot)$1, and a content-free baseline

$p_{\phi}(y|\cdot)$2

Candidates are then ranked using the calibrated entropy

$p_{\phi}(y|\cdot)$3

The sampling algorithm is straightforward: compute the content-free baseline once per template and model, calibrate each candidate’s zero-shot label distribution, rank by ascending entropy, and select the top-$p_{\phi}(y|\cdot)$4 examples. The main experimental setting is one-shot ICL with fixed templates, candidate pools of $p_{\phi}(y|\cdot)$5 unlabeled examples, and $p_{\phi}(y|\cdot)$6 selected demonstration. Evaluation uses 300 test instances per dataset across SST-2, AGNews, TREC, CB, RTE, and DBPedia, and the models are GPT-2 XL, GPT-J, and GPT-3 davinci.

The reported average accuracies across the six tasks are: GPT-2 XL, Random 43.3, MaxIG 46.2, CBS MaxIG 48.8; GPT-J, Random 46.8, MaxIG 47.8, CBS MaxIG 55.9; GPT-3, Random 63.4, MaxIG 68.3, CBS MaxIG 70.3. The paper summarizes these as an average relative improvement of 14.3% across six classification tasks using three LLMs. In four-shot SST-2 with GPT-2 XL, CBS MaxIG outperforms baselines in 13 of 16 class-balance and permutation settings. Additional analyses show that post-calibration at test time is complementary, order probing improves both Random and CBS MaxIG, and replacing gold labels with random labels harms CBS MaxIG more severely than Random.

This formulation is narrower than the 2025 RAG DIG definition. It is explicitly designed for closed-set classification, not open-ended generation, and its IG estimate depends on the prompt template and label space. Nevertheless, it provides a clean entropy-based DIG interpretation: a candidate document is informative when it sharpens the model’s calibrated predictive distribution.

5. Set-level DIG as relevant information gain in retrieval

"Better RAG using Relevant Information Gain" (Pickett et al., 2024) operationalizes DIG at the level of a retrieved set rather than a single document. The core assumption is that there exists an unknown target passage $p_{\phi}(y|\cdot)$7 that best answers a query $p_{\phi}(y|\cdot)$8. A selected set $p_{\phi}(y|\cdot)$9 is good if at least one of its members is close to that latent target. Under a probabilistic uncertainty model over $y$0, the expected set score is

$y$1

and under a Gaussian-kernel similarity model this yields the maximization objective

$y$2

The marginal gain of adding a document $y$3 to an existing set $y$4 is

$y$5

This makes the diversity mechanism explicit. If a candidate is redundant with the current set, then for most $y$6 it does not increase the existing max term, so its marginal contribution is small. The paper states that under the kernel max-coverage model the objective is monotone and submodular, so greedy forward selection inherits the standard $y$7 approximation guarantee.

The practical algorithm begins with candidate triage: retrieve a top-$y$8 set using cosine similarity or FAISS. Then precompute query-to-candidate kernel weights $y$9 and candidate-to-candidate similarities $$p_{\text{smooth}}(t_i) = \frac{1}{W}\sum_{j=i-\lfloor W/2 \rfloor}^{i+\lfloor W/2 \rfloor} p(t_j)$$0, optionally using a cross-encoder for $$p_{\text{smooth}}(t_i) = \frac{1}{W}\sum_{j=i-\lfloor W/2 \rfloor}^{i+\lfloor W/2 \rfloor} p(t_j)$$1 and cosine for $$p_{\text{smooth}}(t_i) = \frac{1}{W}\sum_{j=i-\lfloor W/2 \rfloor}^{i+\lfloor W/2 \rfloor} p(t_j)$$2 in the hybrid variant. Selection seeds the set with the candidate maximizing $$p_{\text{smooth}}(t_i) = \frac{1}{W}\sum_{j=i-\lfloor W/2 \rfloor}^{i+\lfloor W/2 \rfloor} p(t_j)$$3 and then greedily adds the document with highest incremental LogSumExp score. Complexity is dominated by pairwise scoring, especially in cross-encoder variants, with $$p_{\text{smooth}}(t_i) = \frac{1}{W}\sum_{j=i-\lfloor W/2 \rfloor}^{i+\lfloor W/2 \rfloor} p(t_j)$$4 kernel precomputation and $$p_{\text{smooth}}(t_i) = \frac{1}{W}\sum_{j=i-\lfloor W/2 \rfloor}^{i+\lfloor W/2 \rfloor} p(t_j)$$5 greedy selection when implemented naively.

The experiments use the RGB benchmark with simple QA and information integration QA, keeping the LLM (ChatGLM) and evaluation pipeline fixed while varying only the retrieval module. The reported best results at $$p_{\text{smooth}}(t_i) = \frac{1}{W}\sum_{j=i-\lfloor W/2 \rfloor}^{i+\lfloor W/2 \rfloor} p(t_j)$$6 are: for simple QA, D-H 85.6% QA accuracy, D-CC 84.3%, MMR Crosscoder 84.3%, KNN Crosscoder 84.3%, and D-CS NDCG 0.975; for integrated QA, D-CC 42%, D-H 41%, MMR Crosscoder 40%, and D-H NDCG 0.609. The paper also reports that the hybrid variant is robust over a range of $$p_{\text{smooth}}(t_i) = \frac{1}{W}\sum_{j=i-\lfloor W/2 \rfloor}^{i+\lfloor W/2 \rfloor} p(t_j)$$7 values, with best performance at approximately $$p_{\text{smooth}}(t_i) = \frac{1}{W}\sum_{j=i-\lfloor W/2 \rfloor}^{i+\lfloor W/2 \rfloor} p(t_j)$$8 for D-H.

The conceptual contribution of this formulation is that DIG is no longer a pointwise “with versus without one document” difference. Instead, it is an expected coverage functional over a set, aligned with the uncertainty over which passage actually resolves the query. This suggests that DIG can be generalized from marginal utility of individual documents to submodular utility of document sets.

6. Limitations, misconceptions, and theoretical boundaries

A frequent misconception is that DIG is simply another name for semantic relevance. The three papers collectively reject that view. In the 2025 RAG formulation, DIG measures change in answer-generation confidence, not document-query similarity. In the few-shot formulation, the criterion is calibrated entropy reduction in the label distribution, not topical proximity. In the set-selection formulation, relevance is mediated through coverage of latent target passages, and diversity appears through diminishing returns rather than through a separate novelty heuristic (Wang et al., 16 Sep 2025, Liu et al., 2023, Pickett et al., 2024).

A second misconception is that DIG guarantees factual correctness. The 2025 paper explicitly notes that DIG measures confidence change, not fact correctness: a persuasive but wrong passage may raise confidence. It suggests an optional fact-verification module for high-stakes domains. The same paper also identifies residual calibration error, dependence on ground-truth answers during offline DIG label creation, failure to model higher-order document synergy, and long-context “lost in the middle” effects. Potential future extensions named there include submodular selection or Shapley value-based contributions.

The entropy-based few-shot variant has different boundaries. It is defined for closed-set classification, so extending it to open-ended generation requires a tractable output distribution and entropy definition. It also remains sensitive to template choice, label imbalance, order effects, model changes, and distribution shift in the unlabeled candidate pool. CBS mitigates template bias but does not eliminate prompt dependence.

The RIG formulation has its own modeling assumptions. Its theoretical guarantees depend on the kernel max-coverage view and the “one true target passage” premise. Cross-encoder variants incur substantial $$p_{\text{smooth}}(t_i) = \frac{1}{W}\sum_{j=i-\lfloor W/2 \rfloor}^{i+\lfloor W/2 \rfloor} p(t_j)$$9 pairwise scoring overhead, and hybrid scoring, while empirically strong, is described as “less principled” when it mixes score families for $$p_{\phi}(y|x) = \left[\prod_{i=1}^{k} p_{\text{smooth}}(t_i)^{\omega_i \cdot \alpha}\right] \cdot \left[\prod_{j=k+1}^{|y|} p_{\text{smooth}}(t_j)^{1-\alpha}\right]$$0 and $$p_{\phi}(y|x) = \left[\prod_{i=1}^{k} p_{\text{smooth}}(t_i)^{\omega_i \cdot \alpha}\right] \cdot \left[\prod_{j=k+1}^{|y|} p_{\text{smooth}}(t_j)^{1-\alpha}\right]$$1. A single bandwidth parameter $$p_{\phi}(y|x) = \left[\prod_{i=1}^{k} p_{\text{smooth}}(t_i)^{\omega_i \cdot \alpha}\right] \cdot \left[\prod_{j=k+1}^{|y|} p_{\text{smooth}}(t_j)^{1-\alpha}\right]$$2 controls locality and diversity tolerance, and automatic per-query adaptation is left as future work.

From a theoretical perspective, the DIG literature spans several equivalent informational idioms. The 2025 paper notes that if cross-entropy is written as $$p_{\phi}(y|x) = \left[\prod_{i=1}^{k} p_{\text{smooth}}(t_i)^{\omega_i \cdot \alpha}\right] \cdot \left[\prod_{j=k+1}^{|y|} p_{\text{smooth}}(t_j)^{1-\alpha}\right]$$3, then an information-gain style measure is $$p_{\phi}(y|x) = \left[\prod_{i=1}^{k} p_{\text{smooth}}(t_i)^{\omega_i \cdot \alpha}\right] \cdot \left[\prod_{j=k+1}^{|y|} p_{\text{smooth}}(t_j)^{1-\alpha}\right]$$4. The paper nevertheless defines DIG in probability space, while noting that both views preserve the same ordering under monotone transforms. This suggests a unifying interpretation: DIG is a task-conditioned marginal utility signal whose operational form depends on whether the target is a ground-truth sequence, a label distribution, or a latent relevant passage.

In that sense, DIG is best regarded as an umbrella concept for model-conditioned document valuation. Its various formulations differ in supervision, granularity, and computational cost, but they share a common objective: estimate whether a document, or a set of documents, materially reduces uncertainty about the correct output.