V-Usable Information Framework

• V-usable information is a predictive framework that quantifies how much exploitable information about a target variable is available from an input under specific model constraints.
• It generalizes mutual information by basing the measure on achievable log-loss, thereby recovering metrics like the coefficient of determination when predictors are restricted.
• The framework enables empirical estimation, probing of representation layers, and auditing for fairness, privacy, and dataset artifacts in high-dimensional settings.

V-usable information, also termed predictive $\mathcal V$-information, is a variational framework for quantifying how much information about a target variable $Y$ is actually exploitable from an input $X$ by an observer restricted to a predictive family $\mathcal V$. Unlike Shannon mutual information, it explicitly incorporates modeling power and computational constraints, so the measured informativeness depends on which predictors are permitted. In this formulation, information is defined through achievable log-loss rather than unrestricted statistical dependence; as a result, it subsumes mutual information when $\mathcal V$ is unconstrained, recovers familiar quantities such as the coefficient of determination under appropriate restrictions, can increase under computation, and admits PAC-style estimation guarantees in high dimensions (Xu et al., 2020).

1. Variational definition and observer model

The basic objects are random variables $X\in\mathcal X$ and $Y\in\mathcal Y$, together with a predictive family $\mathcal V$ of mappings

$$f : (\mathcal X\cup\{\varnothing\}) \to \mathcal P(\mathcal Y),$$

where $f[x]$ is a predictive distribution over outputs given input $Y$0, and $Y$1 is the corresponding null predictor. In the original formulation, $Y$2 satisfies “optional ignorance”: if a predictor can predict $Y$3 without seeing $Y$4, it can ignore $Y$5 altogether (Xu et al., 2020).

The central variational quantities are the predictive $Y$6-entropies

$Y$7

The predictive $Y$8-information from $Y$9 to $X$0 is then

$X$1

Operationally, this asks how well a $X$2-bounded predictor can compress $X$3 in bits with and without access to $X$4 (Xu et al., 2020).

Two derived pointwise quantities are widely used. Pointwise $X$5-information (PVI) assigns an instance-level information gain

$X$6

where $X$7 is the optimal null predictor and $X$8 the optimal conditional predictor in $X$9. Larger $\mathcal V$0 means that the input $\mathcal V$1 carries more usable information for predicting $\mathcal V$2 (Ethayarajh et al., 2021). A complementary quantity, pointwise $\mathcal V$3-entropy (PVE), measures residual uncertainty for an individual example after fitting $\mathcal V$4: $\mathcal V$5 PVE is used as a low-cost proxy for conditional $\mathcal V$6-entropy on data slices (Vasquez et al., 2024).

A conditional extension isolates information available in one representation beyond another. For a baseline $\mathcal V$7 and an additional representation $\mathcal V$8,

$\mathcal V$9

which parallels conditional mutual information while retaining the predictive-family constraint (Hewitt et al., 2021).

2. Relation to classical information and distinctive theoretical properties

When $\mathcal V$0 is the class of all measurable predictors, the variational entropies coincide with Shannon entropies: $\mathcal V$1 and therefore

$\mathcal V$2

In this sense, predictive $\mathcal V$3-information is a strict generalization of mutual information rather than a competing definition (Xu et al., 2020).

Under restricted predictive families, the framework recovers classical task-specific measures. For linear-Gaussian predictors of the form

$\mathcal V$4

the difference $\mathcal V$5 becomes

$\mathcal V$6

so the theory recovers the unnormalized coefficient of determination. Other choices of $\mathcal V$7 recover mean-absolute-deviation and exponential-family max-entropies (Xu et al., 2020).

Several formal properties differ sharply from Shannon theory. Nonnegativity and monotonicity in the predictive family hold: if $\mathcal V$8, then $\mathcal V$9, so enlarging model capacity cannot reduce usable information (Ethayarajh et al., 2021). At the same time, predictive $X\in\mathcal X$0-information can violate the data-processing inequality. In Shannon theory, $X\in\mathcal X$1 for any function $X\in\mathcal X$2. In the $X\in\mathcal X$3-framework, usable information can increase after preprocessing because the transformation may make the predictive relationship accessible to the restricted observer. The canonical example is RSA decryption: an encrypted representation and its decrypted version have the same Shannon information about the message, yet a computationally bounded $X\in\mathcal X$4-predictor may extract far more usable information after decryption (Xu et al., 2020).

This nonclassical behavior is the theoretical basis for interpreting representation learning as information creation relative to an observer. Successive transforms in a deep network can make progressively more label-relevant structure accessible to simple predictors such as linear classifiers, even when no new Shannon information is introduced (Xu et al., 2020). A later theoretical synthesis extends this observer-relative view to representation similarity: stitching performance can be written as usable conditional information, reconstruction-based metrics estimate usable information under specific predictive constraints, and similarity is therefore relative to the capacity of the predictive family rather than absolute (Almudévar et al., 29 Jan 2026).

3. Computational constraints, empirical estimation, and pointwise variants

The choice of $X\in\mathcal X$5 encodes both model class and computational budget. Concrete examples given in the original framework include “all linear regressors,” “two-layer neural nets of width $X\in\mathcal X$6 and ReLU,” and “$X\in\mathcal X$7-nearest-neighbors with $X\in\mathcal X$8.” Tightening $X\in\mathcal X$9 lowers the amount of information that is usable by that observer (Xu et al., 2020).

Given $Y\in\mathcal Y$0 i.i.d. samples $Y\in\mathcal Y$1, empirical estimates are defined by

$Y\in\mathcal Y$2

and

$Y\in\mathcal Y$3

If $Y\in\mathcal Y$4 and all $Y\in\mathcal Y$5 are bounded in $Y\in\mathcal Y$6, then with probability at least $Y\in\mathcal Y$7,

$Y\in\mathcal Y$8

For many parametric families, $Y\in\mathcal Y$9, and a concrete corollary gives an explicit $\mathcal V$0 bound for linear-Gaussian regressors (Xu et al., 2020).

These guarantees motivate pointwise estimation procedures used throughout later work. In the standard two-model construction, one fine-tunes a model on full inputs to obtain $\mathcal V$1, fine-tunes the same architecture on null inputs to obtain $\mathcal V$2, and then computes $\mathcal V$3 on held-out instances (Ethayarajh et al., 2021). DispaRisk uses a related held-out workflow, but records PVE values $\mathcal V$4 directly and aggregates them over demographic slices (Vasquez et al., 2024).

A distinct approximation is in-context PVI, introduced by Lu et al. for LLMs. Fine-tuning is replaced by two few-shot prompts to the same base model $\mathcal V$5: a null-target prompt $\mathcal V$6 containing labels only, and an input-target prompt $\mathcal V$7 containing full demonstrations plus the query. The in-context estimate is

$\mathcal V$8

Across seven datasets and eight models, the reported stability is substantial: correlation across exemplar sets has average $\mathcal V$9 and median $$f : (\mathcal X\cup\{\varnothing\}) \to \mathcal P(\mathcal Y),$$0, with $$f : (\mathcal X\cup\{\varnothing\}) \to \mathcal P(\mathcal Y),$$1 of model-dataset-shot configurations above $$f : (\mathcal X\cup\{\varnothing\}) \to \mathcal P(\mathcal Y),$$2; correlation across shot counts has average $$f : (\mathcal X\cup\{\varnothing\}) \to \mathcal P(\mathcal Y),$$3 and median $$f : (\mathcal X\cup\{\varnothing\}) \to \mathcal P(\mathcal Y),$$4, with $$f : (\mathcal X\cup\{\varnothing\}) \to \mathcal P(\mathcal Y),$$5 of cases above $$f : (\mathcal X\cup\{\varnothing\}) \to \mathcal P(\mathcal Y),$$6; and for nearly all models and datasets, one-way ANOVA gives small $$f : (\mathcal X\cup\{\varnothing\}) \to \mathcal P(\mathcal Y),$$7-statistics with $$f : (\mathcal X\cup\{\varnothing\}) \to \mathcal P(\mathcal Y),$$8, indicating no significant difference in mean in-context PVI across exemplar sets (Lu et al., 2023).

4. Representations, probing, and training dynamics

Conditional $$f : (\mathcal X\cup\{\varnothing\}) \to \mathcal P(\mathcal Y),$$9-information enables a form of probing that measures information in a representation beyond a baseline. In conditional probing, two probes are trained: a full probe on $f[x]$0 and a baseline probe on $f[x]$1, where $f[x]$2 is a baseline representation such as non-contextual embeddings. The loss difference estimates

$f[x]$3

so any reduction in predictive loss must arise from signal in $f[x]$4 not already present in $f[x]$5 (Hewitt et al., 2021).

In the reported case study, this changes the interpretation of layerwise linguistic information. For ELMo, unconditional $f[x]$6-information for upos is $f[x]$7 bits at $f[x]$8 versus $f[x]$9 bits at $Y$00, but conditional probing yields $Y$01 bits versus $Y$02 bits, shrinking the apparent layerwise drop. For RoBERTa, unconditional probing suggests that upos and xpos information decays after layer 4, whereas conditional probing shows that the information beyond the word embeddings remains around $Y$03–$Y$04 bits through layer 9 and only then declines (Hewitt et al., 2021). This suggests that deeper layers preserve ambiguous contextual cues even when trivial word-identity cues fade.

A related line of work studies training dynamics through a usable-information lower bound

$Y$05

where $Y$06 is a variational decoder and $Y$07 is held-out cross-entropy. Kleinman et al. use this quantity to track minimal sufficient representations during training and report a two-stage motif: usable information about the relevant variable rises rapidly, while semantically meaningful but ultimately irrelevant information also rises early and is later discarded (Kleinman et al., 2020). On CIFAR-10 coarse-vs-fine tasks, usable information about the trained-for coarse label rises from $Y$08 to approximately $Y$09 bit in lock-step with validation accuracy approaching approximately $Y$10, whereas usable information about the fine label first increases to about $Y$11 bits and then declines toward $Y$12 by epoch 200. Larger batch sizes or smaller learning rates eliminate this late-phase forgetting and lead to worse generalization, including about $Y$13 accuracy with $Y$14 (Kleinman et al., 2020).

Usable information has also been used to unify functional and representational similarity. In this formulation, a good stitcher in one direction does not imply similarity, because stitching is inherently asymmetric; robust functional comparison therefore requires bidirectional analysis. Reconstruction-based measures under orthogonal, orthogonal-plus-scale, or affine predictive families define a hierarchy of representational similarity, and standard metrics correlate with the resulting usable-information estimators: the reported empirical values are $Y$15 for CKA, $Y$16 for RSA, and $Y$17 for SVCCA (Almudévar et al., 29 Jan 2026).

5. Dataset auditing, fairness, and privacy leakage

For a fixed model family $Y$18, lower $Y$19-usable information indicates a harder dataset. Ethayarajh et al. therefore recast dataset difficulty as lack of usable information and use PVI to compare datasets, instances, and slices for a given model family (Ethayarajh et al., 2021). In their examples, BART-base extracts approximately $Y$20 bits of usable information on SNLI, BERT-base approximately $Y$21 bits, DistilBERT approximately $Y$22 bits, and GPT-2 approximately $Y$23 bits, with test-accuracy ranks matching the same ordering. Input transformations $Y$24 then expose artifacts by computing $Y$25: on SNLI, shuffling word order hardly reduces usable information, hypothesis-only performance is high while premise-only performance is nearly zero, and in a hate-speech dataset just 50 profane/slur tokens carry most of BERT-usable information (Ethayarajh et al., 2021).

The “data checklist” framework systematizes this logic into ten unit tests, including Viability, Applicability, Exclusivity, Sufficiency, and Necessity, each defined by $Y$26-information inequalities with tolerance $Y$27 (Zhang et al., 2024). On SNLI, the overlap feature $Y$28 yields $Y$29 bits while $Y$30 bits, recovering a known artifact. On SHP preference data, response length is predictive but neither sufficient nor exclusive; on HH-harmless, removing all training pairs with $Y$31 removes about $Y$32 of examples and raises reward accuracy from $Y$33 to $Y$34 and preference accuracy from $Y$35 to $Y$36 under Direct Preference Optimization (Zhang et al., 2024).

Fairness applications use observer-relative uncertainty directly. The original theory already interprets many adversarial-fair methods as minimizing $Y$37 for some adversary class $Y$38, and reports an “attacker-transfer” phenomenon in which a representation fair against one $Y$39-type adversary may still leak information to another (Xu et al., 2020). DispaRisk operationalizes this idea by comparing mean PVE on advantaged and disadvantaged slices: $Y$40 A strong correlation between $Y$41 and downstream fairness metrics such as Demographic Disparity or Equalized Opportunity is taken as evidence that usable-information disparities predict bias amplification. On Census-Income KDD, the FNN+GELU family has the most negative $Y$42 and the largest observed $Y$43 among the reported feed-forward families (Vasquez et al., 2024).

Privacy leakage from gradients has likewise been formulated in usable-information terms. In collaborative learning, the gradient variable $Y$44 may leak either latent attributes $Y$45 or original inputs $Y$46. The empirical usable information from $Y$47 to $Y$48 is defined as the gap between the best null-input cross-entropy and the best gradient-conditioned cross-entropy over an adversary family $Y$49 (Mo et al., 2021). Layerwise analysis shows that original information is easiest to invert from early layers in shallow networks and from middle layers in deeper networks, while latent attributes increase through the convolutional feature extractor, peak at the first fully connected layer, and then fall. Reported interventions include batch aggregation, which nearly eliminates original-information leakage when the target gradient is mixed with at least $Y$50 other samples, and differential privacy noise, whose most effective placement depends on whether the goal is to suppress original or latent leakage (Mo et al., 2021).

6. Structure learning, LLMs, multi-task learning, and domain-specific extensions

The original empirical study demonstrated that predictive $Y$51-information is more effective than mutual information for several downstream problems. In high-dimensional structure learning, replacing Shannon mutual-information estimators such as InfoNCE, NWJ, and MINE with $Y$52 yields much lower wrong-edge rates for Chow–Liu tree recovery, even when $Y$53 is misspecified. On the DREAM5 benchmark for gene regulatory network inference, a polynomial-Gaussian $Y$54 outperforms kernel- and kNN-based Shannon mutual-information estimators in AUC for edge prediction. On Moving-MNIST, $Y$55 with PixelCNN++ predictors decreases in $Y$56, allowing Chu–Liu to recover the causal chain of frames, whereas Shannon mutual information cannot distinguish frame order when the dynamics are deterministic (Xu et al., 2020).

Pointwise $Y$57-information has also been used to organize task relatedness in multi-task learning. Li et al. compare PVI distributions across tasks using a paired $Y$58-test for two-task groupings and one-way ANOVA for larger groupings; tasks whose PVI distributions are not significantly different, with $Y$59, are treated as related enough to benefit from joint learning (Li et al., 2024). On 15 NLP datasets, PVI-based groupings yield joint learners that are competitive with fewer total parameters, with the largest reported gain on CommitmentBank, where $Y$60 improves by over $Y$61, and a two-task RoBERTa-Large MTL system uses roughly half the total parameter footprint of two separate models (Li et al., 2024).

Within LLMs, usable information has been used both diagnostically and interventionally. In retrieval-augmented QA, layerwise $Y$62-usable information is measured with a logit-lens decoder

$Y$63

and the empirical curve of $Y$64 often rises in early-to-middle layers before plateauing or declining (Yuan et al., 22 Apr 2025). The Context-aware Layer Enhancement method chooses a layer $Y$65 where contextual usable information is maximal and applies either amplification,

$Y$66

or residual enhancement into later layers. On CounterFact with Llama2-7B, Exact Match rises from $Y$67 to $Y$68 overall and from $Y$69 to $Y$70 on the “Unknown” subset under CaLE-A; on NQ-Swap, Exact Match rises from $Y$71 to $Y$72 (Yuan et al., 22 Apr 2025).

A further extension treats predictive $Y$73-information as a task-specific image-quality metric for sub-ideal observers. In a stylized MR image-restoration study, the quantity $Y$74 is computed by standard cross-entropy minimization for CNN- or ResNet-based numerical observers and compared to downstream performance (Lu et al., 30 Sep 2025). The reported relationship to ROC analysis is nearly linear on binary tasks, with $Y$75, while $Y$76-information continues to rise in settings where AUC or accuracy saturate and extends directly to multi-class tasks where ROC analysis is difficult (Lu et al., 30 Sep 2025).

Taken together, these developments establish V-usable information as a model-relative notion of informativeness that is simultaneously theoretical and operational. It functions as a variational generalization of mutual information, an instance-level hardness metric, a conditional tool for representation analysis, a practical estimator for high-dimensional structure learning, and an auditing primitive for fairness, privacy, dataset artifacts, transfer, and context use in large models (Xu et al., 2020).