Information-Theoretic Probing Framework

• Information-theoretic probing is a framework that measures mutual information between learned representations and linguistic targets using MDL, Bayesian methods, and variational bounds.
• It employs controls and randomized baselines to distinguish true encoded information from probe capacity, finite-data effects, and overfitting.
• The approach quantifies probe effort and layer-wise behavior through metrics like description length and Bayesian mutual information, guiding model selection and analysis.

Information-theoretic probing is an approach to analyzing learned representations in which the central object is not probe accuracy itself, but an information-theoretic quantity relating a representation to a target property. In the basic formulation, if $R$ denotes a learned representation and $T$ a linguistic label, probing is cast as estimating $I(T;R)=H(T)-H(T\mid R)$. Subsequent work extended this perspective in several directions: minimum description length (MDL) treats probing as data compression; Bayesian mutual information replaces classical mutual information in finite-data regimes; variational bounds connect linear probing, MINE, InfoNCE, and fine-tuning; and Bayesian evidence reframes probing as model selection for inductive bias. Across these variants, the common aim is to separate information encoded in the representation from effects due to probe capacity, finite data, regularization, and memorization (Pimentel et al., 2020, Voita et al., 2020, Pimentel et al., 2021, Choi et al., 2023, Immer et al., 2021).

1. Mutual information as the core operationalization

A standard formalization takes $X$ to be a pre-computed representation and $Y$ the discrete label to be probed. Mutual information is then

$I(X;Y)=H(Y)-H(Y\mid X).$

Because the true conditional $p(Y\mid X)$ is unknown, one trains a probe $q_\theta(Y\mid X)$ and evaluates its cross-entropy. In the formulation of Pimentel et al., if $R$ is the representation and $T$ the target label, the empirical conditional cross-entropy

$T$0

upper-bounds $T$1, which yields the lower bound

$T$2

The bound becomes tight when $T$3 in KL-divergence, so improving probe fit tightens the mutual-information estimate (Pimentel et al., 2020).

This perspective also gives an exact decomposition of probe cross-entropy. If $T$4 is trained by minimizing

$T$5

then

$T$6

Since $T$7 is fixed, a low cross-entropy can arise only from either a large $T$8 or a small approximation error $T$9. This recovers the Hewitt–Liang “rich-representation vs. probe-learns-task” dichotomy exactly, but in explicitly information-theoretic form (Zhu et al., 2020).

Within this formulation, the earlier preference for deliberately weak probes is not information-theoretically justified. A more expressive probe family can better approximate the unknown conditional $I(T;R)=H(T)-H(T\mid R)$0, attain a lower $I(T;R)=H(T)-H(T\mid R)$1, and therefore produce a tighter lower bound on $I(T;R)=H(T)-H(T\mid R)$2. In that specific sense, the framework argues that “bigger probes are better” when the goal is estimation of information already present in the representation (Pimentel et al., 2020).

2. Controls, randomized baselines, and probe selection

Information-theoretic probing did not eliminate the need for controls; rather, it gave them a formal role. One line of work studies randomized targets. In the control-task construction, true labels $I(T;R)=H(T)-H(T\mid R)$3 are replaced by random labels $I(T;R)=H(T)-H(T\mid R)$4 drawn independently of $I(T;R)=H(T)-H(T\mid R)$5, and the selectivity score is

$I(T;R)=H(T)-H(T\mid R)$6

A parallel construction randomizes the representation, defining $I(T;R)=H(T)-H(T\mid R)$7 independently of $I(T;R)=H(T)-H(T\mid R)$8, with information-gain target

$I(T;R)=H(T)-H(T\mid R)$9

and estimator

$X$0

The two criteria satisfy

$X$1

and, under a perfectly random control, their difference is constant with respect to model capacity. On this basis, the two selection criteria rank probe architectures identically (Zhu et al., 2020).

The same section of the literature also introduced deterministic control functions for contextualized representations. If $X$2 is a type-level control representation, then the contextual gain is

$X$3

Empirically, this construction was used to compare multilingual BERT to type-level controls such as fastText and learnable one-hot embeddings. On part-of-speech tagging across eleven languages, the contextual gain was small: BERT added at most approximately $X$4 more information than fastText in the five languages where it did so, with negative estimation-error gains in the others. On dependency labeling, gains were larger but still modest: at most approximately $X$5 more information in English, and less than $X$6 in over half the languages (Pimentel et al., 2020).

These results situate control-based probing in a sharper way than accuracy comparisons alone. Randomized labels or representations remove memorization effects; type-level controls estimate what portion of the information is already available without context. Empirically, on POS-tag probing over English, French, and Spanish UD data, more than $X$7 hyperparameter settings produced highly agreeing rankings under $X$8 and $X$9, indicating that control-task and control-function selection criteria are equivalent for model comparison (Zhu et al., 2020).

3. Minimum description length and the notion of effort

A distinct information-theoretic reformulation treats probing as data compression. Given a dataset $Y$0, Alice knows both $Y$1 and $Y$2, while Bob knows only $Y$3. If both parties agree on a fixed probabilistic model $Y$4, Shannon’s source coding theorem gives the optimal codelength

$Y$5

so probe loss becomes a codelength. MDL probing adds the cost of the model itself: $Y$6 The central quantity is therefore not only the final predictive quality, but the amount of effort required to achieve that quality (Voita et al., 2020).

Two practical estimators were proposed. Variational coding uses a prior $Y$7 and a variational approximation $Y$8, yielding

$Y$9

The first term is the cost of transmitting the model relative to the prior; the second is the expected data codelength. Minimizing this objective is equivalent to maximizing the ELBO. Online coding, by contrast, avoids explicitly sending the model: the sender and receiver agree on architecture, initialization, optimizer, and a schedule $I(X;Y)=H(Y)-H(Y\mid X).$0. The first block is sent with a uniform code, and later blocks are sent using predictors trained on earlier blocks, giving

$I(X;Y)=H(Y)-H(Y\mid X).$1

This quantity is closely linked to the area under the probe-accuracy learning curve (Voita et al., 2020).

The significance of MDL probing is that it operationalizes effort in two concrete ways. In variational coding, effort is model size through the KL term; in online coding, effort is the amount of data needed before the probe compresses labels well. This makes it possible for two probes with similar accuracy to differ sharply in information-theoretic quality. Empirically, standard accuracy was nearly identical for true PoS tags and random control tags unless the probe was manually shrunk, whereas MDL was much lower for true labels than for random labels without manual tuning. At the embedding layer, control-task accuracy sometimes even exceeded linguistic accuracy, but its codelength was twice as large. MDL scores were also stable across five random seeds and across ten hyperparameter settings, while raw accuracy rankings could flip (Voita et al., 2020).

4. Bayesian mutual information and finite-data probing

Classical mutual information assumes the true joint distribution $I(X;Y)=H(Y)-H(Y\mid X).$2 is known. In probing, this assumption creates unintuitive conclusions: by the Data-Processing Inequality, any representation $I(X;Y)=H(Y)-H(Y\mid X).$3 can only lose information about $I(X;Y)=H(Y)-H(Y\mid X).$4; and because $I(X;Y)=H(Y)-H(Y\mid X).$5 under the true distribution, classical theory says that seeing data does not change entropy. The Bayesian framework replaces these quantities with posterior-predictive beliefs of a Bayesian agent (Pimentel et al., 2021).

Suppose the agent models a random variable $I(X;Y)=H(Y)-H(Y\mid X).$6 with likelihood family $I(X;Y)=H(Y)-H(Y\mid X).$7 and prior $I(X;Y)=H(Y)-H(Y\mid X).$8. Given observed data $I(X;Y)=H(Y)-H(Y\mid X).$9, the posterior predictive distribution is

$p(Y\mid X)$0

The agent’s surprisal for a fresh sample $p(Y\mid X)$1 is

$p(Y\mid X)$2

and its posterior-predictive cross-entropy is

$p(Y\mid X)$3

If the agent has predictive beliefs $p(Y\mid X)$4 and $p(Y\mid X)$5, then Bayesian mutual information is defined as

$p(Y\mid X)$6

Under Bernstein–von Mises conditions and infinite data, $p(Y\mid X)$7, so $p(Y\mid X)$8, recovering classical mutual information (Pimentel et al., 2021).

This replacement yields three properties emphasized in the framework. First, “data can add information”: $p(Y\mid X)$9 strictly decreases with more data as $q_\theta(Y\mid X)$0 converges to $q_\theta(Y\mid X)$1, so $q_\theta(Y\mid X)$2 grows as $q_\theta(Y\mid X)$3 accumulates, mirroring learning curves. Second, “processing can help”: Bayesian mutual information need not obey the classical Data-Processing Inequality, because an appropriate transformation can simplify the agent’s posterior. Third, “information can hurt”: with a weak prior and a high-capacity model, early data can mislead the posterior so that $q_\theta(Y\mid X)$4, making $q_\theta(Y\mid X)$5. In probing terms, negative Bayesian mutual information captures overfitting (Pimentel et al., 2021).

For probing, the setup takes $q_\theta(Y\mid X)$6 to be an input sentence, $q_\theta(Y\mid X)$7 its learned representation, and $q_\theta(Y\mid X)$8 the target linguistic label. A probe is a Bayesian agent for $q_\theta(Y\mid X)$9, typically implemented as a parameterized softmax $R$0 together with an unconditional model $R$1 for the marginal. The prior is chosen so that initially $R$2, hence $R$3. As $R$4 grows, the posterior-predictive beliefs converge and $R$5 increases toward the true $R$6. The curve of $R$7 versus training-set size directly measures how easy it is to extract $R$8 from $R$9 under limited data. In practice, the exact posterior is intractable for neural probes, so the framework approximates it by a MAP estimate for the conditional and a Dirichlet-smoothed marginal (Pimentel et al., 2021).

The empirical studies followed this design on five embedding types—random, fastText, BERT, RoBERTa, and ALBERT—using UD treebanks for English, Basque, Marathi, and Turkish, on POS tagging and dependency-arc labeling. Random and fastText converged to the same $T$0 in the limit, but random embeddings overfit sooner and required more data to catch up. Among contextual models, ALBERT yielded the highest $T$1 with moderate data, while BERT and RoBERTa overfit more at low data. Arc-labeling curves rose more slowly than POS curves, indicating a harder probing task (Pimentel et al., 2021).

5. Variational bounds, probe capacity, and layer-wise behavior

A further development makes the link between probing and variational mutual-information estimation explicit. For an intermediate representation $T$2, mutual information satisfies

$T$3

and by the Donsker–Varadhan representation,

$T$4

Approximating the optimal critic by a neural network $T$5 gives the MINE lower bound $T$6; InfoNCE provides an alternative lower bound of the same kind (Choi et al., 2023).

Within this framework, a standard $T$7-way linear probe has critic

$T$8

and its softmax cross-entropy objective is equivalent to maximizing a variational lower bound on $T$9. Because the Donsker–Varadhan form does not structurally restrict $T$00, one may also let the critic consume downstream computation. In that sense, a linear probe on layer $T$01 and fine-tuning from layer $T$02 onward are mathematically the same procedure: both maximize a variational bound on $T$03, differing only in critic capacity (Choi et al., 2023).

This formulation also addresses the often-observed “convex” or peaky layer-wise performance curves. For a Markov chain $T$04, the Data Processing Inequality guarantees that the true $T$05 is non-increasing in $T$06. Non-monotonic empirical curves arise because the estimator $T$07 depends on critic capacity and geometric separability. Early layers may contain more true information but be less linearly separable, so a simple probe underestimates $T$08; intermediate layers may be most linearly separable, so the same probe yields a larger estimate; late layers may then decline again as separability drops under frozen features (Choi et al., 2023).

The paper formalizes this connection through margin. In a binary, balanced setting, if the classes are linearly separable with margin $T$09, Theorem 1 states

$T$10

Larger linear margin therefore implies a tighter variational bound and a more faithful MI estimate; the margin itself can be used as a measure of representational goodness. The same work also derives bounds relating estimated accuracy $T$11 to $T$12, showing that accuracy gives only a coarse bound on mutual information. Empirically, on CommonPhone with a frozen 24-layer XLS-R backbone, fine-tuning yielded the highest $T$13, linear probing the lowest, and word classification displayed a middle-layer peak in estimated mutual information (Choi et al., 2023).

6. Graph structures, inductive bias, and recurring interpretive debates

Information-theoretic probing has also been extended beyond label classification to graph-valued linguistic structure. In Bird’s Eye, a sentence-level linguistic graph $T$14 is embedded into a continuous space $T$15 using DeepWalk, and probing estimates $T$16 with a MINE objective. Because raw mutual information is difficult to compare across formalisms, the framework defines the normalized Mutual Information Gap,

$T$17

using random noise and noisy self-information as controls. Worm’s Eye then performs perturbation analysis on local substructures through the score

$T$18

On Penn Treebank dependencies and AMR 2.0, syntax MIG was high, approximately $T$19–$T$20, peaking in middle layers, while semantics MIG was lower, approximately $T$21–$T$22, and fairly flat. BERT encoded both syntactic and semantic graph structure, but syntactic information to a greater extent (Hou et al., 2021).

A different refinement shifts attention from information content to inductive bias. In this view, if $T$23 is a representation and $T$24 the target, the Bayes-optimal log-loss is

$T$25

while a variational probe $T$26 incurs risk

$T$27

Their difference,

$T$28

is the inductive bias gap. Rather than selecting probes by held-out loss alone, the framework computes the marginal likelihood

$T$29

for each probe architecture and prior, approximated by a Laplace approximation around the MAP estimate with a Kronecker-factored Hessian approximation. This implements Occam’s razor directly: highly expressive probes receive lower evidence on simple data if their complexity is not warranted. In the reported POS-tagging example for English, fastText achieved slightly higher evidence than BERT, suggesting a better inductive bias for that task under this framework (Immer et al., 2021).

Taken together, these formulations distinguish several quantities that standard probe accuracy conflates: asymptotic mutual information, contextual gain relative to controls, description length, finite-data ease of extraction, and inductive bias under a probe family. This suggests that many disagreements in the probing literature arise because different frameworks operationalize different questions. Some methods ask how much information is in principle present; some ask how much extra information context adds beyond a control; some ask how many bits or examples are required to recover it; and some ask which representation best aligns with the inductive bias of a selected hypothesis class (Pimentel et al., 2020, Pimentel et al., 2021, Voita et al., 2020, Immer et al., 2021).