--- title: Information-Theoretic Probing Framework url: https://www.emergentmind.com/topics/information-theoretic-probing-framework type: topic --- # Information-Theoretic Probing Framework Information-theoretic probing is an approach to analyzing learned representations in which the central object is not probe accuracy itself, but an information-theoretic quantity relating a representation to a target property. In the basic formulation, if \(R\) denotes a learned representation and \(T\) a linguistic label, probing is cast as estimating \(I(T;R)=H(T)-H(T\mid R)\). Subsequent work extended this perspective in several directions: minimum description length (MDL) treats probing as data compression; Bayesian mutual information replaces classical mutual information in finite-data regimes; variational bounds connect linear probing, MINE, InfoNCE, and fine-tuning; and Bayesian evidence reframes probing as model selection for inductive bias. Across these variants, the common aim is to separate information encoded in the representation from effects due to probe capacity, finite data, regularization, and memorization [2004.03061] [2003.12298] [2109.03853] [2312.10019] [2110.08388]. ## 1. Mutual information as the core operationalization A standard formalization takes \(X\) to be a pre-computed representation and \(Y\) the discrete label to be probed. Mutual information is then \[ I(X;Y)=H(Y)-H(Y\mid X). \] Because the true conditional \(p(Y\mid X)\) is unknown, one trains a probe \(q_\theta(Y\mid X)\) and evaluates its cross-entropy. In the formulation of Pimentel et al., if \(R\) is the representation and \(T\) the target label, the empirical conditional cross-entropy \[ \hat H_q(T\mid R)= - \frac{1}{N}\sum_{i=1}^N \log q_\theta(t_i\mid r_i) \] upper-bounds \(H(T\mid R)\), which yields the lower bound \[ I(T;R)\ge H(T)-\hat H_q(T\mid R). \] The bound becomes tight when \(q_\theta(t\mid r)\approx p(t\mid r)\) in KL-divergence, so improving probe fit tightens the mutual-information estimate [2004.03061]. This perspective also gives an exact decomposition of probe cross-entropy. If \(q_\theta\) is trained by minimizing \[ H(p,q_\theta)\equiv -E_{x,y\sim p(x,y)}\log q_\theta(y\mid x), \] then \[ H(p,q_\theta)=H(Y)-I(X;Y)+D_{KL}(p(Y\mid X)\|q_\theta(Y\mid X)). \] Since \(H(Y)\) is fixed, a low cross-entropy can arise only from either a large \(I(X;Y)\) or a small approximation error \(D_{KL}(p\|q_\theta)\). This recovers the Hewitt–Liang “rich-representation vs. probe-learns-task” dichotomy exactly, but in explicitly information-theoretic form [2009.07364]. Within this formulation, the earlier preference for deliberately weak probes is not information-theoretically justified. A more expressive probe family can better approximate the unknown conditional \(p(t\mid r)\), attain a lower \(\hat H_q(T\mid R)\), and therefore produce a tighter lower bound on \(I(T;R)\). In that specific sense, the framework argues that “bigger probes are better” when the goal is estimation of information already present in the representation [2004.03061]. ## 2. Controls, randomized baselines, and probe selection Information-theoretic probing did not eliminate the need for controls; rather, it gave them a formal role. One line of work studies randomized targets. In the control-task construction, true labels \(Y\) are replaced by random labels \(Y_{\mathrm{ctrl}}=c(Y)\) drawn independently of \(X\), and the selectivity score is \[ \Delta_h = H(p_c,q_{\theta_c}) - H(p,q_\theta). \] A parallel construction randomizes the representation, defining \(X_{\mathrm{ctrl}}=c(X)\) independently of \(Y\), with information-gain target \[ G = I(Y;X)-I(Y;X_{\mathrm{ctrl}}) \] and estimator \[ \widetilde G = H(p_c,q_{\phi_c})-H(p,q_\phi)=:\Delta_p. \] The two criteria satisfy \[ \Delta_h = I(X;Y)-\varepsilon_h,\qquad \Delta_p = I(X;Y)-\varepsilon_p, \] and, under a perfectly random control, their difference is constant with respect to model capacity. On this basis, the two selection criteria rank probe architectures identically [2009.07364]. The same section of the literature also introduced deterministic control functions for contextualized representations. If \(c(R)\) is a type-level control representation, then the contextual gain is \[ G(T;R,c)=I(T;R)-I(T;c(R))=H(T\mid c(R))-H(T\mid R)\ge 0. \] Empirically, this construction was used to compare multilingual BERT to type-level controls such as fastText and learnable one-hot embeddings. On part-of-speech tagging across eleven languages, the contextual gain was small: BERT added at most approximately \(9\%\) more information than fastText in the five languages where it did so, with negative estimation-error gains in the others. On dependency labeling, gains were larger but still modest: at most approximately \(12\%\) more information in English, and less than \(5\%\) in over half the languages [2004.03061]. These results situate control-based probing in a sharper way than accuracy comparisons alone. Randomized labels or representations remove memorization effects; type-level controls estimate what portion of the information is already available without context. Empirically, on POS-tag probing over English, French, and Spanish UD data, more than \(10{,}000\) hyperparameter settings produced highly agreeing rankings under \(\Delta_h\) and \(\Delta_p\), indicating that control-task and control-function selection criteria are equivalent for model comparison [2009.07364]. ## 3. Minimum description length and the notion of effort A distinct information-theoretic reformulation treats probing as data compression. Given a dataset \(D=\{(x_i,y_i)\}_{i=1}^n\), Alice knows both \(x_{1:n}\) and \(y_{1:n}\), while Bob knows only \(x_{1:n}\). If both parties agree on a fixed probabilistic model \(p(y\mid x)\), Shannon’s source coding theorem gives the optimal codelength \[ L_p(y_{1:n}\|x_{1:n}) = -\sum_{i=1}^n \log_2 p(y_i\mid x_i), \] so probe loss becomes a codelength. MDL probing adds the cost of the model itself: \[ L(D\mid R)=L(\mathrm{model})+L_p(y_{1:n}\|x_{1:n}). \] The central quantity is therefore not only the final predictive quality, but the amount of effort required to achieve that quality [2003.12298]. Two practical estimators were proposed. Variational coding uses a prior \(\alpha(\theta)\) and a variational approximation \(\beta(\theta)\), yielding \[ L^{\mathrm{var}}_\beta(y_{1:n}\|x_{1:n}) = KL(\beta\|\alpha)-E_{\theta\sim\beta}\sum_{i=1}^n \log_2 p_\theta(y_i\mid x_i). \] The first term is the cost of transmitting the model relative to the prior; the second is the expected data codelength. Minimizing this objective is equivalent to maximizing the ELBO. Online coding, by contrast, avoids explicitly sending the model: the sender and receiver agree on architecture, initialization, optimizer, and a schedule \(1=t_0H_B(X\mid D)\), making \(I_B(Y\to X\mid D)<0\). In probing terms, negative Bayesian mutual information captures overfitting [2109.03853]. For probing, the setup takes \(S\) to be an input sentence, \(R=f(S)\in\mathbb{R}^d\) its learned representation, and \(T\) the target linguistic label. A probe is a Bayesian agent for \(p(T\mid R)\), typically implemented as a parameterized softmax \(q(t\mid r,\theta)\) together with an unconditional model \(q(t\mid \phi)\) for the marginal. The prior is chosen so that initially \(q(t\mid r,D_0)=q(t\mid D_0)=\mathrm{uniform}\), hence \(I_B(R\to T\mid D_0)=0\). As \(|D|\) grows, the posterior-predictive beliefs converge and \(I_B(R\to T\mid D)\) increases toward the true \(I(S;T)\). The curve of \(I_B\) versus training-set size directly measures how easy it is to extract \(T\) from \(R\) under limited data. In practice, the exact posterior is intractable for neural probes, so the framework approximates it by a MAP estimate for the conditional and a Dirichlet-smoothed marginal [2109.03853]. The empirical studies followed this design on five embedding types—random, fastText, BERT, RoBERTa, and ALBERT—using UD treebanks for English, Basque, Marathi, and Turkish, on POS tagging and dependency-arc labeling. Random and fastText converged to the same \(I_B\) in the limit, but random embeddings overfit sooner and required more data to catch up. Among contextual models, ALBERT yielded the highest \(I_B\) with moderate data, while BERT and RoBERTa overfit more at low data. Arc-labeling curves rose more slowly than POS curves, indicating a harder probing task [2109.03853]. ## 5. Variational bounds, probe capacity, and layer-wise behavior A further development makes the link between probing and variational mutual-information estimation explicit. For an intermediate representation \(H^i=f^{1\ldots i}(X)\), mutual information satisfies \[ I(Y;H^i)=D_{KL}(P_{Y,H^i}\,\|\,P_Y\otimes P_{H^i}), \] and by the Donsker–Varadhan representation, \[ I(Y;H^i) = \sup_T \left\{ E_{P_{Y,H^i}}[T(y,h)] - \log E_{P_Y\otimes P_{H^i}}[e^{T(y,h)}] \right\}. \] Approximating the optimal critic by a neural network \(T_\theta\) gives the MINE lower bound \(\hat I_\theta(Y;H^i)\le I(Y;H^i)\); InfoNCE provides an alternative lower bound of the same kind [2312.10019]. Within this framework, a standard \(C\)-way linear probe has critic \[ T_\theta(y,h)=\mathbf{1}\{y\}^\top(Wh+b), \] and its softmax cross-entropy objective is equivalent to maximizing a variational lower bound on \(I(Y;H^i)\). Because the Donsker–Varadhan form does not structurally restrict \(T_\theta\), one may also let the critic consume downstream computation. In that sense, a linear probe on layer \(i\) and fine-tuning from layer \(i\) onward are mathematically the same procedure: both maximize a variational bound on \(I(Y;H^i)\), differing only in critic capacity [2312.10019]. This formulation also addresses the often-observed “convex” or peaky layer-wise performance curves. For a Markov chain \(X\to H^1\to \cdots \to H^L\), the Data Processing Inequality guarantees that the true \(I(Y;H^i)\) is non-increasing in \(i\). Non-monotonic empirical curves arise because the estimator \(\hat I_\theta(Y;H^i)\) depends on critic capacity and geometric separability. Early layers may contain more true information but be less linearly separable, so a simple probe underestimates \(I\); intermediate layers may be most linearly separable, so the same probe yields a larger estimate; late layers may then decline again as separability drops under frozen features [2312.10019]. The paper formalizes this connection through margin. In a binary, balanced setting, if the classes are linearly separable with margin \(d\), Theorem 1 states \[ |I(Y;H)-\hat I_\theta(Y;H)|