DiffMean: Mean Contrast in Stats & Neural Representations

• DiffMean is a method that computes the contrast between two means by subtracting one from the other, underpinning both statistical inference and neural representation analysis.
• In statistics, it estimates effect size and uncertainty using classical approaches and Bayesian credible bounds, ensuring robust hypothesis testing even in complex models.
• In transformer interpretability, DiffMean identifies linear activation directions that correlate with concept detection and steering, offering practical insights for neural network analysis.

Searching arXiv for papers on “Difference-in-Means / DiffMean” to ground the article. Difference-in-Means (DiffMean) denotes a family of mean-contrast constructions built by subtracting one empirical mean from another. In the cited literature, the term spans at least two technically distinct but structurally related uses: a classical statistical contrast such as $D=\bar X-\bar Y$ for two-sample inference, and a representation-space direction $w=\mu^{+}-\mu^{-}$ extracted from neural activations for concept detection or activation steering. Related Bayesian variants, notably the most difference in means $\delta_M$ and the least difference in means $\delta_L$, replace the raw contrast by conservative credible bounds that quantify null strength or practical significance [(Shilane et al., 2012); (Wu et al., 28 Jan 2025); (Corliss et al., 2022); (Corliss et al., 2022)].

1. Conceptual scope and nomenclature

Across the cited work, the common object is a contrast between two means. In classical inference, the target is usually a population mean difference $\Delta=\mu_1-\mu_2$, estimated by the sample difference $D=\bar X-\bar Y$. In recent transformer studies, the same algebraic template is applied to hidden states: one averages residual-stream activations over a positive set and a negative set, then uses the resulting mean-shift vector as a detector or steering direction [(Shilane et al., 2012); (Vennemeyer et al., 25 Sep 2025); (Prokopiou et al., 29 May 2026)].

Setting	Object	Representative definition
Two-sample inference	Mean-difference estimator	$D=\bar X-\bar Y$
Null/effect-strength statistics	Credible bound on $|\mu_y-\mu_x|$	$\delta_M=Q_{\mathrm{raw}}(1-\alpha_{\delta_M})$, $$\delta_L=\mathrm{sign}(\bar Y-\bar X)\max\{|\bar Y-\bar X|-t_{\nu,1-\alpha}s,0\}$$
Transformer representation analysis	Mean-shift direction in activation space	$w=\mu^{+}-\mu^{-}$0, sometimes unit-normalized

This shared structure is important because it explains why DiffMean appears in otherwise distant literatures. In each case, the central question is not merely whether two groups differ, but how a mean contrast can be turned into an interpretable scalar or direction. In statistical settings, that interpretation concerns uncertainty, effect size, and hypothesis testing. In mechanistic interpretability, it concerns linear separability, causal steering, and subspace geometry (Corliss et al., 2022, Corliss et al., 2022, Wu et al., 28 Jan 2025).

2. Classical statistical formulation

For independent two-sample inference, the basic DiffMean estimator is

$w=\mu^{+}-\mu^{-}$1

with target parameter $w=\mu^{+}-\mu^{-}$2. In the negative-binomial setting treated by Shilane and Bean, $w=\mu^{+}-\mu^{-}$3 is unbiased for $w=\mu^{+}-\mu^{-}$4, and under a normal approximation its variance is

$w=\mu^{+}-\mu^{-}$5

leading to the interval

$w=\mu^{+}-\mu^{-}$6

A notable result is that the normal approximation is considerably more robust in approximating the distribution of the mean difference than the corresponding one-sample theory would suggest, even in highly dispersed Negative Binomial models (Shilane et al., 2012).

The same literature also emphasizes that a raw mean difference is not itself a standardized effect size. Aoki et al. define the unequal-variance effect-size statistic

$w=\mu^{+}-\mu^{-}$7

where $w=\mu^{+}-\mu^{-}$8 is Welch’s $w=\mu^{+}-\mu^{-}$9, $\delta_M$0, and $\delta_M$1 is the Welch–Satterthwaite degrees of freedom. This construction is explicitly aligned with Welch’s unequal-variance $\delta_M$2 test, in contrast to Hedges’ $\delta_M$3, which assumes equal variances (Aoki et al., 2019). The point is not to replace the mean difference, but to standardize it under heteroskedasticity.

These results establish the baseline meaning of DiffMean: a direct estimator of a mean contrast. Subsequent developments focus on how to preserve that contrast while correcting for nonstandard designs, post-selection, missingness, or a different inferential goal.

3. Bayesian bounds: null strength and practical significance

Corliss et al. recast difference in means as a conservative posterior bound. Under a Behrens–Fisher model with noninformative prior

$\delta_M$4

they define the most difference in means

$\delta_M$5

where $\delta_M$6 is the $\delta_M$7th upper quantile of the posterior of $\delta_M$8, so that

$\delta_M$9

Thus, $\delta_L$0 is an upper credible bound on the magnitude of the effect, intended to quantify null strength rather than detect nonzero effects (Corliss et al., 2022).

This construction supports an equivalence-style test against a negligible-effect threshold $\delta_L$1: $\delta_L$2 One rejects $\delta_L$3 if $\delta_L$4. A distinctive property is threshold-independence after estimation: once $\delta_L$5 is computed, different values of $\delta_L$6 can be applied with no re-estimation required. Corliss et al. also define the relative most-difference statistic

$\delta_L$7

for $\delta_L$8, to compare null strength across experiments with heterogeneous raw scales or control means that differ by orders of magnitude (Corliss et al., 2022).

The complementary construction is the least difference in means $\delta_L$9, introduced for effect strength and practical significance. Under the two-sample $\Delta=\mu_1-\mu_2$0-approximation,

$\Delta=\mu_1-\mu_2$1

with $\Delta=\mu_1-\mu_2$2 and $\Delta=\mu_1-\mu_2$3. Here $\Delta=\mu_1-\mu_2$4 is a lower credible bound on effect magnitude. Practical-significance testing against threshold $\Delta=\mu_1-\mu_2$5 reduces to the rule $\Delta=\mu_1-\mu_2$6 (Corliss et al., 2022).

The two constructions are dual in role. $\Delta=\mu_1-\mu_2$7 is designed for concluding that an effect is negligible, whereas $\Delta=\mu_1-\mu_2$8 is designed for concluding that an effect is meaningfully non-negligible. In simulation, $\Delta=\mu_1-\mu_2$9 and $D=\bar X-\bar Y$0 were the only statistics whose comparison error stayed below $D=\bar X-\bar Y$1 across all null-strength scenarios, and $D=\bar X-\bar Y$2 and its relative form were the only candidates whose pairwise error rate was consistently $D=\bar X-\bar Y$3 across all four independently varied measures of effect strength (Corliss et al., 2022, Corliss et al., 2022).

4. Selective inference, missingness, and partially matched designs

A recurrent theme in the difference-in-means literature is that the contrast itself may be simple while its valid inferential calibration is not. After $D=\bar X-\bar Y$4-means clustering, for example, the naive test based on the observed cluster means fails because the same data are used both to form clusters and to test differences between them. Tsukurimichi et al. address this by conditioning on the full $D=\bar X-\bar Y$5-means assignment path and showing that, under the null, the norm contrast

$D=\bar X-\bar Y$6

has a truncated $D=\bar X-\bar Y$7-law, yielding a finite-sample selective $D=\bar X-\bar Y$8-value that controls selective Type I error exactly (Chen et al., 2022).

Chen and Gao et al. develop the analogous feature-wise problem after hierarchical or $D=\bar X-\bar Y$9-means clustering. For a fixed feature $D=\bar X-\bar Y$0, the observed contrast is

$D=\bar X-\bar Y$1

and the selective $D=\bar X-\bar Y$2-value is obtained from a truncated-Gaussian pivot over the set of perturbations that preserve the clustering event. The paper proves exact finite-sample Type I error control under the Gaussian model and shows that the naive two-sample Z-test is heavily anti-conservative in simulation (Chen et al., 2023).

When data are paired but partially missing under MCAR, Li et al. define a weighted mean-difference estimator

$D=\bar X-\bar Y$3

with a Wald-type statistic

$D=\bar X-\bar Y$4

They derive optimal weights by minimizing the asymptotic variance and report that the proposed test with optimal weights performs well and outperforms existing methods in practical situations, while keeping Type I error near nominal in the reported simulations (Li et al., 2021).

A related problem arises in partially matched pre–post studies. Pomponio et al. propose the Quantile-based $D=\bar X-\bar Y$5-test for correlated samples, which uses all $D=\bar X-\bar Y$6 observations and replaces the unknown correlation $D=\bar X-\bar Y$7 by a conservative lower confidence bound $D=\bar X-\bar Y$8 estimated from the matched subset via Fisher’s $D=\bar X-\bar Y$9-transform. Their results show nominal Type I error probability and more power than both the paired-only $|\mu_y-\mu_x|$0-test and the two-sample $|\mu_y-\mu_x|$1-test in the reported settings (Pomponio et al., 2023).

5. DiffMean in transformer representation spaces

In transformer interpretability, DiffMean is a representation-level procedure for extracting a direction associated with a concept, behavior, or attribute. The generic recipe is to partition examples into positive and negative sets, extract hidden activations at a chosen site, compute the corresponding means, and subtract them. Depending on the paper, the resulting direction is used for linear probing, activation addition, or both (Wu et al., 28 Jan 2025, Vennemeyer et al., 25 Sep 2025, Prokopiou et al., 29 May 2026).

Paper	Activation site	DiffMean construction and use
"AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders" (Wu et al., 28 Jan 2025)	All token representations at layer $|\mu_y-\mu_x|$2	$|\mu_y-\mu_x|$3, $|\mu_y-\mu_x|$4, detection by $|\mu_y-\mu_x|$5, steering by $|\mu_y-\mu_x|$6
"Sycophancy Is Not One Thing: Causal Separation of Sycophantic Behaviors in LLMs" (Vennemeyer et al., 25 Sep 2025)	Post-layernorm residual stream at EOS	$|\mu_y-\mu_x|$7, used as both probe and steering vector
"Latent Space Disentanglement via Activation Steering for Interpretable Attribute Control in Symbolic Music Generation" (Prokopiou et al., 29 May 2026)	Residual stream at the last valid token	$|\mu_y-\mu_x|$8, injected as $|\mu_y-\mu_x|$9

AxBench defines positives and negatives from labeled concept data, aggregates token representations from the selected layer, and normalizes the mean difference to unit norm before scoring or steering. This normalized direction is then used for token-wise dot-product detection and activation-addition steering (Wu et al., 28 Jan 2025).

The sycophancy study uses a behavior-specific raw DiffMean direction without additional regularization: $\delta_M=Q_{\mathrm{raw}}(1-\alpha_{\delta_M})$0 Positive examples are those where a target behavior is present, negatives are all other examples, and activations are taken at the EOS token of the model response. At inference time, steering modifies each residual activation after the user prompt by adding $\delta_M=Q_{\mathrm{raw}}(1-\alpha_{\delta_M})$1 (Vennemeyer et al., 25 Sep 2025).

In symbolic music generation, the authors form non-overlapping concept clusters $\delta_M=Q_{\mathrm{raw}}(1-\alpha_{\delta_M})$2 and $\delta_M=Q_{\mathrm{raw}}(1-\alpha_{\delta_M})$3 from the 20th and 80th empirical quantiles of an attribute’s distribution, balance each set to $\delta_M=Q_{\mathrm{raw}}(1-\alpha_{\delta_M})$4 examples, and compute layer-specific centroids $\delta_M=Q_{\mathrm{raw}}(1-\alpha_{\delta_M})$5 and $\delta_M=Q_{\mathrm{raw}}(1-\alpha_{\delta_M})$6 from residual-stream activations at the last valid token. The resulting direction $\delta_M=Q_{\mathrm{raw}}(1-\alpha_{\delta_M})$7 is injected during inference, and multi-attribute control is refined by Gram-Schmidt Orthogonalization in a Dual Steering framework (Prokopiou et al., 29 May 2026).

6. Empirical behavior, strengths, and limitations

The empirical profile of DiffMean differs by task. In AxBench, representation-based methods such as difference-in-means perform the best for concept detection, while prompting outperforms all existing methods for steering. DiffMean achieved mean ROC AUC $\delta_M=Q_{\mathrm{raw}}(1-\alpha_{\delta_M})$8 across four evaluation sites, compared with $\delta_M=Q_{\mathrm{raw}}(1-\alpha_{\delta_M})$9 for Probe, $$\delta_L=\mathrm{sign}(\bar Y-\bar X)\max\{|\bar Y-\bar X|-t_{\nu,1-\alpha}s,0\}$$0 for ReFT-r1, $$\delta_L=\mathrm{sign}(\bar Y-\bar X)\max\{|\bar Y-\bar X|-t_{\nu,1-\alpha}s,0\}$$1 for Prompt, $$\delta_L=\mathrm{sign}(\bar Y-\bar X)\max\{|\bar Y-\bar X|-t_{\nu,1-\alpha}s,0\}$$2 for SAE, and $$\delta_L=\mathrm{sign}(\bar Y-\bar X)\max\{|\bar Y-\bar X|-t_{\nu,1-\alpha}s,0\}$$3 for PCA. The benchmark therefore identifies DiffMean as a very strong baseline for open-vocabulary concept detection, but not as the strongest steering method in that benchmark (Wu et al., 28 Jan 2025).

In the sycophancy study, DiffMean directions isolate three behaviors—Sycophantic Agreement, Genuine Agreement, and Sycophantic Praise—as distinct linear directions. SyPr becomes linearly separable by layer 8 with AUROC approximately $$\delta_L=\mathrm{sign}(\bar Y-\bar X)\max\{|\bar Y-\bar X|-t_{\nu,1-\alpha}s,0\}$$4–$$\delta_L=\mathrm{sign}(\bar Y-\bar X)\max\{|\bar Y-\bar X|-t_{\nu,1-\alpha}s,0\}$$5, while SyA and GA are only moderately separated early and then split sharply around layers $$\delta_L=\mathrm{sign}(\bar Y-\bar X)\max\{|\bar Y-\bar X|-t_{\nu,1-\alpha}s,0\}$$6–$$\delta_L=\mathrm{sign}(\bar Y-\bar X)\max\{|\bar Y-\bar X|-t_{\nu,1-\alpha}s,0\}$$7, reaching AUROC $$\delta_L=\mathrm{sign}(\bar Y-\bar X)\max\{|\bar Y-\bar X|-t_{\nu,1-\alpha}s,0\}$$8. Subspace geometry shows SyA and GA nearly collinear early, with cosine approximately $$\delta_L=\mathrm{sign}(\bar Y-\bar X)\max\{|\bar Y-\bar X|-t_{\nu,1-\alpha}s,0\}$$9, then separating to approximately $w=\mu^{+}-\mu^{-}$00, while SyPr stays orthogonal to both with cosine $w=\mu^{+}-\mu^{-}$01. Steering selectivity was reported as approximately $w=\mu^{+}-\mu^{-}$02 for SyA, $w=\mu^{+}-\mu^{-}$03 for GA, and $w=\mu^{+}-\mu^{-}$04 for SyPr in Qwen3-30B, and on TruthfulQA the SyA steering selectivity was $w=\mu^{+}-\mu^{-}$05 (Vennemeyer et al., 25 Sep 2025).

In symbolic music generation, the reported evidence supports the Linear Representation Hypothesis in the tested domain. Steering magnitude $w=\mu^{+}-\mu^{-}$06 showed strong linear correlations with attribute shift: for Pitch, Pearson $w=\mu^{+}-\mu^{-}$07 and $w=\mu^{+}-\mu^{-}$08; for Duration, Pearson $w=\mu^{+}-\mu^{-}$09 and $w=\mu^{+}-\mu^{-}$10. In strong context-override tests, pitch vectors succeeded $w=\mu^{+}-\mu^{-}$11 of the time with average shift $w=\mu^{+}-\mu^{-}$12 semitones, and duration vectors succeeded $w=\mu^{+}-\mu^{-}$13 with average shift $w=\mu^{+}-\mu^{-}$14 ticks. The same paper also reports that concept entanglement can be substantial, with pitch/duration cosine similarity up to $w=\mu^{+}-\mu^{-}$15, and that very large $w=\mu^{+}-\mu^{-}$16 lead to perceptual degradation marked by an exponential rise in $w=\mu^{+}-\mu^{-}$17 (Prokopiou et al., 29 May 2026).

Several misconceptions are corrected by these results. First, DiffMean is not a single inferential object: it denotes both a classical mean contrast and a hidden-state mean-shift direction. Second, strong performance in concept detection does not imply superior steering performance in every benchmark; AxBench explicitly reported the opposite for prompting versus representation-based steering (Wu et al., 28 Jan 2025). Third, current activation-space uses rely on linear contrasts of means, so nonlinear entanglement may still exist, and orthogonalization or subspace removal may be needed when behaviors or attributes are not cleanly separated (Vennemeyer et al., 25 Sep 2025, Prokopiou et al., 29 May 2026).