Accurate and Efficient Statistical Testing for Word Semantic Breadth

Abstract: Measuring the breadth of a word's meaning, or its spread across contexts, has become feasible with contextualized token embeddings. A word type can be represented as a cloud of token vectors, with dispersion-based statistics serving as proxies for contextual diversity (Nagata and Tanaka-Ishii, ACL2025). These measurements are useful for deciding appropriate sense distinctions when constructing thesauri and domain-specific dictionaries. However, when comparing the breadth of two word types, naive hypothesis testing on dispersion can be misleading: differences in semantic direction can masquerade as dispersion differences, inflating Type-I error and yielding "statistically significant" outcomes even when there is no true breadth difference. This is problematic because significance testing should distinguish genuine effects from incidental fluctuations in small-difference regimes. We propose a Householder-aligned permutation test to isolate dispersion differences from directional differences. Our method applies a single Householder reflection to align the mean directions of the two word types and then performs a permutation test on the aligned token clouds, yielding calibrated, non-parametric p-values. For practicality, we introduce a GPU-oriented implementation that batches permutations and linear algebra operations. Empirically, our alignment reduced Type-I error by 32.5% while preserving sensitivity to genuine breadth differences, and achieved a 23x speedup over the CPU baseline.

• The paper introduces a Householder-aligned permutation test that removes mean-direction differences before comparing word-embedding dispersion, preventing inflated false positives.
• The method reduced Type-I error by up to 32.5% and improved precision from 41.7% to 62.5% at a ranking gap of 10, though with a modest power trade-off.
• A vectorized GPU implementation using blocked GEMM processing achieved a 23× speedup, reducing 20,000-permutation runtime from 31,750 ms to 1,377 ms for large-scale analyses.

Motivation and problem statement

Dispersion-based statistics computed from contextualized token embeddings have emerged as corpus-driven proxies for a word type's semantic breadth: a word whose token vectors, after $\ell_2$-normalization onto the unit sphere, form a widely spread cloud is taken to be contextually more diverse than one whose cloud is concentrated. Such measurements are useful in lexicography, where they can inform decisions about sense granularity and annotation effort. However, comparing the breadth of two word types requires statistical testing, and the paper identifies a failure mode of the obvious approach: applying a standard two-sample permutation test directly to dispersion statistics conflates two distinct sources of geometric difference. Two word types may have identical dispersions but different mean directions on the sphere; under naive label permutation, permuted groups mix points from both directional regions and appear artificially dispersed, so the permutation null no longer corresponds to the intended hypothesis. The result is inflated Type-I error — "significant" breadth differences declared where none exist — precisely in the small-difference regime where significance testing is most needed.

The paper's formalization makes this explicit. Given unit-normalized token sets $X$ and $Y$ for two words, the null hypothesis is that the two dispersions are equal while mean directions are allowed to differ; mean direction is treated as a nuisance factor. Under this null, exchangeability of pooled samples fails unless the directional mismatch is removed first.

Householder-aligned permutation test

The proposed remedy is a single orthogonal transformation applied before permutation. Let $\hat{\mu}_x$ and $\hat{\mu}_y$ be the normalized sample mean directions of $X$ and $Y$. With axis $$\mathbf{u} = (\hat{\mu}_x - \hat{\mu}_y)/\|\hat{\mu}_x - \hat{\mu}_y\|_2$$, the Householder reflection $$\mathbf{H} = \mathbf{I} - 2\mathbf{u}\mathbf{u}^\top$$ satisfies $\mathbf{H}\hat{\mu}_x = \hat{\mu}_y$. Applying $X$0 to every vector in $X$1 aligns the mean directions while preserving all within-set distances and rotation-invariant dispersion statistics, since $X$2 is orthogonal.

The test statistic is built from the mean resultant length (MRL) $X$3, which relates monotonically to the concentration parameter of directional models such as the von Mises–Fisher distribution via a mapping $X$4. Semantic breadth is defined as inverse concentration $X$5, and the observed statistic is the log-volume difference $X$6 computed on the aligned set $X$7. A Monte Carlo permutation $X$8-value with the standard $X$9 correction is then obtained by pooling the aligned data and randomly reassigning labels over $Y$0 permutations.

Two design choices deserve note. First, the alignment is computed once from the observed pair and the aligned pooled set is held fixed across permutations ("fixed-space" design); re-estimating $Y$1 per permutation would make the geometry depend on permuted labels. A split-half sanity check on same-word occurrences confirms that baseline and proposed methods yield nearly identical $Y$2-values when mean directions already coincide, as expected. Second, the authors prove (via a triangle-inequality argument) that the Householder transform maximizes the MRL of the merged set among orthogonal transformations, providing a principled justification for choosing it over alternatives such as Procrustes alignment, which requires pointwise correspondences unavailable for unpaired token sets of unequal size.

GPU-accelerated permutation inference

Because permutation testing dominates the computational cost ($Y$3 per word pair), the paper introduces a vectorized GPU formulation. Each permutation is encoded as a sign vector with exactly $Y$4 entries of $Y$5 and $Y$6 entries of $Y$7; stacking these into a sign matrix $Y$8, the signed group-difference sums are obtained in a single GEMM call $Y$9. Since group sums must total a fixed vector $\hat{\mu}_x$0, both group sums — and hence both permuted MRLs and test statistics — are recovered without a second matrix multiplication. Permutations are processed in blocks with streaming exceedance counts, giving $\hat{\mu}_x$1 working memory regardless of $\hat{\mu}_x$2. The Householder transform itself is applied without materializing the $\hat{\mu}_x$3 matrix, using the rank-one update $\hat{\mu}_x$4. Half-precision GEMM with float32 accumulation guards against numerical drift near $\hat{\mu}_x$5, and sign blocks can be reused across word pairs sharing the same group sizes.

Benchmarked on BCCWJ/BERT-large embeddings with $\hat{\mu}_x$6 permutations, the GPU implementation achieves 0.069 ms per permutation versus 1.588 ms on CPU — a 23× speedup (31,750 ms down to 1,377 ms total). This makes large-scale testing across thousands of word pairs practical.

Empirical evaluation

The main evaluation uses BNC with ModernBERT, sampling 300 word pairs per dispersion-ranking gap from 1 to 10 at $\hat{\mu}_x$7. Following prior practice, pairs within a ranking gap of 100 are treated as having no true breadth difference, so rejections at small gaps constitute Type-I errors; WordNet synset counts serve as gold standard, and precision measures how often rejected pairs genuinely differ in sense count.

Key results:

Gap	Type-I error (baseline → proposed)	Precision (baseline → proposed)
5	0.017 → 0.013	0.600 → 0.750
9	0.027 → 0.023	0.375 → 0.286
10	0.040 → 0.027 (32.5% reduction)	0.417 → 0.625

At gap=10 the method simultaneously reduces Type-I error from 4.0% to 2.67% and improves precision from 41.7% to 62.5%; at gaps 1–3 both methods control error near or below nominal level. Across all ten gaps, the proposed method wins on Type-I error in three cases, ties in six, and loses once (gap=4); precision shows the same pattern with a single loss at gap=9. These gains indicate that removing the directional confounder lets the test focus on genuine dispersion differences rather than semantic-direction artifacts.

Additional experiments across three corpus–model combinations (BNC/BERT-tiny, BNC/ModernBERT, BCCWJ/BERT-large) with 500 pairs each at $\hat{\mu}_x$8 confirm the pattern: under the null condition (gap=50), rejection rates drop from 3.2–4.0% (baseline) to 2.2–2.4% (proposed), while under the alternative condition (gap=100) rejection rates fall from 6.4–9.0% to 4.4–6.2%. The latter constitutes an acknowledged power trade-off: alignment yields a more stringent null distribution and slightly lower power, which the authors accept in exchange for better calibration. Qualitative examples are consistent with expectations — broad words like mark (30 WordNet senses) show high dispersion proxy values, while specialized nouns like colitis (1 sense) show low dispersion.

Limitations and open questions

The paper concedes several constraints plainly. Evaluation coverage is restricted to a limited set of corpora and encoder families; broader validation across genres, languages, and architectures remains necessary. More fundamentally, the dispersion statistic inherits the unimodal, isotropic assumptions of von Mises–Fisher-style modeling, whereas real token clouds can be multimodal or anisotropic; the authors do not claim to resolve anisotropy in general, and such geometric deviations may affect both effect sizes and calibration. Tokenizer constraints (subword splitting) further limit which word types can be analyzed cleanly. The additional experiments also reveal that improved calibration comes at some cost in statistical power, and the paper does not characterize when this trade-off becomes unfavorable. Finally, extension to sentence-level embeddings is deferred because anisotropy concerns complicate dispersion interpretation at that level — leaving open whether the framework transfers beyond word-type token clouds.

Conclusion

This work reframes dispersion-based comparison of word semantic breadth as a calibrated hypothesis-testing problem and shows that naive permutation tests fail in a specific, diagnosable way: mean-direction differences break exchangeability and inflate false positives. A single Householder reflection removes the nuisance factor provably and cheaply, reducing Type-I error by up to 32.5% while preserving or improving precision against WordNet sense counts, and a GEMM-centric GPU implementation delivers a 23× speedup that makes resampling-based inference feasible at scale. The result offers lexicographers a principled tool for prioritizing sense-annotation effort, though its generality across anisotropic embedding geometries and non-word-level representations remains to be established.