Theory for dependent token-level samples and high-dimensional regimes

Extend the theoretical analysis of the ASK-NN asymmetric two-sample test to dependent token-level samples and high-dimensional regimes in order to better align the method with large-language-model applications and improve hallucination detection in practical retrieval-augmented-generation systems.

Background

The theoretical results in the paper establish asymptotic normality and consistency under assumptions based on independent samples and fixed-dimensional analysis. The conclusion explicitly leaves unresolved the extension of this theory to dependent token-level samples and high-dimensional regimes, which are characteristic of hidden-state sequences in LLMs and are important for practical hallucination detection.

References

Several directions remain open. First, the asymptotic calibration can be inaccurate for real hidden-state embeddings, where dependence, anisotropy, and high dimensionality are substantial; permutation or block-resampling calibration may provide more robust alternatives. Second, extending the theory to dependent token-level samples and high-dimensional regimes would make the method better aligned with LLM applications and improve hallucination detection in practical RAG systems.

— ASK-NN: An Asymmetric Nearest-Neighbor Test that detects Distribution Drifts in Natural Language  (2607.15607 - Zakharov et al., 17 Jul 2026) in Section Conclusion