Efficient EigenScore (EES) Overview

• The paper introduces EES as an efficient approximation to EigenScore by replacing explicit covariance and eigendecomposition with Chebyshev-based DOS estimation.
• EES quantifies hallucination risk by measuring semantic variability across multiple generated outputs, outputting values on a scaled [-1,1] range.
• Integrated into the SenD training protocol, EES enables real-time monitoring and achieves up to a twofold speed improvement in model training.

Searching arXiv for the primary paper and closely related EigenScore references. Efficient EigenScore (EES) is an unsupervised hallucination detection metric introduced in “Hallucination Detox: Sensitivity Dropout (SenD) for LLM Training” as a computationally cheaper approximation to the original EigenScore used to quantify semantic variability across multiple generations for the same prompt (Mohammadzadeh et al., 2024). In that formulation, hallucination risk is linked to the spectrum of a regularized covariance matrix built from representations of multiple sampled outputs: higher spectral spread indicates greater semantic inconsistency and, in the paper’s interpretation, greater hallucination or confabulation tendency. EES preserves that spectral viewpoint but replaces explicit covariance construction and eigendecomposition with a Density of States (DOS) approximation based on Chebyshev polynomials, the Kernel Polynomial Method (KPM), and stochastic trace estimation. The result is presented both as an approximation to EigenScore and as a standalone metric with a different numerical scale.

1. Definition and conceptual basis

EES is defined in Section 3.4, “Efficient EigenScore Approximation,” and in the appendix section “Efficient EigenScore (EES) Derivation.” Its starting point is the paper’s use of the original EigenScore of Chen et al. (INSIDE), written as

$$ES = \mathbb{E}(Y \mid x, \theta) = \frac{1}{K} \sum_{i=1}^K \log(\lambda_i),$$

where $\lambda = \{ \lambda_1, \ldots, \lambda_K \}$ are the eigenvalues of the regularized covariance matrix $\Sigma + \alpha I$, computed from the representations of $K$ generated outputs for the same input $x$. The paper states that these are “typically 10 in our experiments.” The retained intuition is that if a model is more likely to hallucinate, then multiple high-temperature generations for the same prompt will be more semantically diverse, and that diversity will appear in the covariance spectrum of output embeddings (Mohammadzadeh et al., 2024).

The paper introduces EES because standard EigenScore requires explicit covariance construction and spectral decomposition, which it identifies as expensive for larger hidden layers, larger embedding matrices, and repeated inference during training-time monitoring. EES is therefore framed as a scalability device for the SenD protocol. At the same time, the paper explicitly states that EES is not merely a rescaled copy of EigenScore. It notes that “EigenScore ranges from $[0,\infty)$, whereas the approximation, referred to as Efficient EigenScore (EES), outputs values between $[-1,1]$,” and concludes that EES “can be seen as a standalone metric for hallucination detection.” This establishes a dual status: EES is derived as an approximation, but operationally interpreted on its own scale.

At the sample level, the metric is computed from multiple generated outputs for a single prompt. The broader paper uses sentence embeddings from the penultimate layer; the description in the data indicates that these embeddings are formed by averaging penultimate-layer token states and combining that with the last token state. The columns of the embedding matrix then correspond to distinct generations for the same input. Higher EES values are interpreted comparatively as indicating greater semantic variability among generations and thus greater hallucination risk, while lower values indicate more stable outputs.

2. Spectral derivation and DOS formulation

The derivation begins by denoting with $H$ the covariance matrix underlying EigenScore. The first key identity is the standard trace–log relation

$$\text{trace}(\log(H)) = \sum_{i=1}^{N} \log(\lambda_i),$$

where $\lambda_i$ are the eigenvalues of $\lambda = \{ \lambda_1, \ldots, \lambda_K \}$0. The appendix then rewrites the same quantity through the spectral density $\lambda = \{ \lambda_1, \ldots, \lambda_K \}$1: $\lambda = \{ \lambda_1, \ldots, \lambda_K \}$2 For a symmetric matrix $\lambda = \{ \lambda_1, \ldots, \lambda_K \}$3 with eigendecomposition

$\lambda = \{ \lambda_1, \ldots, \lambda_K \}$4

the spectral density is defined as

$\lambda = \{ \lambda_1, \ldots, \lambda_K \}$5

and for any analytic test function $\lambda = \{ \lambda_1, \ldots, \lambda_K \}$6,

$\lambda = \{ \lambda_1, \ldots, \lambda_K \}$7

Setting $\lambda = \{ \lambda_1, \ldots, \lambda_K \}$8 connects the DOS integral to the EigenScore objective (Mohammadzadeh et al., 2024).

To avoid eigendecomposition, the paper uses the Kernel Polynomial Method and first-kind Chebyshev polynomials with recurrence

$\lambda = \{ \lambda_1, \ldots, \lambda_K \}$9

The DOS is approximated as

$\Sigma + \alpha I$0

where

$\Sigma + \alpha I$1

and the moments are estimated stochastically by

$\Sigma + \alpha I$2

The weighted basis is given as

$\Sigma + \alpha I$3

A practical simplification appears when the derivation is rewritten on the embedding matrix $\Sigma + \alpha I$4. Instead of constructing the covariance matrix explicitly, the paper writes the coefficient in terms of $\Sigma + \alpha I$5 as

$\Sigma + \alpha I$6

The data notes that the summation index is likely a typo and should run over the $\Sigma + \alpha I$7 random vectors rather than to $\Sigma + \alpha I$8, but the intended meaning is that one works with $\Sigma + \alpha I$9 and avoids direct covariance construction. The matrix-Chebyshev recurrence is then

$K$0

$K$1

$K$2

The paper’s key computational remark is that each term can be computed with a matrix-vector multiplication.

The final approximation is obtained by truncating the DOS expansion and projecting $K$3 into the same Chebyshev basis. With

$K$4

and

$K$5

the final score used in Algorithm 1 is

$K$6

The lower integration bound is $K$7 because, after normalization, the relevant spectrum is taken to lie in $K$8.

3. Algorithm, data representation, and numerical ingredients

The practical input to EES is an embedding matrix

$K$9

where $x$0 is the hidden or embedding dimension and $x$1 is the number of sampled generations. Each column is one generation’s sentence embedding, and the metric is computed at the input or sample level: for a given prompt, generate $x$2 outputs at high temperature, embed each output, form $x$3, and compute one EES value (Mohammadzadeh et al., 2024).

Algorithm 1 begins by standardizing and scaling $x$4. The mean and standard deviation are computed as

$x$5

$x$6

followed by standardization

$x$7

The largest singular value is then estimated with the power method,

$x$8

and the matrix is rescaled: $x$9 This scaling is used because the KPM/Chebyshev approximation assumes spectrum supported in a bounded interval.

The remaining steps are straightforward in structure. The algorithm initializes DOS and log-coefficient arrays,

$[0,\infty)$0

computes the DOS coefficients $[0,\infty)$1 using random vectors $[0,\infty)$2, the matrix-Chebyshev recurrence, and stochastic trace estimation, computes $[0,\infty)$3 using

$[0,\infty)$4

with Gaussian Quadrature, and returns

$[0,\infty)$5

The paper identifies two EES-specific hyperparameters: $[0,\infty)$6, the number of Chebyshev moments, and $[0,\infty)$7, the number of stochastic trace estimation samples. Its appendix states that “selecting a moment value of under 50 would provide a balanced trade-off between accuracy and calculation time,” and the scaling figure uses a moments value of $[0,\infty)$8. By contrast, the exact $[0,\infty)$9 used in the main experiments is not clearly specified. The paper also leaves several implementation points underspecified or inconsistent: notation alternates between covariance matrix $[-1,1]$0, embedding matrix $[-1,1]$1, and $[-1,1]$2; the mapping from the original regularization $[-1,1]$3 to the approximate computation is not fully clarified; and the asserted $[-1,1]$4 output range is not fully proved in the text.

4. Function within Sensitivity Dropout (SenD)

EES is not an isolated diagnostic in the paper; it is integrated into the SenD training protocol as the scalable hallucination signal that makes repeated monitoring practical. The abstract states that this efficient metric is integrated into the protocol “allowing SenD to be both computationally scalable and effective at reducing hallucinations,” and Section 4 says that “SenD integrates the EES method discussed in Section \ref{sec:ees} to enhance computational efficiency while addressing variance in SEI behavior” (Mohammadzadeh et al., 2024).

Within Algorithm 2, EES appears as part of the stopping or continuation criterion: $[-1,1]$5 Here $[-1,1]$6 is an acceptable loss convergence range and $[-1,1]$7 is an acceptable confabulation or EES convergence range. In that role, EES functions as an online hallucination-monitoring signal during training. The broader SenD loop splits data into a training subset and a tracking subset, trains for several checkpoints, records penultimate-layer representations on the tracking subset, identifies Sensitive Embedding Indices (SEIs), drops those indices for subsequent checkpoints, and repeats until both loss and EES are sufficiently low.

The paper also makes clear that EES is not the mechanism used to choose SEIs. SEI selection is based on checkpoint-wise embedding variability,

$[-1,1]$8

with the top-$[-1,1]$9 indices selected by thresholding $H$0. EES instead provides the hallucination metric that monitors whether this intervention is moving the model toward lower semantic instability. This distinction matters conceptually: EES measures the effect of the training protocol on confabulation tendency, but SEI selection itself is based on embedding variability across checkpoints.

5. Efficiency claims, empirical behavior, and interpretation

The paper’s principal efficiency claim is approximately a twofold speedup relative to standard EigenScore. The abstract states that EES “approximates the traditional EigenScore in 2x speed,” the contributions section says it achieves “up to 2x speedup with minimal effects on accuracy,” and the appendix reports two concrete timing comparisons (Mohammadzadeh et al., 2024). In a Pythia 1B finetuning trajectory on HELM, the paper states that “the trajectories, magnitude and shape of the graphs are nearly identical while EES takes only 4 minutes to calculate and EigenScore takes approximately 8.” In the matrix-scaling figure, with moments $H$1, it reports that at matrix size $H$2, EES takes around $H$3 seconds whereas EigenScore takes around $H$4 seconds. The paper does not provide hardware details for these timing results.

The empirical validation of EES itself is primarily indirect but coherent. The strongest direct evidence is the close correlation with EigenScore during Pythia 1B finetuning on HELM: the paper states that EES and regular EigenScore have nearly identical trajectories and closely aligned magnitude and trend, despite differing scale. It also uses EES throughout the SenD experiments on Pythia 1B, Llama 3.2 1B, and Llama 3.1 8B across HELM, MedHALT, and LegalBench. The qualitative pattern reported is that SenD generally yields lower EES during training, smoother or lower-variance training trajectories, and usually lower final EES than standard training.

To relate EES to external factuality criteria, the paper evaluates final models with FactScore and HaluEval Summarization. For Pythia 1B, the reported final-model results are:

• HaluEval Summarization: Accuracy $H$5 vs $H$6, Correctness $H$7 vs $H$8, and Exact Match $H$9 vs $$\text{trace}(\log(H)) = \sum_{i=1}^{N} \log(\lambda_i),$$0, comparing SenD to normal training.
• FactScore: $$\text{trace}(\log(H)) = \sum_{i=1}^{N} \log(\lambda_i),$$1 vs $$\text{trace}(\log(H)) = \sum_{i=1}^{N} \log(\lambda_i),$$2 at $$\text{trace}(\log(H)) = \sum_{i=1}^{N} \log(\lambda_i),$$3 points and $$\text{trace}(\log(H)) = \sum_{i=1}^{N} \log(\lambda_i),$$4 vs $$\text{trace}(\log(H)) = \sum_{i=1}^{N} \log(\lambda_i),$$5 at $$\text{trace}(\log(H)) = \sum_{i=1}^{N} \log(\lambda_i),$$6 points.

The paper interprets these results as showing correlation between low EES and improved factuality. It also states “up to 40%” improvement in reliability or factuality, specifically in connection with FactScore changes such as $$\text{trace}(\log(H)) = \sum_{i=1}^{N} \log(\lambda_i),$$7 to $$\text{trace}(\log(H)) = \sum_{i=1}^{N} \log(\lambda_i),$$8 or $$\text{trace}(\log(H)) = \sum_{i=1}^{N} \log(\lambda_i),$$9 to $\lambda_i$0. Because EES operates on a scale different from standard EigenScore, its values are meant to be read comparatively rather than through thresholds transferred from the original metric.

6. Scope, limitations, and terminological disambiguation

The paper identifies several limitations or gaps in the current EES presentation. First, EES is on a different scale from EigenScore and is therefore not numerically interchangeable with it. Second, approximation quality depends on the number of moments: more moments increase fidelity but also runtime. Third, EES is still not a single-pass probe; it requires multiple generations and an embedding matrix over those outputs. Fourth, several implementation details remain underspecified, including the exact $\lambda_i$1 used in experiments, the precise relationship between covariance, Gram matrix, and normalization, and the regularization mapping from $\lambda_i$2. Finally, the paper emphasizes close correlation and trend preservation rather than exact equality with EigenScore (Mohammadzadeh et al., 2024).

The name also requires careful disambiguation. In the LLM decoding paper “Auxiliary-Hyperparameter-Free Sampling: Entropy Equilibrium for Text Generation,” EES refers to Entropy Equilibrium Sampling, a truncation-based generation method unrelated to spectral hallucination detection (Cai et al., 30 Nov 2025). Likewise, “EigenScore: OOD Detection using Covariance in Diffusion Models” introduces an OOD detector based on leading eigenvalues of posterior covariance in diffusion models; despite the shared emphasis on spectral structure and covariance, that method concerns unsupervised distribution-shift detection rather than training-time hallucination monitoring in autoregressive LLMs (Shoushtari et al., 8 Oct 2025). Within the SenD paper, “Efficient EigenScore” therefore denotes a specific approximation to an embedding-covariance hallucination metric, not a generic label for all eigenvalue-based scores.

A plausible implication is that EES belongs to a broader class of spectral diagnostics that summarize semantic instability through low-order matrix functionals rather than supervised labels. Even so, the concrete scope established in the source is narrower: EES is the SenD paper’s scalable approximation to EigenScore, derived through DOS and Chebyshev expansions, and used as an unsupervised hallucination signal during training.