EigenTrack: Spectral Monitoring for LLMs & VLMs

• EigenTrack is a spectral-temporal reliability monitor that tracks the geometry of hidden activations to detect hallucinations and out-of-distribution errors.
• It computes covariance-spectrum descriptors (e.g., spectral entropy, eigenvalue gaps, KL divergence) and processes them with a lightweight recurrent classifier for per-step risk scoring.
• The method emphasizes early detection and interpretability by linking deviations from the Marchenko–Pastur baseline to shifts in internal representations, enabling timely intervention.

Searching arXiv for the cited EigenTrack paper and related spectral/RMT source. EigenTrack is a spectral-temporal reliability monitor for LLMs and vision-LLMs (VLMs) that detects hallucination and out-of-distribution (OOD) behavior by tracking the geometry of hidden activations rather than output probabilities. It operates during ordinary generation, converts streaming internal states into covariance-spectrum descriptors, and feeds those descriptors to a lightweight recurrent classifier that emits a per-step risk score for hallucination or OOD drift. The method is presented as an interpretable, single-forward-pass detector that preserves temporal context and targets early warning, including before erroneous tokens or multimodal outputs appear (Ettori et al., 19 Sep 2025).

1. Problem setting and motivation

EigenTrack is defined around two failure modes. In this setting, hallucination is the production of fluent but factually incorrect or ungrounded content relative to the prompt or provided context; for VLMs, this includes invented visual details or misinterpretations. OOD error denotes behavior induced by inputs outside the model’s training or distributional support, causing internal representations and outputs to deviate unpredictably. The stated value of early detection is that it enables verification, retrieval, intervention, or user clarification before failure surfaces, while avoiding wasted compute on long generations (Ettori et al., 19 Sep 2025).

The method is motivated by limitations of three detector families. Black-box methods such as SelfCheckGPT and prompt-resampling or ensemble approaches require additional forward passes and remain snapshot-based. Grey-box methods such as DetectGPT-style output-layer heuristics depend on logit or curvature statistics that degrade under domain shift and do not use internal-state geometry. White-box methods that inspect hidden activations often remain static or per-token, thereby discarding temporal context and global structure. EigenTrack is positioned as a response to these limitations by combining internal activations, spectral compression, and recurrent temporal modeling within a single generation pass (Ettori et al., 19 Sep 2025).

A central premise is that hidden activations encode global covariance geometry and that this geometry changes systematically under uncertainty or drift. This premise is elaborated in the associated thesis, which frames EigenTrack within a broader spectral-geometry and Random Matrix Theory (RMT) perspective: in-distribution factual reasoning tends to produce low-rank structure with dominant eigen-directions, whereas hallucination and OOD drift can move the spectrum toward an isotropic, noise-like regime (Ettori, 25 Feb 2026). This suggests that reliability monitoring can be recast as online tracking of activation-spectrum dynamics.

2. Spectral geometry and random-matrix foundations

Let $h_t \in \mathbb{R}^d$ denote the hidden activation at time step $t$, or, for multi-layer monitoring, let selected layers be concatenated into

$$v_t = [h_t^{\ell_1} \parallel \dots \parallel h_t^{\ell_m}] \in \mathbb{R}^{md}.$$

EigenTrack maintains either a sliding window of recent activations or cumulative streaming statistics. For a cumulative stream, the running mean and covariance are updated using Welford-style recursions,

$\mu_t = \mu_{t-1} + \frac{h_t - \mu_{t-1}}{t},$

$S_t = S_{t-1} + (h_t - \mu_{t-1})(h_t - \mu_t)^T,$

$C_t = \frac{S_t}{t-1}.$

For a sliding window, one may instead use a rolling buffer and compute

$C_t = \frac{1}{N} H_t^T H_t.$

The top-$k$ eigenvalues $$\lambda_{t,1} \ge \lambda_{t,2} \ge \dots \ge \lambda_{t,k}$$ are then computed via truncated SVD or power/Lanczos iterations, and normalized as

$$p_{t,i} = \frac{\lambda_{t,i}}{\sum_{j=1}^k \lambda_{t,j}}.$$

These quantities define the compact spectral state on which the detector operates (Ettori et al., 19 Sep 2025).

The thesis formulation expresses the windowed activation matrix as $t$0 and the sample covariance as

$t$1

with aspect ratio $t$2. The empirical spectral distribution is compared with the Marchenko–Pastur (MP) law, whose bulk support for i.i.d. noise of variance $t$3 is

$t$4

Within this RMT framing, deviations from the MP bulk quantify structure, while the Baik–Ben Arous–Péché phase transition identifies sufficiently strong spikes as outlier eigenvalues above the bulk edge. In the thesis, those outliers are interpreted as structured, task-related directions, whereas the bulk is associated with noise-like variability (Ettori, 25 Feb 2026).

This formalism gives EigenTrack an explicit null model. The paper states that for near-isotropic LayerNorm’d activations, divergence from MP provides a principled baseline. A plausible implication is that the method’s interpretability partly derives from this baseline: the detector does not merely learn an opaque score, but organizes deviations in terms of entropy, gaps, concentration, and distance from a random-matrix reference spectrum.

3. Spectral descriptors and temporal risk modeling

EigenTrack computes a feature vector from the current covariance spectrum. The core descriptors include spectral entropy

$t$5

which is high when variance is dispersed across many modes and low when variance is concentrated in a few dominant modes; eigenvalue gaps, including the leading gap

$t$6

and the multi-gap vector $t$7; and KL divergence from a baseline spectrum $t$8,

$t$9

Baseline choices include a uniform spectrum, an in-domain empirical spectrum, and an MP baseline. Additional features include cumulative KL, spectral slope, condition number $$v_t = [h_t^{\ell_1} \parallel \dots \parallel h_t^{\ell_m}] \in \mathbb{R}^{md}.$$0, central-tendency statistics, cumulative variance explained, and, in the thesis formulation, distributional deviations such as the Kolmogorov–Smirnov discrepancy, Wasserstein-1 distance, spike count $$v_t = [h_t^{\ell_1} \parallel \dots \parallel h_t^{\ell_m}] \in \mathbb{R}^{md}.$$1, and excess tail mass beyond the MP upper edge (Ettori et al., 19 Sep 2025).

At each generation step, these descriptors are assembled into a stream

$$v_t = [h_t^{\ell_1} \parallel \dots \parallel h_t^{\ell_m}] \in \mathbb{R}^{md}.$$2

A lightweight recurrent head then updates its hidden state and outputs a per-step risk score,

$$v_t = [h_t^{\ell_1} \parallel \dots \parallel h_t^{\ell_m}] \in \mathbb{R}^{md}.$$3

The recurrent head may be an RNN, GRU, or LSTM, with the main paper reporting that GRU heads are strongest empirically. Training uses labels marking the onset of hallucination or OOD within sequences and optimizes binary cross-entropy; class imbalance may be handled using focal loss, class weights, or balanced sampling, while calibration uses temperature scaling or Platt scaling to make scores thresholdable across models (Ettori et al., 19 Sep 2025).

The paper emphasizes two salient instability regimes. In a noise-like dispersion regime, entropy rises, KL-to-MP falls, and eigengaps shrink, producing a flatter and more isotropic spectrum. In a spiked-covariance deviation regime, top eigenvalues detach, gaps widen, and entropy can drop. The feature set is designed to capture both regimes because hallucination and domain shift do not manifest identically across tasks. The thesis further reports that hallucinated sequences characteristically exhibit higher entropy, lower KL divergence from MP, and narrower eigengaps, which is consistent with drift toward an MP-like bulk (Ettori, 25 Feb 2026).

4. Online algorithm, complexity, and integration in LLMs and VLMs

EigenTrack is explicitly described as a single-forward-pass monitor. During normal generation, it hooks into intermediate layers, captures activations, updates the streaming window, computes a top-$$v_t = [h_t^{\ell_1} \parallel \dots \parallel h_t^{\ell_m}] \in \mathbb{R}^{md}.$$4 spectrum, derives spectral features, updates the recurrent head, and emits a risk score. The algorithmic summary in the paper is: build a sliding-window $$v_t = [h_t^{\ell_1} \parallel \dots \parallel h_t^{\ell_m}] \in \mathbb{R}^{md}.$$5 over concatenated layer activations $$v_t = [h_t^{\ell_1} \parallel \dots \parallel h_t^{\ell_m}] \in \mathbb{R}^{md}.$$6; extract top-$$v_t = [h_t^{\ell_1} \parallel \dots \parallel h_t^{\ell_m}] \in \mathbb{R}^{md}.$$7 eigenvalues via truncated SVD or power/Lanczos; compute normalized spectrum, entropy, gaps, $$v_t = [h_t^{\ell_1} \parallel \dots \parallel h_t^{\ell_m}] \in \mathbb{R}^{md}.$$8, KL-to-baseline, and auxiliary features; feed the result into a lightweight recurrent head, preferably a GRU; and trigger warnings when $$v_t = [h_t^{\ell_1} \parallel \dots \parallel h_t^{\ell_m}] \in \mathbb{R}^{md}.$$9 and its temporal pattern indicate rising uncertainty or drift (Ettori et al., 19 Sep 2025).

Per-step cost is dominated by spectral extraction. The paper gives approximate complexity for power/Lanczos as $\mu_t = \mu_{t-1} + \frac{h_t - \mu_{t-1}}{t},$0 matrix-vector multiplies per iteration, with few iterations, and for truncated or randomized SVD on an $\mu_t = \mu_{t-1} + \frac{h_t - \mu_{t-1}}{t},$1 window as approximately $\mu_t = \mu_{t-1} + \frac{h_t - \mu_{t-1}}{t},$2. Storage is either an $\mu_t = \mu_{t-1} + \frac{h_t - \mu_{t-1}}{t},$3 activation buffer for sliding-window computation or $\mu_t = \mu_{t-1} + \frac{h_t - \mu_{t-1}}{t},$4 for online mean/covariance plus $\mu_t = \mu_{t-1} + \frac{h_t - \mu_{t-1}}{t},$5 for subspace tracking. Efficiency techniques listed in the paper include Oja’s rule for subspace tracking, randomized SVD with small oversampling, step downsampling, monitoring only mid-layer blocks, feature-frequency throttling, CountSketch, and diagonal covariance approximations when needed (Ettori et al., 19 Sep 2025).

The method exposes explicit accuracy–latency trade-offs. Larger windows stabilize spectra but increase latency; the paper states that $\mu_t = \mu_{t-1} + \frac{h_t - \mu_{t-1}}{t},$6–$\mu_t = \mu_{t-1} + \frac{h_t - \mu_{t-1}}{t},$7 is a strong operating point across LLMs, while the thesis describes a knee around $\mu_t = \mu_{t-1} + \frac{h_t - \mu_{t-1}}{t},$8–$\mu_t = \mu_{t-1} + \frac{h_t - \mu_{t-1}}{t},$9 tokens and notes that experiments often use $S_t = S_{t-1} + (h_t - \mu_{t-1})(h_t - \mu_t)^T,$0 (Ettori, 25 Feb 2026). Likewise, $S_t = S_{t-1} + (h_t - \mu_{t-1})(h_t - \mu_t)^T,$1–$S_t = S_{t-1} + (h_t - \mu_{t-1})(h_t - \mu_t)^T,$2 captures most variance while keeping cost low, and layer selection matters: early or mid transformer layers often display clearer distributional shifts, whereas late layers more directly track semantic certainty but can be noisier (Ettori et al., 19 Sep 2025).

For LLMs, EigenTrack monitors a small set of early transformer feed-forward blocks plus one mid-layer and forms token-level streaming features, with optional session-level aggregation over sentences or turns. For VLMs, visual patches or frame tokens are treated analogously; the paper recommends monitoring cross-attention or fusion layers to capture text-vision interaction drift, and describes multimodal covariance formation either by stacking text and vision activations or by using block-diagonal structure plus cross-covariance terms. Practical deployment guidance includes thresholds $S_t = S_{t-1} + (h_t - \mu_{t-1})(h_t - \mu_t)^T,$3–$S_t = S_{t-1} + (h_t - \mu_{t-1})(h_t - \mu_t)^T,$4, alerts triggered after $S_t = S_{t-1} + (h_t - \mu_{t-1})(h_t - \mu_t)^T,$5 consecutive high-risk steps with examples $S_t = S_{t-1} + (h_t - \mu_{t-1})(h_t - \mu_t)^T,$6–$S_t = S_{t-1} + (h_t - \mu_{t-1})(h_t - \mu_t)^T,$7, and resetting the recurrent state and window at topic or turn boundaries to mitigate transient false positives (Ettori et al., 19 Sep 2025).

5. Empirical evaluation and reported results

The reported evaluation covers both LLMs and VLMs. LLM backbones include LLaMa in 1B, 3B, and 7B sizes, Qwen in 1.8B and 7B, and Mistral 7B; the VLM evaluated is LLaVa 7B. Hallucination experiments use HaluEval, described as HotpotQA-based, with an automated pipeline involving a main model, a question generator, and an answer judge to label factual versus hallucinated outputs in QA-style generations up to 128 tokens. OOD experiments contrast WebQuestions as in-distribution with Eurlex as OOD, and for VLMs pair questions with Flickr8k images. Metrics include AUROC, F1, AUPRC internally, detection latency, false positive rate, and calibration via ECE (Ettori et al., 19 Sep 2025).

The main paper reports hallucination-detection AUROC in the range 0.82–0.94 across models, with GRU heads strongest. On LLaMa-7B, EigenTrack achieves 0.89 AUROC, compared with HaloScope 0.86, LapEigvals 0.87, INSIDE 0.81, and SelfCheckGPT 0.84 on a larger LLaMa-30B. For OOD detection, reported AUROC is 0.85–0.96, and on LLaMa-7B EigenTrack reaches 0.92 AUROC, exceeding cosine distance, energy score, and ODIN baselines in aggregate comparisons. The paper also reports that compact feature subsets—particularly KL-to-MP, leading eigenvalues, intermediate gaps, and entropy—account for most discriminative power, and that removing them degrades AUROC by approximately 0.1 (Ettori et al., 19 Sep 2025).

The thesis provides a more granular LLaMa hallucination table at sliding window length 30 tokens. It reports AUROC values of 0.842 for LLaMa 1B, 0.861 for LLaMa 3B, and 0.894 for LLaMa 7B, against LapEigvals, INSIDE, SelfCheckGPT, and HaloScope baselines. It also notes that AUROC rises sharply within the first few generated tokens and then saturates, supporting the claim that risk rises early enough to enable early stopping or intervention. Temporal plots are described as showing feature divergence approximately 10–20 steps before wrong answers are emitted in case studies (Ettori, 25 Feb 2026).

Setting	Reported result
Hallucination AUROC across models	0.82–0.94
LLaMa-7B hallucination AUROC	0.89
OOD AUROC across models	0.85–0.96
LLaMa-7B OOD AUROC	0.92
Strong operating window length	25–50 tokens
Default-style deployment ranges	$S_t = S_{t-1} + (h_t - \mu_{t-1})(h_t - \mu_t)^T,$8–$S_t = S_{t-1} + (h_t - \mu_{t-1})(h_t - \mu_t)^T,$9, $C_t = \frac{S_t}{t-1}.$0–$C_t = \frac{S_t}{t-1}.$1

The interpretability analyses in both sources are tied to feature importance and spectral trajectory visualization. The paper states that SHAP analyses show GRUs emphasizing KL divergence, eigenvalue gaps, and entropy, whereas LSTMs prioritize top-$C_t = \frac{S_t}{t-1}.$2 eigenvalues plus entropy. This suggests that the recurrent architecture changes which aspects of spectral evolution are most salient, even though the underlying descriptor family is shared (Ettori et al., 19 Sep 2025).

6. Interpretation, deployment role, and limitations

EigenTrack’s interpretability claim rests on mapping spectral descriptors to representation dynamics. When entropy increases, gaps shrink, and KL-to-MP decreases, the activation spectrum approaches an isotropic bulk associated with weaker structure and rising uncertainty. When leading eigenvalues separate and gaps widen, the spectrum exhibits dominant modes indicative of stronger structure or, depending on context, aberrant spiked deviations under domain shift. The thesis makes this mapping explicit using the MP bulk and BBP outlier framework: outlier eigenvalues and large eigengaps correlate with structured, task-relevant internal representations, while loss of spikes and convergence toward the bulk correlate with noise-dominated variability (Ettori, 25 Feb 2026).

The paper describes several concrete deployment actions when the risk score exceeds threshold: defer to retrieval or verification, ask clarifying questions, narrow scope, slow generation, switch to a more cautious decoding policy, log the event for offline analysis, or shorten outputs until risk subsides. To keep user experience smooth, the recommendation is to act only on persistent signals, such as $C_t = \frac{S_t}{t-1}.$3 for $C_t = \frac{S_t}{t-1}.$4 consecutive steps. Because the method is architecture-agnostic and does not modify the base model, its operational role is that of an attached monitor rather than a decoder replacement (Ettori et al., 19 Sep 2025).

Several limitations are explicitly noted. Sensitivity depends on layer choice, window length, descriptor selection, and calibration. Topic shifts and conversational turn transitions can create transient false positives, motivating reset logic and change-point smoothing. Spectral extraction adds compute, though approximations and sparse layer sampling can mitigate it. The thesis further notes that activation scaling and normalization affect $C_t = \frac{S_t}{t-1}.$5 estimation and the aspect ratio $C_t = \frac{S_t}{t-1}.$6, and that strong temporal nonstationarity, heavy tails, or violations of MP assumptions can make the MP baseline less appropriate; in such cases, heavy-tailed MP variants or empirical baselines may be required (Ettori, 25 Feb 2026).

Within the broader spectral program developed in the thesis, EigenTrack shares its conceptual basis with RMT-KD: both use MP/BBP structure to separate signal from noise, but EigenTrack applies that distinction to reliability monitoring rather than compression. A plausible implication is that the method’s significance lies less in any single descriptor than in the operationalization of activation-spectrum dynamics as an online control signal for generation-time safety and distribution-shift awareness.