Does Bielik Know What It Doesn't Know? Activation Dispersion Separates Entity Familiarity from Factual Reliability Across Model Scale

Abstract: LLMs hallucinate most about entities they have never seen. We ask whether a model's activations betray entity familiarity before a single answer token is generated, and whether that signal predicts the factual reliability of the answers. On four Polish Bielik models (1.5B-11B parameters), we probe four entity domains (athletes, cities, writers, musicians), each with 42 well-known, 42 obscure-but-real, and 42 fabricated entities addressed by a one-sentence question (504 prompts per model). Two unsupervised, single-forward-pass dispersion measures over post-SwiGLU MLP activations, inverse participation ratio and spectral entropy, separate known from fabricated entities at AUROC 0.95-1.00 across all domains and scales; a supervised linear probe reaches 0.99-1.00. Both clear selection-aware permutation floors of about 0.70-0.74 (empirical p<=1e-3), survive held-out layer selection (0.93-0.99), and persist on real names (known vs. obscure-but-real: 0.96-1.00). The signal transfers across entity types (mean off-diagonal AUROC 0.92-0.99); a matched-template counterfactual shows the only large drops are template-caused, not entity-type effects, and the signal is diffuse across heads. This representational signal is already at ceiling at 1.5B, whereas behavioral factual reliability scales sharply: 0, 2, 10, and 19 of 42 known athletes are answered fully correctly by the 1.5B, 4.5B, 7B, and 11B models under a strict judge. Within known entities, separating correct from hallucinated answers is much harder (probe 0.93; dispersion no better than a first-token-entropy baseline). A five-sample semantic-entropy baseline reaches only 0.71-0.83 at 5x the inference cost. Despite this internal awareness, the models almost never abstain: an audit of 2,520 answers finds 2 refusals and 1 hedge. Entity familiarity and factual reliability are distinct phenomena on different scaling curves.

• The paper shows that two simple, single-pass activation-dispersion measures distinguish known from fabricated entities with AUROC scores of 0.946–0.984 across Bielik models, while supervised probes reach 0.990–1.000.
• The paper identifies separate scaling curves: familiarity signals are near-ceiling even at 1.5B parameters, but strictly correct answers about known athletes increase from 0/42 at 1.5B to 19/42 at 11B.
• The paper finds that activation signals do not reliably distinguish factual answers from hallucinations and that Bielik models almost never abstain, suggesting familiarity gates could route uncertain cases to retrieval or safer response policies.

Overview

This paper asks whether a LLM's activations reveal entity familiarity before any answer token is generated, and whether that signal bears on factual reliability. The study examines four instruction-tuned Bielik v3.0 checkpoints (1.5B, 4.5B, Minitron-7B, and 11B) on a three-condition dataset spanning four Polish entity domains — athletes, cities, writers, and musicians — each containing 42 famous ("known"), 42 real-but-obscure ("unknown-real"), and 42 fabricated but morphologically plausible entities. The headline detector is deliberately minimalist: two closed-form dispersion statistics over post-SwiGLU MLP activations at a single token position, computed from one forward pass with no labels and no auxiliary model. The central finding is a dissociation: a near-ceiling representational familiarity signal present at every scale, alongside behavioral factual reliability that scales steeply from essentially zero to moderate levels.

Method

For each prompt the authors record post-SwiGLU MLP activations via forward hooks, residual-stream hidden states, attention maps, and logit-lens distributions. Signals are read at two positions: the last entity token and the last prompt token; all headline numbers use the conservative prompt point, where lexical identity of the input span no longer dominates. From the activation vector $a$, normalized weights $p_i = a_i^2 / \sum_j a_j^2$ yield the inverse participation ratio $\mathrm{IPR} = \sum_i p_i^2$ and the Shannon entropy $S = -\sum_i p_i \log p_i$. Activations are winsorized above the $q=0.99$ quantile to control for massive-activation outliers (Sun et al., 2024). A supervised $\ell_2$-regularized logistic probe on raw hidden states provides the linearly decodable ceiling.

The evaluation methodology is unusually careful about selection effects. Because every headline number maximizes over layers and over the metric pair, the authors calibrate optimism with three tools: a selection-aware permutation null (10,000 permutations for dispersion, re-selecting the best layer per permutation), bootstrap confidence intervals, and held-out layer selection over 200 stratified half-splits. Selection-aware floors land at approximately 0.70–0.74, well above the naive 0.5 baseline. Behavioral labels (athletes only) come from an LLM judge under strict and soft rubrics; semantic entropy over five samples with LLM clustering serves as the multi-sample baseline (Kuhn et al., 2023, Jr, 2024); and an answer-level LLM audit classifies refusals across all 2,520 sampled athlete answers.

Dispersion separates known from fabricated entities

On the athletes domain, the better unsupervised dispersion metric separates known from fabricated entities with AUROC between 0.946 (1.5B) and 0.984 (4.5B); the supervised probe reaches 0.990–1.000 everywhere. These values clear the selection-aware permutation floors ($p \le 10^{-4}$ for dispersion) and survive held-out layer selection at 0.928–0.970 out-of-sample. The result holds across all four domains: best dispersion reaches 0.95–1.00 in cities, writers, and musicians as well, with within-domain separability highest on cities (probe 1.000).

Control contrasts establish that the signal tracks familiarity rather than fabricated-string artifacts. Known versus unknown-real — both real names with natural subword statistics — separates at 0.971–1.000, as strong as or stronger than the headline contrast. Conversely, unknown-real versus fabricated, where neither condition should be familiar, still separates at 0.699–0.824, indicating a residual lexical/plausibility component. The authors treat this as an upper bound on the non-epistemic contribution: too large for the headline to be read as purely epistemic, far too small to account for 0.94–0.98, and cleanly excluded by the known-vs-unknown-real control. Token count alone reaches only 0.681/0.612 AUROC on athletes under the two tokenizers.

Cross-domain probe transfer preserves most separability, with mean off-diagonal AUROC of 0.987/0.959/0.919/0.931 across the scale sweep. All large transfer drops involve cities, the only domain using a different question stem ("Czym jest" rather than "Kim jest"). A matched-template counterfactual — re-extracting cities and writers under a shared neutral stem — shows the target-side drop is template-caused: writers-to-cities transfer recovers to 0.999–1.000 at every scale. A residual cities-as-source asymmetry (probe 0.82–0.98 outward transfer) remains unexplained, plausibly reflecting a tighter places manifold. Per-head analysis shows the attention-entropy signal is diffuse: only 1–2 heads appear in all four domains' top-20, with cross-domain Spearman correlation rising modestly from 0.27 to 0.56 with scale. The global statistics therefore read a broadly distributed correlate of familiarity, not a localized circuit.

Two dissociated scaling curves

The paper's central empirical claim is that entity familiarity and factual reliability move on different scaling curves measured on the same dataset and same entities. The representational familiarity signal is at or near ceiling already at 1.5B (best-of-pair dispersion 0.94–0.98 at every scale; the individual metrics flip between entropy and IPR across scales, so only the pair is scale-stable). Behavioral reliability rises sharply: fully correct answers about known athletes go from 0/42 (1.5B) to 2/42, 10/42, and 19/42 (11B) under a strict judge, and 6→16→24→33 of 42 under a soft key-facts rubric. This ordering matches frequency-based accounts of factual recall (Mallen et al., 2022, Kandpal et al., 2022): entity recognition arrives long before reliable facts about the entity. The Minitron-7B point fits the behavioral curve cleanly between 4.5B and 11B, indicating pruning and distillation preserved both the familiarity geometry and scale-appropriate reliability.

A depth analysis shows the probe saturates from roughly 20% relative depth onward in every model, while dispersion occupies a common mid-depth band ([0.48, 0.77] for spectral entropy). Among extended metrics, attention entropy is robust across scale (0.839–0.991), logit-lens entropy strengthens with scale (0.722 → ≈0.98), and effective rank is unstable (0.673–0.942) — global measures are not interchangeable.

The behavioral axis resists detection

Within known athletes, separating correct answers from hallucinations is much harder than separating familiar from unfamiliar entities. The full model × threshold grid is inconsistent: dispersion sometimes nominally beats the probe, the probe falls below its own permutation floor at 1.5B, and the strongest adequately powered cell (11B probe 0.929 against a floor of 0.744) is suggestive rather than established given $n=42$. Dispersion at 11B is indistinguishable from the first-token-entropy baseline, with its best layer embedding-adjacent — a noise flag. The honest conclusion is that no tested metric reads factual reliability the way dispersion reads familiarity; the negative result is framed as the paper's most useful distinction.

Semantic entropy behaves complementarily rather than competitively. On the familiarity contrast it reaches only 0.71–0.83 with non-overlapping confidence intervals versus dispersion, at five times the inference cost. On its home turf — the judge-labeled correctness contrast — it wins, reaching up to 0.87 where dispersion fails. Mean semantic entropy on known entities falls monotonically with scale (0.91 → 0.26), showing larger models become self-consistent about familiar entities even when the consistent answer contains fabricated details.

The abstention gap

Despite carrying a near-perfectly decodable internal familiarity signal at every scale, the Bielik models almost never decline to answer. An LLM audit of all 2,520 sampled athlete answers finds exactly 2 refusals and 1 hedged answer (99.88% direct assertions), both refusals from the largest model. A Gemma-4 control bounds the concern that the one-sentence instruction suppresses abstention: Gemma-4-12B refuses condition-selectively under identical prompts (all-five-sample refusals on 24/42 fabricated and 28/42 unknown-real athletes, versus 7/42 known), while also replicating the familiarity signal and dissociation qualitatively — weaker, still rising with scale, and weakest precisely in the domains whose famous tier is most Polish-local. The representational precondition for calibrated abstention thus exists at every Bielik scale; the behavior barely exists at any. The authors attribute this to tuning and alignment rather than capability.

Limitations

Several limitations bear directly on interpretation. The lexical confound is quantified but not eliminated: approximate length matching leaves a measurable residual component (unknown-real vs. fabricated at 0.70–0.82), and the winsorization quantile $q=0.99$ is the most favorable of four settings swept, on the smallest model only — specific layer-level values shift substantially with $q$, though the familiarity signal persists at every setting. Sample sizes are small ($p_i = a_i^2 / \sum_j a_j^2$0 per detection contrast, $p_i = a_i^2 / \sum_j a_j^2$1 per behavioral cell), so single-model AUROC differences below roughly 0.05 should not be interpreted. Behavioral, semantic-entropy, and refusal claims rest on the athletes domain alone. Judge circularity is acknowledged: one Claude model family occupies every rung of the evaluation ladder, so the behavioral scaling curves have not been confirmed by a second judge family, and the strict judge conflates confabulation with truncation and parse defaults. Answer sampling was unseeded, making behavioral counts properties of frozen artifacts rather than fresh runs. Finally, the main study covers one Polish family plus two Gemma-4 checkpoints on one language; a full multi-family, multi-language treatment remains open.

Conclusion

The paper establishes, with careful uncertainty quantification, that closed-form single-pass activation dispersion reliably detects entity familiarity across four domains and four model scales in a Polish model family, that this signal is distinct from factual reliability — which scales steeply and resists all tested internal detectors — and that the models almost never act on their own representational knowledge of ignorance. The practical implication is a free prefill-time familiarity gate for routing toward retrieval or abstention, with the explicit caveat that it flags unfamiliar entities rather than hallucinated content about familiar ones.