Surprise Score: Quantifying Unexpectedness

• Surprise score is a quantitative measure that computes expectation violation relative to a reference model through metrics like Shannon surprisal and KL divergence.
• It is applied across domains such as finance, language modeling, reinforcement learning, and network detection to flag anomalies and drive predictions.
• The score serves as both a prediction target and an adaptive signal, effectively guiding machine learning models and quantitative hypothesis testing.

Searching arXiv for the cited papers to ground the article in current literature. The expression surprise score denotes a family of quantitative constructs that measure expectation violation relative to a specified model, prior, context, or null hypothesis. In recent literature, the term is used for standardized unexpected earnings in corporate finance, Shannon surprisal and Bayesian surprise in dynamic prediction, deviation-based intrinsic signals in reinforcement learning, percentile-normalized similarity in embedding spaces, surprisal variance in language generation, and hypergeometric or KL-based quality functions for network structure discovery. Across these settings, the score serves as a prediction target, an exploration or adaptation signal, an anomaly score, a diversity measure, or a statistical test of structure (Shu et al., 4 Oct 2025, Dinparastdjadid et al., 2023, Bachlechner et al., 2023, Traag et al., 2015).

1. Conceptual foundations

In cognitive science and neuroscience, surprise is widely conceived as the subjective feeling that arises when an event violates an agent’s expectations. Early psychological models describe surprise as a two-phase emotion consisting of detection of expectation violation followed by cognitive appraisal and orienting response. The literature further identifies two necessary ingredients: uncertainty, because beliefs about the world are probabilistic, and subjectivity, because surprise depends on the observer’s own generative model (Dinparastdjadid et al., 2023).

These ideas motivate several formalizations. A first class is probabilistic mismatch, or Shannon surprise, written as $S(x;P)=-\log P(x)$. A second class is belief mismatch, or Bayesian surprise, written as a KL divergence between posterior and prior beliefs. A third class is observation mismatch, often represented as unsigned or signed prediction error. The same taxonomy appears in traffic-scene forecasting, where Residual Information, Bayesian Surprise, and Antithesis are treated as distinct operationalizations of “surprising” events (Dinparastdjadid et al., 2023).

Other literatures introduce related but non-equivalent constructions. In volatile-environment learning, Bayes Factor Surprise is the ratio of predictive probabilities under a reset prior versus the current belief, $$S_{BF}(y_{t+1};\pi^{(t)})=P(y_{t+1};\pi^{(0)})/P(y_{t+1};\pi^{(t)})$$, and it directly modulates the adaptation rate in change-point inference (Liakoni et al., 2019). In belief revision, relative surprise is defined over propositional worlds by

$s^\mu(v,w)=d(v,w)-d(v,\mu),$

with $d$ the Hamming distance and $d(v,\mu)$ the minimum distance from world $v$ to the new information $\mu$. This makes surprise explicitly context-sensitive: it is measured not only against prior expectations but also against the admissible worlds introduced by the new information (Haret, 2021).

2. Principal mathematical forms

A large fraction of surprise scores can be grouped by the mathematical primitive they use. One widely used primitive is the negative log-probability. In temporal anomaly detection, a stationary low-rank model yields an observed log-likelihood $LL(B_t,\hat\pi)$ for each access matrix, a regressor predicts the expected log-likelihood $\hat w^\top v_t$, and the final surprise score is the absolute deviation

$$\mathrm{Surprise}(t)=|LL(B_t,\hat\pi)-\hat w^\top v_t|.$$

In language modeling, token surprisal is defined as

$$S_{BF}(y_{t+1};\pi^{(t)})=P(y_{t+1};\pi^{(0)})/P(y_{t+1};\pi^{(t)})$$0

and caption-set diversity is then quantified by the variance of caption-level or token-level surprisals (Gutflaish et al., 2017, Ilinykh et al., 6 Nov 2025).

A second major primitive is the KL divergence. In dynamic forecasting, Bayesian surprise is

$$S_{BF}(y_{t+1};\pi^{(t)})=P(y_{t+1};\pi^{(0)})/P(y_{t+1};\pi^{(t)})$$1

while in open-ended scientific discovery the same divergence is used between prior and posterior beliefs over a hypothesis:

$$S_{BF}(y_{t+1};\pi^{(t)})=P(y_{t+1};\pi^{(0)})/P(y_{t+1};\pi^{(t)})$$2

In community detection, asymptotic surprise becomes a KL divergence between the observed fraction of intra-community edges and its null-model expectation,

$$S_{BF}(y_{t+1};\pi^{(t)})=P(y_{t+1};\pi^{(0)})/P(y_{t+1};\pi^{(t)})$$3

In self-supervised image clustering, the surprise score is a sum of binary KL terms over clusters whose empirical match rate exceeds the null rate,

$$S_{BF}(y_{t+1};\pi^{(t)})=P(y_{t+1};\pi^{(0)})/P(y_{t+1};\pi^{(t)})$$4

These formulations share the same information-theoretic core but differ in what is being compared: beliefs, graph densities, or cluster-match frequencies (Agarwal et al., 30 Jun 2025, Traag et al., 2015, Zhang et al., 8 Jul 2026).

A third family standardizes or ranks deviations relative to a reference set. In earnings-surprise prediction, FinCall-Surprise defines the surprise score as standardized unexpected earnings,

$$S_{BF}(y_{t+1};\pi^{(t)})=P(y_{t+1};\pi^{(0)})/P(y_{t+1};\pi^{(t)})$$5

where $$S_{BF}(y_{t+1};\pi^{(t)})=P(y_{t+1};\pi^{(0)})/P(y_{t+1};\pi^{(t)})$$6 and $$S_{BF}(y_{t+1};\pi^{(t)})=P(y_{t+1};\pi^{(0)})/P(y_{t+1};\pi^{(t)})$$7 are the mean and standard deviation of analyst forecasts issued within 30 days after the conference call (Shu et al., 4 Oct 2025). In embedding-based similarity, the surprise score is the percentile of a key-query similarity within an ensemble:

$$S_{BF}(y_{t+1};\pi^{(t)})=P(y_{t+1};\pi^{(0)})/P(y_{t+1};\pi^{(t)})$$8

In DISRC, surprise is the latent-space deviation

$$S_{BF}(y_{t+1};\pi^{(t)})=P(y_{t+1};\pi^{(0)})/P(y_{t+1};\pi^{(t)})$$9

with both latent vectors L2-normalized before comparison (Bachlechner et al., 2023, Kini et al., 24 Jan 2026).

3. Prediction targets and control signals

In corporate-finance forecasting, the surprise score is not merely descriptive; it is the benchmark target. FinCall-Surprise computes $s^\mu(v,w)=d(v,w)-d(v,\mu),$0 from reported EPS and analyst forecasts, then applies a three-way partition with threshold $s^\mu(v,w)=d(v,w)-d(v,\mu),$1:

$s^\mu(v,w)=d(v,w)-d(v,\mu),$2

The neutral region is excluded, and models are trained or prompted to predict $s^\mu(v,w)=d(v,w)-d(v,\mu),$3. Fine-tuning uses standard binary cross-entropy, and evaluation reports Accuracy, Macro Precision, Macro Recall, and Macro F1 because the retained dataset is heavily skewed toward positive surprises. The benchmark further reports that apparently high accuracy can be an illusion caused by class imbalance, that some specialized financial models show weaknesses in instruction-following and language generation, and that audio and visual inputs provide only limited gains under current models (Shu et al., 4 Oct 2025).

In autonomous scientific discovery, surprise is turned into a binary reward. AutoDS models prior and posterior beliefs over a hypothesis as Beta distributions elicited by repeated LLM sampling, computes KL divergence between the two, and counts a hypothesis as surprising only when the expected belief crosses a decision threshold $s^\mu(v,w)=d(v,w)-d(v,\mu),$4 and the posterior mean differs from the prior mean. The reward is then

$s^\mu(v,w)=d(v,w)-d(v,\mu),$5

and this reward enters a UCT score inside MCTS with progressive widening. Under a budget of 500 hypothesis evaluations across 21 real-world datasets, AutoDS yields 5–29% more surprisals than baselines, and AutoDS+MCTS is best on 17 of 21 datasets (Agarwal et al., 30 Jun 2025).

Sequential experimentation uses surprise in a different control loop. A Gaussian-process surrogate is refit after each experiment; Shannon surprise is the negative log predictive density of the new observation, and Bayesian surprise is the KL divergence between old and new Gaussian posteriors. A two-state policy compares the surprise of a new point to the surprise of an observation on the GP’s $s^\mu(v,w)=d(v,w)-d(v,\mu),$6-$s^\mu(v,w)=d(v,w)-d(v,\mu),$7 credible band. If the point is not surprising, the policy explores using a maximin space-filling rule; if it is surprising, the policy exploits by re-sampling in the neighborhood, discarding isolated surprises as noise and continuing local sampling while surprise persists. On Camelback, Branin, and a grinding-process dataset, the surprise-guided policies attain the lowest median RMSE and the smallest IQR among the compared methods (Ahmed et al., 2021).

In deep reinforcement learning, DISRC inserts surprise into reward shaping. The latent deviation $s^\mu(v,w)=d(v,w)-d(v,\mu),$8 is multiplied by a decaying factor $s^\mu(v,w)=d(v,w)-d(v,\mu),$9 to form a negative bonus $d$0, which is then added to a reward normalized by an EMA of its absolute value. Because the shaped reward enters the TD target, surprise influences the TD error and therefore the effective gradient scale. On MiniGrid-DoorKey-8x8, DISRC reaches the first successful episode 33% faster than vanilla DQN, with lower reward standard deviation and higher reward AUC; on MiniGrid-LavaCrossingS9N1 it achieves higher final reward and the highest AUC of all agents (Kini et al., 24 Jan 2026).

4. Language, similarity, recommendation, and voting

In image captioning, surprise is used to measure diversity rather than correctness. The relevant quantity is surprisal variance, defined as the variance across five captions per image of the mean token surprisal under a scorer $d$1. On the MSCOCO Karpathy test split, a caption-trained n-gram LLM yields human surprisal variance roughly twice that of the tested vision-LLMs under greedy decoding, whereas rescoring the same captions with a general-language GPT-2 model reverses the conclusion and makes model captions appear more diverse. The central methodological result is that conclusions about diversity can invert completely when the scorer changes (Ilinykh et al., 6 Nov 2025).

In embedding-based NLP, surprise becomes an ensemble-normalized similarity. For a base similarity $d$2, a query $d$3, a key $d$4, and an ensemble $d$5, the surprise score is the percentile rank of $d$6 within the distribution of similarities $d$7. The paper reports 10–15% better performance compared to raw cosine similarity on zero- and few-shot document classification and improved clustering metrics, and it emphasizes that the ensemble provides the context needed to capture a contrast effect absent from ordinary pairwise similarity (Bachlechner et al., 2023).

In recommender systems, surprise is treated as a finite resource relative to what a user already knows. If $d$8 is the known item set and $d$9 the unknown set, the instantaneous surprise of recommending item $d(v,\mu)$0 is

$d(v,\mu)$1

and cumulative surprise for a recommendation sequence is defined recursively. Because order matters, the framework defines a user-specific maximum potential surprise $d(v,\mu)$2 and minimum potential surprise $d(v,\mu)$3 over all permutations of the unknown items, then normalizes any recommendation list by

$d(v,\mu)$4

MovieLens experiments show MSI variants near the upper bound, LSI variants near the lower bound, and standard item-kNN in between across multiple item representations and distance functions (Lima et al., 2018).

Voting theory uses surprise to aggregate not only observed votes but also participants’ predictions of other voters. For alternative $d(v,\mu)$5, the observed vote share is

$d(v,\mu)$6

the average predicted share is

$d(v,\mu)$7

and the surprisingly popular score is

$d(v,\mu)$8

or equivalently a vote-to-prediction ratio. Pairwise constructions extend the method to full rankings using partial votes and partial predictions. Experimental results show that even limited prediction information improves recovery relative to classical vote-only rules (Hosseini et al., 2021).

5. Structural surprise in graphs and networks

In community detection, surprise is a statistical quality function for a partition of a graph. The original definition is the negative logarithm of the hypergeometric tail probability of observing at least $d(v,\mu)$9 internal edges when $v$0 edges are placed uniformly at random over the $v$1 possible node pairs:

$v$2

For large graphs, this yields the asymptotic form

$v$3

where $v$4 is the observed internal-edge fraction and $v$5 its null-model expectation. This approximation supports an efficient Louvain-style optimization procedure and a straightforward extension to weighted graphs. The 2015 analysis concludes that surprise is nearly unaffected by the usual resolution limit for realistically sized networks, but may overestimate the number of communities, while modularity may underestimate them (Traag et al., 2015).

The multi-resolution extension introduces a parameter $v$6 through

$v$7

which shifts the null-model expectation and allows finer or coarser partitions to be favored. Theoretical analysis on loop networks identifies a critical community count at which merges become favorable and shows that asymptotic surprise has higher resolution than modularity, though it still exhibits a first-type resolution limit at fixed $v$8. An improved Louvain algorithm with common-neighbors precondensation and final vertex-level refinement is proposed to optimize the score more effectively, and experiments report stronger tolerance against the second-type resolution limit and successful recovery of multi-scale communities in synthetic benchmarks (Xiang et al., 2018).

The framework is extended further to mesoscale structures in weighted networks. Here surprise is formulated as a p-value over binary communities, binary bimodular structures, weighted communities, weighted bimodular structures, and two enhanced variants that jointly encode binary and weighted constraints. The weighted versions use negative-hypergeometric or multivariate negative-hypergeometric distributions, while the enhanced versions combine hypergeometric and negative-hypergeometric components. The paper reports correct Type I error control under random-graph nulls, strong performance on ring-of-cliques, LFR, relaxed-caveman, and core–periphery benchmarks, and applications to social, financial, ecological, and trade networks (Marchese et al., 2021).

6. Interpretation, validation, and recurring limitations

A recurring theme is that a surprise score is only meaningful relative to its reference model. In earnings forecasting, the score depends on the analyst-consensus distribution and the chosen threshold $v$9; because positive surprises dominate, macro-averaged metrics are required to avoid misleading accuracy claims (Shu et al., 4 Oct 2025). In image captioning, the reference model is the scorer, and changing from a caption-trained n-gram model to a general-language GPT-2 model can invert whether humans or models appear more diverse (Ilinykh et al., 6 Nov 2025). In similarity scoring, the reference is the ensemble; the same raw cosine similarity can map to a very different surprise percentile depending on the background distribution of similarities (Bachlechner et al., 2023).

The literature also documents domain-specific pathologies. In traffic forecasting, Shannon surprisal has a non-zero floor and requires arbitrary discretization, while full Bayesian surprise captures all belief changes, including mode-narrowing and other unsurprising information gain; Residual Information and Antithesis are proposed partly to address these issues (Dinparastdjadid et al., 2023). In belief revision, the relative-surprise operator satisfies Success, Consistency, Extensionality, Neutrality, Flipping, Addition, restricted Sub/Superexpansion, and Best-of-Worst-Surprise, but it violates Vacuity and unrestricted Subexpansion/Superexpansion (Haret, 2021). In community detection, surprise is more discriminative than modularity for many small communities, yet it can over-split very large communities (Traag et al., 2015).

Several papers explicitly evaluate whether surprise aligns with human judgments. In AutoDS, human evaluation over 1,620 LLM-surprising hypotheses shows that Bayesian-surprise reward correlates with human surprisal on 67% of cases, with Gwet’s AC1 between 0.61 and 0.71 across familiarity levels (Agarwal et al., 30 Jun 2025). In driving-scenario curation, the best surprise-potential instantiation achieves Spearman correlation $\mu$0 with a human-aligned reward model and AUC-ROC of 0.86 when the top 10% of scenarios by that reward are treated as interactive (Ding et al., 8 Feb 2025). These results suggest that surprise scores are often most useful when interpreted not as absolute quantities, but as task-specific functionals whose meaning is fixed by an explicit prior, null hypothesis, or scoring distribution (Agarwal et al., 30 Jun 2025, Ding et al., 8 Feb 2025).