Cross-Model Likelihood Scoring

• Cross-model likelihood scoring is a framework that assigns comparable numerical scores to competing models using Bayesian marginal likelihood, cross-validation log-scores, and proper scoring rules.
• It demonstrates that, under logarithmic scoring, exhaustive cross-validation can be equivalent to marginal likelihood, thereby ensuring predictive adequacy and coherence through exchangeability.
• The approach extends to modern applications such as language model evaluation and cross-tokenizer scoring, addressing challenges like prior sensitivity, computational intractability, and model misspecification.

Searching arXiv for relevant papers on cross-model likelihood scoring and closely related model-scoring frameworks. Cross-model likelihood scoring refers to assigning comparable numerical scores to competing models so that they can be ranked or selected. Across the literature, the term encompasses several closely related practices: scoring models by marginal likelihood or Bayes factors, by cross-validation log-scores, by alternative proper scoring rules such as the Hyvärinen score, by divergence-based surrogates in likelihood-free simulation settings, and, more recently, by using one LLM to generate an auxiliary forecast that a separate scorer model evaluates through conditional likelihood of hidden text. A recurrent theme is that the score is not merely a computational convenience: under explicit assumptions, it determines what notion of predictive adequacy, coherence, or robustness is being optimized (Fong et al., 2019, Shao et al., 2017, Corander et al., 2022, Ranard, 11 May 2026, Phan et al., 16 Dec 2025).

1. Foundations: likelihoods, predictive scores, and coherence

For a Bayesian model $\mathcal{M}$ with likelihood family $\{f_\theta(y): \theta \in \Theta\}$, prior $\pi(\theta)$, and data $y_{1:n}=(y_1,\dots,y_n)$, the marginal likelihood is

$$p_{\mathcal{M}}(y_{1:n})=\int f_\theta(y_{1:n})\,d\pi(\theta) =\int \Bigl(\prod_{i=1}^n f_\theta(y_i)\Bigr)\,d\pi(\theta),$$

and Bayes factors compare models through ratios of such quantities. The same object admits the prequential factorization

$$\log p_{\mathcal{M}}(y_{1:n})=\sum_{i=1}^n \log p_{\mathcal{M}}(y_i\mid y_{1:i-1}),$$

where $p_{\mathcal{M}}(y_i\mid y_{1:i-1})$ is the posterior predictive. In the fixed-hypothesis formulation, the ordinary log-likelihood $\log P(D\mid HI)$ can likewise be expressed through repeated use of the chain rule, without requiring i.i.d. sampling (Fong et al., 2019, Mana, 2019).

The predictive quantity that recurs throughout this literature is the log posterior predictive probability

$$s(\tilde y \mid y_{1:n})=\log p_{\mathcal{M}}(\tilde y\mid y_{1:n}),$$

used as a scoring rule. In the general Bayesian framework summarized in "On the marginal likelihood and cross-validation" (Fong et al., 2019), a model scoring function $g(l)$ is called coherent under exchangeability if cumulative sequential scores equal the log of a corresponding batch score for all priors, parameter spaces, and sample sizes. Under continuity and monotonicity conditions, the only coherent choice is $\{f_\theta(y): \theta \in \Theta\}$0. In the usual Bayesian case with $\{f_\theta(y): \theta \in \Theta\}$1 and $\{f_\theta(y): \theta \in \Theta\}$2, coherence therefore forces the log posterior predictive score. A central consequence is the corollary that the marginal likelihood is the unique coherent marginal score for Bayesian inference under the paper’s assumptions (Fong et al., 2019).

Exchangeability is the structural condition that makes these identities non-arbitrary. If the joint law is invariant under permutations, the cumulative prequential score must not depend on data ordering. This requirement is explicit in the uniqueness theorem for coherent scoring rules and underlies the later equivalence results linking likelihood and cross-validation (Fong et al., 2019).

2. Equivalence with cross-validation

A principal result of the recent theory is that marginal-likelihood-based comparison and cross-validation-based comparison are not disjoint paradigms when the score is logarithmic. For leave-$\{f_\theta(y): \theta \in \Theta\}$3-out cross-validation, the per-datum score is defined by averaging the predictive score over all $\{f_\theta(y): \theta \in \Theta\}$4 held-out subsets of size $\{f_\theta(y): \theta \in \Theta\}$5. Proposition 2 in "On the marginal likelihood and cross-validation" shows that

$\{f_\theta(y): \theta \in \Theta\}$6

where $\{f_\theta(y): \theta \in \Theta\}$7 is the exhaustive leave-$\{f_\theta(y): \theta \in \Theta\}$8-out score computed with the log posterior predictive. Thus the Bayesian marginal likelihood is exactly a cumulative leave-$\{f_\theta(y): \theta \in \Theta\}$9-out cross-validation score averaged over all values of $\pi(\theta)$0 and all held-out test sets. The result gives a combinatorial reinterpretation of evidence as exhaustive predictive validation across all training/test splits, including prior-only prediction when $\pi(\theta)$1 (Fong et al., 2019).

A parallel identity appears in "A relation between log-likelihood and cross-validation log-scores" (Mana, 2019). There, for a fixed hypothesis $\pi(\theta)$2 and data $\pi(\theta)$3, the log-likelihood $\pi(\theta)$4 is written as an average over all leave-one-out cross-validation log-scores computable from all subsets of the data, and the construction is stated to generalize to any $\pi(\theta)$5-fold cross-validation log-scores. The key expressions average conditional log probabilities $\pi(\theta)$6 over all subset sizes $\pi(\theta)$7 and all subsets $\pi(\theta)$8 of that size. In this formulation, the result is algebraic and uses only the chain rule, not independence assumptions (Mana, 2019).

Taken together, these results place cross-model likelihood scoring on a unified prequential footing. In the Bayesian setting, evidence is an exhaustive cross-validation criterion built from log posterior predictive probabilities. In the fixed-model probability-calculus setting, the joint log-likelihood is a weighted average over all conceivable cross-validation log-scores for all subsets of data. This suggests that disagreements between evidence and a specific practical CV procedure are due less to an underlying conceptual incompatibility than to which subset of predictive conditionals is being averaged and how priors enter the predictive distribution (Fong et al., 2019, Mana, 2019).

3. Beyond the log-score

The log-score is not the only proper scoring rule used for cross-model comparison, but alternative rules change both the computational problem and the asymptotic target. "Bayesian model comparison with the Hyvärinen score: computation and consistency" introduces the Hyvärinen score

$\pi(\theta)$9

for continuous data, together with its prequential version

$y_{1:n}=(y_1,\dots,y_n)$0

The score is proper, local of order 2, and homogeneous, hence invariant to multiplicative rescaling of the density. That homogeneity makes it insensitive to normalizing constants and therefore suitable when priors are vague or improper, or when only unnormalized likelihoods are available. The paper defines the H-factor $y_{1:n}=(y_1,\dots,y_n)$1, develops SMC and SMC$y_{1:n}=(y_1,\dots,y_n)$2 estimators for tractable parametric models and nonlinear non-Gaussian state-space models, and proves asymptotic consistency under strong regularity assumptions. The asymptotic divergence minimized by the Hyvärinen score is a Fisher-type divergence rather than Kullback–Leibler divergence (Shao et al., 2017).

For simulator-based models without tractable likelihoods, "Likelihood-free Model Choice for Simulator-based Models with the Jensen--Shannon Divergence" replaces likelihood by the Jensen–Shannon divergence and derives JSD-Razor,

$y_{1:n}=(y_1,\dots,y_n)$3

with a computable Schwarz-like approximation

$y_{1:n}=(y_1,\dots,y_n)$4

The criterion is derived for categorical simulator outputs, related to Occam-type evidence integrals, and shown to be consistent in nested models, including a misspecified case in which all models are outside the truth (Corander et al., 2022).

Score or object	Characteristic property	Representative setting
Log posterior predictive / marginal likelihood	Unique coherent marginal score under exchangeability	Bayesian model comparison (Fong et al., 2019)
Hyvärinen score / H-score	Proper, local, homogeneous	Vague priors, unnormalized or intractable likelihoods (Shao et al., 2017)
JSD-Razor / SIC-JSD	Likelihood-free, divergence-based	Simulator-based categorical models (Corander et al., 2022)
clipLL2 lift	Bounded per-token likelihood lift	Cross-model LM benchmarking (Ranard, 11 May 2026)
Conversion probability across tokenizers	Shared probability space via BPE recursion	LM distillation with different tokenizers (Phan et al., 16 Dec 2025)

These alternatives do not merely substitute one metric for another. They alter the notion of discrepancy being minimized: KL divergence for the log-score, Fisher-type divergence for the Hyvärinen score, and Jensen–Shannon divergence for JSD-Razor. A plausible implication is that cross-model likelihood scoring is best understood as a broader class of model comparison procedures indexed by a predictive or divergence geometry rather than by a single canonical criterion.

4. Sensitivity, misspecification, and failure modes

The equivalence between evidence and exhaustive predictive scoring immediately clarifies prior sensitivity. Because

$y_{1:n}=(y_1,\dots,y_n)$5

the marginal likelihood includes terms corresponding to very small training sets and, at $y_{1:n}=(y_1,\dots,y_n)$6, prior-only prediction. "On the marginal likelihood and cross-validation" emphasizes that a non-negligible portion of the marginal likelihood evaluates the model by its performance when trained on almost no data. With vague priors on real-valued parameters, these terms can become arbitrarily large and negative, producing Lindley-type effects. The paper therefore proposes cumulative cross-validation after a preparatory training phase,

$y_{1:n}=(y_1,\dots,y_n)$7

with suggested choices $y_{1:n}=(y_1,\dots,y_n)$8, $y_{1:n}=(y_1,\dots,y_n)$9, or $$p_{\mathcal{M}}(y_{1:n})=\int f_\theta(y_{1:n})\,d\pi(\theta) =\int \Bigl(\prod_{i=1}^n f_\theta(y_i)\Bigr)\,d\pi(\theta),$$0, so that model assessment begins only after a substantial amount of data has been observed (Fong et al., 2019).

A different failure mode arises under model misspecification in causal discovery. "On the pitfalls of Gaussian likelihood scoring for causal discovery" studies additive noise models in which the true errors are non-Gaussian but candidate DAGs are ranked by Gaussian log-likelihood, which reduces to minimizing $$p_{\mathcal{M}}(y_{1:n})=\int f_\theta(y_{1:n})\,d\pi(\theta) =\int \Bigl(\prod_{i=1}^n f_\theta(y_i)\Bigr)\,d\pi(\theta),$$1, the sum of residual log-standard-deviations under the best regressors. The paper proves a negative result: under non-Gaussian noise, Gaussian likelihood scores can prefer the wrong causal graph even asymptotically and even when the regressors are the true population conditional expectations. In the linear data-generating model, for every permutation $$p_{\mathcal{M}}(y_{1:n})=\int f_\theta(y_{1:n})\,d\pi(\theta) =\int \Bigl(\prod_{i=1}^n f_\theta(y_i)\Bigr)\,d\pi(\theta),$$2,

$$p_{\mathcal{M}}(y_{1:n})=\int f_\theta(y_{1:n})\,d\pi(\theta) =\int \Bigl(\prod_{i=1}^n f_\theta(y_i)\Bigr)\,d\pi(\theta),$$3

with strict inequality in generic non-Gaussian cases once nonlinear conditional expectations arise under a wrong order. The analysis further shows that allowing heteroskedastic Gaussian scoring can improve the backward model’s score still more, making orientation harder rather than easier (Schultheiss et al., 2022).

The Hyvärinen-score literature highlights a different trade-off. Robustness to vague or improper priors is gained at the price of stronger smoothness assumptions, dependence on first and second derivatives with respect to the observation, and higher Monte Carlo variance than log-evidence estimators. The paper also notes that prequential H-scores depend on data ordering in finite samples, although the asymptotic limits are permutation-invariant for large $$p_{\mathcal{M}}(y_{1:n})=\int f_\theta(y_{1:n})\,d\pi(\theta) =\int \Bigl(\prod_{i=1}^n f_\theta(y_i)\Bigr)\,d\pi(\theta),$$4 (Shao et al., 2017).

These cases delimit the scope of the concept. Cross-model likelihood scoring is not intrinsically reliable simply because it is likelihood-based. Its trustworthiness depends on whether the chosen score aligns with the statistical structure at issue: exchangeability, prior informativeness, correct conditional distributional form, differentiability, or availability of a meaningful simulator discrepancy.

5. Cross-model scoring in modern language-model evaluation

A recent extension of the idea appears in "Likelihood scoring for continuations of mathematical text: a self-supervised benchmark with tests for shortcut vulnerabilities" (Ranard, 11 May 2026). Here a paper provides visible context $$p_{\mathcal{M}}(y_{1:n})=\int f_\theta(y_{1:n})\,d\pi(\theta) =\int \Bigl(\prod_{i=1}^n f_\theta(y_i)\Bigr)\,d\pi(\theta),$$5 and hidden continuation $$p_{\mathcal{M}}(y_{1:n})=\int f_\theta(y_{1:n})\,d\pi(\theta) =\int \Bigl(\prod_{i=1}^n f_\theta(y_i)\Bigr)\,d\pi(\theta),$$6; a forecast model writes an auxiliary string $$p_{\mathcal{M}}(y_{1:n})=\int f_\theta(y_{1:n})\,d\pi(\theta) =\int \Bigl(\prod_{i=1}^n f_\theta(y_i)\Bigr)\,d\pi(\theta),$$7; and a separate scorer model $$p_{\mathcal{M}}(y_{1:n})=\int f_\theta(y_{1:n})\,d\pi(\theta) =\int \Bigl(\prod_{i=1}^n f_\theta(y_i)\Bigr)\,d\pi(\theta),$$8 assigns next-token probabilities to $$p_{\mathcal{M}}(y_{1:n})=\int f_\theta(y_{1:n})\,d\pi(\theta) =\int \Bigl(\prod_{i=1}^n f_\theta(y_i)\Bigr)\,d\pi(\theta),$$9 with and without $$\log p_{\mathcal{M}}(y_{1:n})=\sum_{i=1}^n \log p_{\mathcal{M}}(y_i\mid y_{1:i-1}),$$0. The per-token log-likelihood lift is

$$\log p_{\mathcal{M}}(y_{1:n})=\sum_{i=1}^n \log p_{\mathcal{M}}(y_i\mid y_{1:i-1}),$$1

and the principal bounded metric is

$$\log p_{\mathcal{M}}(y_{1:n})=\sum_{i=1}^n \log p_{\mathcal{M}}(y_i\mid y_{1:i-1}),$$2

where $$\log p_{\mathcal{M}}(y_{1:n})=\sum_{i=1}^n \log p_{\mathcal{M}}(y_i\mid y_{1:i-1}),$$3. The benchmark is label-free because there are no human annotations; evaluation depends only on whether the forecast makes the ground-truth continuation more likely under a frozen scorer (Ranard, 11 May 2026).

The main testbed is equation-suffix prediction: 1363 equation-suffix cuts from 138 recent physics and mathematics papers, with predictor models GPT-5.5, Opus 4.7, and GPT-5.4 nano, and scorer models Qwen3-8B and Kimi K2.6. Forecasts from all three predictor families improve clipped likelihood over the same-budget context control under both scorers, and the ranking GPT-5.5 $$\log p_{\mathcal{M}}(y_{1:n})=\sum_{i=1}^n \log p_{\mathcal{M}}(y_i\mid y_{1:i-1}),$$4 Opus 4.7 $$\log p_{\mathcal{M}}(y_{1:n})=\sum_{i=1}^n \log p_{\mathcal{M}}(y_i\mid y_{1:i-1}),$$5 GPT-5.4 nano holds under both scorers. For Qwen3-8B, mean lift over same-budget context is 0.201 for GPT-5.5 high reasoning, 0.184 for Opus 4.7 medium, and 0.108 for GPT-5.4 nano high; for Kimi K2.6 the corresponding figures are 0.170, 0.153, and 0.076. Within model families, the benchmark detects monotone reasoning-effort gradients: GPT-5.5 high $$\log p_{\mathcal{M}}(y_{1:n})=\sum_{i=1}^n \log p_{\mathcal{M}}(y_i\mid y_{1:i-1}),$$6 medium $$\log p_{\mathcal{M}}(y_{1:n})=\sum_{i=1}^n \log p_{\mathcal{M}}(y_i\mid y_{1:i-1}),$$7 low $$\log p_{\mathcal{M}}(y_{1:n})=\sum_{i=1}^n \log p_{\mathcal{M}}(y_i\mid y_{1:i-1}),$$8 none, Opus 4.7 medium $$\log p_{\mathcal{M}}(y_{1:n})=\sum_{i=1}^n \log p_{\mathcal{M}}(y_i\mid y_{1:i-1}),$$9 low, and GPT-5.4 nano high $p_{\mathcal{M}}(y_i\mid y_{1:i-1})$0 medium $p_{\mathcal{M}}(y_i\mid y_{1:i-1})$1 low (Ranard, 11 May 2026).

The same paper subjects this cross-model score to shortcut audits. Controls include an empty scaffold, a same-budget recent-context control, a triple-budget context control, and a context-only SFT scorer obtained by LoRA fine-tuning Qwen3-8B on prompts without forecasts. On a 731-cut subset, mean clipLL2 relative to the empty scaffold is +0.096 for same-budget context, +0.164 for triple-budget context, +0.200 for context-only SFT, +0.298 for GPT-5.5 high-reasoning forecasts, and +0.569 for the true suffix used as forecast. GPT-5.5 forecasts therefore remain above the stronger context-only route, whereas analogous experiments show that GPT-5.4 nano forecasts do not significantly beat the SFT control. The longer prose/TeX continuation setting, built from 661 cuts from 40 papers, yields positive but noisier lift concentrated near the beginning of the target (Ranard, 11 May 2026).

This formulation changes the meaning of cross-model likelihood scoring in two respects. First, the model being evaluated and the model assigning likelihood are intentionally distinct. Second, the score is used not only for selection but also for shortcut-vulnerability analysis before reinforcement learning or model-selection optimization is applied. This suggests an emerging use of likelihood scoring as a static diagnostic for reasoning-sensitive generation tasks rather than solely as a classical criterion for statistical model choice.

6. Probability-space alignment and cross-tokenizer scoring

A separate modern extension concerns cases in which the competing or collaborating models do not even share a tokenizer. "Cross-Tokenizer Likelihood Scoring Algorithms for LLM Distillation" studies a teacher LM over vocabulary $p_{\mathcal{M}}(y_i\mid y_{1:i-1})$2 and a student LM over vocabulary $p_{\mathcal{M}}(y_i\mid y_{1:i-1})$3, and asks how to compute teacher probabilities in the student token space. The paper formalizes BPE vocabularies recursively, defines canonical encodings by

$p_{\mathcal{M}}(y_i\mid y_{1:i-1})$4

and introduces conversion probability: the probability that a teacher-generated string, once decoded and re-encoded with $p_{\mathcal{M}}(y_i\mid y_{1:i-1})$5, begins with a target student prefix $p_{\mathcal{M}}(y_i\mid y_{1:i-1})$6 (Phan et al., 16 Dec 2025).

The core technical device is the set of relative cover encodings. In the subset regime $p_{\mathcal{M}}(y_i\mid y_{1:i-1})$7, Lemma 1 states that for a valid encoding $p_{\mathcal{M}}(y_i\mid y_{1:i-1})$8,

$p_{\mathcal{M}}(y_i\mid y_{1:i-1})$9

This yields exact likelihoods in the subset regime and supports next-token probabilities for sequential sampling with only $\log P(D\mid HI)$0 model evaluations per token. In the general regime of arbitrary BPE vocabularies, the paper introduces a rigorous lossless recursive procedure based on adjacent vocabulary transitions, and complements it with a beam-search-based fast approximation to keep large-vocabulary settings practical (Phan et al., 16 Dec 2025).

The application is distillation. Once teacher probabilities are represented over the student vocabulary, standard KL-style training becomes well defined in the subset regime, and a Partial KL loss is introduced for the general regime when only a small candidate set of student tokens is scored. Empirically, vocabulary trimming of Qwen2.5-1.5B yields up to a 12% reduction in memory footprint while also improving baseline performance up to 4% on the evaluated tasks, and in the arbitrary-tokenizer setting the method improves GSM8K accuracy by more than 2% over the current state of the art (Phan et al., 16 Dec 2025).

In conceptual terms, this work treats tokenization as a latent segmentation problem and cross-model likelihood scoring as the problem of moving probabilities between discrete sample spaces without losing probabilistic meaning. That perspective is consistent with the broader literature: whether the obstacle is exchangeability, improper priors, intractable likelihoods, hidden continuation scaffolds, or tokenizer mismatch, the central question remains how to define a comparable score that preserves the intended semantics of prediction and model fit.