Likelihood scoring for continuations of mathematical text: a self-supervised benchmark with tests for shortcut vulnerabilities

Abstract: We introduce an automatically generated benchmark for predicting hidden text in technical papers. A paper supplies visible context $X$ and a hidden continuation $Y$; the evaluated model writes an auxiliary forecast string $Z$, and a separate scorer assigns next-token probability to $Y$ both with and without conditioning on $Z$. This gives a label-free test of whether $Z$ transmits information about the continuation, compared against controls where $Z$ is recent context rather than a forecast. Our main testbed is equation-suffix prediction: the predictor sees context and the first part of a displayed equation, then forecasts the rest. The task mixes surface-level arXiv/TeX text modeling with reasoning-sensitive inference; the suffix is one of many roughly equivalent continuations, so the benchmark is read statistically rather than item-by-item. On 1363 equation continuations from 138 recent physics and mathematics papers, forecasts from GPT-5.5, Opus 4.7, and GPT-5.4 nano all improve clipped likelihood over the context control under both Qwen3-8B and Kimi K2.6 scorers, distinguishing model families and reasoning-effort settings without human labels. To emulate shortcuts where $Z$ further primes the scorer rather than making a useful forecast, we also fine-tune the scorer on context-only prompts and apply it to held-out papers as a stronger control. GPT-5.5 forecasts still beat this fine-tuned control; GPT-5.4 nano forecasts do not. Longer prose/TeX continuations show positive but noisier lift over controls, concentrated near the beginning of the target. These results support cross-model likelihood scoring as a static benchmark and as a setup for probing shortcut vulnerabilities before reinforcement learning or model-selection optimization is applied.

• The paper introduces a benchmark using likelihood lift scoring to evaluate technical continuations by comparing predicted strings with context-only controls.
• It employs controlled experiments with advanced scorer designs to expose shortcut vulnerabilities like context-stuffing and scorer priming.
• Findings show that higher reasoning effort and robust scoring metrics lead to significant improvements in model ranking and performance.

Automated Benchmarking for Mathematical Text Continuation via Likelihood Scoring

Benchmark Design and Methodology

The paper introduces an automatically generated benchmark targeting technical text continuation prediction, with a focus on equation-suffix forecasting in arXiv manuscripts. The core protocol provides visible context $X$, a hidden continuation $Y$, and evaluates whether a predicted string $Z$ from a model improves next-token likelihood scores for $Y$ when compared to context-based controls. Two LLMs are employed: one generates the auxiliary string $Z$ (the predictor) and another assigns likelihoods to $Y$ (the scorer). The primary metric is the per-token log-likelihood lift, measuring information transmitted by $Z$ about $Y$ conditional on $X$.

To account for shortcut vulnerabilities such as context-stuffing and scorer priming, the paper uses controls including recent context excerpts and supervised fine-tuned (SFT) scorer variants. The scoring metric (clipLL2), which clips catastrophic token losses, robustly mitigates outlier penalties from syntactically divergent but mathematically equivalent equation continuations.

Main Results: Equation-Suffix Prediction

On a dataset comprising 1363 equation-suffix targets from 138 math/physics arXiv papers, model forecasts consistently yield positive likelihood lift over same-budget context controls under both Qwen3-8B and Kimi K2.6 scorers. Forecasts from GPT-5.5 exhibit the strongest lift, followed by Opus 4.7 and GPT-5.4 nano. Gradient analysis across reasoning-effort settings (as determined by API-reported hidden token counts) reveals monotonic ordering: higher reasoning effort correlates with higher likelihood lift, confirming the benchmark's sensitivity to both predictor quality and internal compute allocation.

Forecasts also outperform a context-only SFT scorer, indicating the benchmark's resilience to context-based scaffold priming; GPT-5.5 forecasts survive this strong shortcut control, whereas GPT-5.4 nano does not. Analyses on per-cut distributions demonstrate that likelihood improvement is broadly distributed and not limited to tail effects.

Extension to Prose/TEX Continuation

An additional experiment evaluates longer technical continuations with mixed prose and TEX, using 661 cuts from 40 manuscripts. While the likelihood lift remains positive and separates model families, it is concentrated near the beginning of the forecasted target and is markedly noisier. Adjacent reasoning-effort contrasts are less robust, and the regime is shown to be more sensitive to scorer choice and length-dependent artifacts. SFT-based controls in this regime further illustrate vulnerability to adversarial context-only strategies, emphasizing the necessity of scaffold-aware scorer design.

Vulnerability Auditing and Shortcut Controls

Systematic analysis of shortcut vulnerabilities highlights the importance of non-adversarial controls for reliable reward signaling. The study deploys context-only budget controls and SFT-trained scorers as static probes. The scaffold design tightly restricts the attack surface, but does not exhaustively defend against forecast-plus-hacking or idiosyncratic adversarial exploits. The benchmark establishes that strong predictors transmit genuine information that is not recoverable by shortcut strategies, evidenced by their ability to surpass context-only SFT controls and approach oracle performance bounds.

Theoretical and Practical Implications

The paper positions likelihood-based reward signals as a robust alternative to LLM-as-judge evaluations, offering reproducibility and reference-grounded benchmarking for technical continuation tasks. The approach sits between abundant next-token prediction (surface-level modeling) and scarce, verifiable technical problems (RLVR style), covering a valuable portion of the technical reasoning benchmark spectrum.

Static benchmarking with likelihood scorers provides strong evidence that auxiliary forecast strings can encode useful information for technical continuations, potentially informing reinforcement-learning objectives for technical domains. The method supports model ranking and reasoning-effort discrimination without reliance on human labeling or rubric-based judgment.

Limitations and Future Directions

The benchmark is limited by its static, non-adversarial protocol: predictors directly optimized against the reward could exploit vulnerabilities outside the current control set. The domain is restricted to equation-rich physics and mathematics, and scorer/model selection substantially influences results in longer continuations. Notably, adjacent reasoning-effort comparisons are less clean outside equation-suffix tasks, and the lack of an Opus reasoning-effort sweep constrains cross-provider capability ordering.

Future work should address stronger adversarial optimization of $Z$, extend coverage to broader technical domains, and experiment with RL pilot studies using likelihood-lift rewards. Improvements could include targeted task selection heuristics and further scorer diversification to probe cross-model generalization.

Conclusion

This paper establishes an automated benchmark for technical text continuation via likelihood lift scoring, robustly separating model families and reasoning-effort settings while auditing shortcut vulnerabilities (2605.10810). The equation-suffix prediction task demonstrates strong statistical signal, supporting its use for model ranking and as a foundation for reward design in technical domains. The protocol is amenable to further extension, adversarial auditing, and RL-oriented reward optimization, offering a practical and theoretically sound approach to benchmarking mathematical reasoning in LLMs.