StakeBench: Evaluating Language Understanding Grounded in Market Commitment

Abstract: Existing financial NLP benchmarks often rely on labels supplied by outside observers, measuring how language is perceived rather than what speakers have committed to in the market. We introduce StakeBench, an evaluation framework for language understanding grounded in market commitment. StakeBench links 560,876 comments from 2,261 resolved markets to verified position, action, and market-odds records across Polymarket and Manifold. Supervision is derived from observable market behavior. Position sides, post-comment trading actions, and market-odds trajectories replace human annotation. Four diagnostic tasks test whether models detect market commitment, identify the revealed side, anticipate future action, and perform collective odds projection. Three commitment-aware metrics measure alignment with revealed preferences rather than perceived sentiment. Validity audits and explicit interpretation boundaries help distinguish observable commitment signals from latent belief and causal market-odds impact. Across 15 LLMs and 18 topics and platform settings, models partially recover position-side signals, with Directed Accuracy from 0.506 to 0.599, but show structural failures on later tasks. Ten of the fifteen models collapse to one or two action labels in future action anticipation, and no model consistently improves on the naive odds-direction baseline in collective odds projection. Model scale is not correlated with performance, finance-domain tuning does not improve revealed-side identification, and platform incentives strongly shape higher-order results. StakeBench is packaged with evaluation code and dataset under CC-BY 4.0.

• The paper introduces StakeBench, a 560,876-comment benchmark linking financial language to verified market positions, subsequent trades, and market odds across Polymarket and Manifold.
• The paper finds that LLMs detect whether speakers hold positions relatively well, but struggle to infer position sides, anticipate future actions, and outperform stake-weighted baselines for market-direction forecasts.
• The paper shows that model scale and finance-specific training do not reliably improve commitment understanding, while data coverage, platform differences, and selection bias limit conclusions for behavioral prediction tasks.

Motivation and positioning

Financial NLP benchmarks largely evaluate how text is classified by external observers—sentiment, stance, or outcome labels—rather than what speakers have demonstrably committed to in a market. StakeBench addresses this gap by grounding supervision in revealed preferences: each comment is linked to the author's verified market position, subsequent trading behavior, and the collective odds trajectory of the underlying market. The design draws on Samuelson's revealed-preference framework and Spence's costly-signaling model, treating a paid position as behavioral evidence that perception-based annotation cannot supply. Compared with prior resources—FinBen, PIXIU, SemEval-2016 Task 6, Autocast, ForecastBench—no existing benchmark jointly connects individual language to verified positions, later actions, and market-odds signals (2605.26074).

Dataset construction

StakeBench contains 560,876 comments from 2,261 resolved markets across 18 topic–platform combinations drawn from Polymarket (real money; 514,314 comments, 2,070 markets) and Manifold (play money; 46,562 comments, 191 markets). Position side and stake magnitude are reconstructed deterministically by replaying public trade histories up to each comment timestamp via public APIs. The dual-platform design is deliberate: Polymarket supplies real-money commitment with heavy-tailed stakes, while Manifold offers denser position coverage (80.5% vs. 27.2%) and direct resolution metadata.

The authors are explicit that all labels are behavioral proxies rather than measurements of private belief or causal price impact, and they document interpretation boundaries for each task accordingly.

Four diagnostic tasks

The benchmark defines a progressive hierarchy:

• G1 — Market Commitment Detection: binary classification of whether the commenter holds any position.
• G2 — Revealed-Side Identification: which side (YES/NO) the commenter's position supports, with abstention allowed.
• G3 — Future Action Anticipation: four-class prediction of the first net position action within 48 hours (flip, increase, decrease, hold).
• G4 — Collective Odds Projection: whether aggregated comment signals predict next-window odds direction beyond a stake-weighted baseline $\tilde{d}$.

Three commitment-aware metrics supplement standard accuracy: Commitment-Calibrated Accuracy (CCA), Costly Signal Delta (CSD, the DA difference between highest and lowest stake quartiles), and Commitment Gap (CG/MCG), which measures shortfall against revealed-preference reference scores $O_2$, $O_3$, $O_4$. Three falsification audits—a Wrong-Bettor test, a stake-magnitude dose-response test, and a within-user difference-in-differences—probe alternative explanations such as smart-money effects or outcome prediction.

Evaluation setup

Fifteen LLMs spanning closed-source (GPT-5.5, Gemini-2.5-Flash, Claude-Haiku-4.5), general open-weight (Qwen3 family, Gemma2-9B, DeepSeek-R1-8B, Llama-3.2-3B, Mistral-7B), and finance-domain models (FinMA-30B, Fino1-8B/14B, Finance-Chat-7B) are evaluated under fixed prompting with greedy decoding. Macro averaging over topic–platform splits prevents high-volume markets from dominating scores.

Results

G1 transfers almost perfectly across platforms. Eleven of fifteen models exceed 0.65 accuracy, led by Qwen3-32B at 0.747 (F1 0.711); cross-platform rank correlation is $\rho = 0.975$ with negligible mean gap ($\delta = 0.016$, $p = 0.73$). Committed speech thus carries discourse-level markers stable across incentive regimes—the most transferable layer of the hierarchy.

G2 reveals a hard ceiling near chance-plus-margin. Directed Accuracy spans only 0.506 to 0.599, indicating that many position sides are not recoverable from surface text. GPT-5.5 leads (DA 0.599, CCA 0.678), showing stronger signal in high-stake comments. Notably, FinMA-30B predicts market resolution (DA 0.538) better than commenter side (0.506)—the reverse of general models—suggesting finance-domain tuning induces outcome-oriented rather than commitment-oriented reading. Model orderings also diverge sharply across platforms ($\rho = 0.286$).

G3 exposes structural class collapse. Ten of fifteen models produce usable recall on only one or two action classes; Fin-Chat-7B predicts "flip" for 100% of examples, and mean hold recall is 22.9% despite hold being 47.6% of the ground truth—models systematically over-predict behavioral change. Scores are stable across 6h–72h windows, so this is not a horizon artifact. The paper concedes a data limitation here: only 14.1% of positioned Polymarket comments can be matched to subsequent position changes (vs. 86.0% on Manifold) due to incompatible API identifiers, making the Polymarket G3 subset non-random and likely biased toward high-frequency traders.

G4 shows no model beats the naive baseline. All $\Delta_{\mathrm{PLA}}$ values are negative ($-0.022$ to $O_2$0): no LLM adds collective odds-direction signal beyond the stake-weighted majority of revealed positions. Positive gains concentrate on Manifold (27 of 32 model–split instances), consistent with more residual inefficiency on play-money platforms. G4 is uncorrelated with G1–G3 ($O_2$1), so speaker-level understanding and crowd-level projection measure distinct abilities; the task chain correlates significantly through G1→G2→G3 but breaks at G4.

Three cross-cutting findings stand out: model scale is uncorrelated with performance across the 3B–32B range ($O_2$2, $O_2$3); paired comparisons show finance-domain tuning does not improve speaker-level tasks (e.g., FinMA-30B trails Qwen3-32B by 0.245 on G1 accuracy); and MCG ranges from 0.242 (GPT-5.5) to 0.440 (FinMA-30B), dominated by the G3 term.

Validity audits

The audits partially support the commitment-grounding premise but with caveats the authors state plainly. In the Wrong-Bettor test, 25 of 116 eligible pairs reach significance, with commitment-following rates reaching 0.71–0.81 on one Polymarket politics split—but effects concentrate on a small set of topics. Dose-response evidence is limited to Manifold economics (12 of 154 significant pairs). The within-user DiD has macro mean +0.026, and post-position language aligns more strongly with the user's own side (DA 0.621 vs. 0.508 pre-position against resolution). Statistical reliability declines along the task chain: after BH-FDR correction, G1 retains 92.8% of discoveries while G3 slices retain only 30–41%, so lower-task conclusions rest on weaker statistical footing.

Limitations and open questions

The paper identifies several constraints. CSD and DiD audits are underpowered on some topics. The Polymarket G3 coverage gap introduces selection bias toward active traders. Fixed prompts and LLM probes measure evaluation behavior under one protocol, not best achievable performance. Public comments may contaminate future pretraining corpora. G4 results may reflect market efficiency as much as model limits, and the benchmark cannot separate the two. All data are English-only, and position-holding commenters differ modestly from non-holders on observable dimensions (all effect sizes below Cohen's $O_2$4). Open questions include whether alternative prompting or fine-tuning on position context can lift G3 above degenerate collapse, and whether residual G4 signal exists on real-money platforms at all.

Conclusion

StakeBench contributes an auditable, annotation-free evaluation protocol linking language to verified market commitments, with explicit behavioral targets and interpretation boundaries. Its empirical findings are sobering: current LLMs partially recover who holds a position and weakly infer its side, but fail to anticipate subsequent trading actions and add no value over a naive stake-weighted baseline for odds projection. The evidence that financial pretraining correlates with weaker speaker-level reasoning than comparable general models suggests that training objectives incorporating position context may be more productive than domain-specific continued pretraining—an hypothesis the benchmark itself does not test.