PolBench: Dual Benchmark Evaluation

• PolBench is a dual-purpose benchmark that assesses community note generation on U.S. political tweets and LLM forecasting/trading on prediction market data.
• For the Community Notes domain, it employs hybrid filtering and gap taxonomy to improve evaluation of moderation under cold-start conditions with high recall and precision.
• In the prediction-market context, PolyBench synchronizes order book snapshots with news metadata to enable LLM-based forecasting and simulated trading performance.

Searching arXiv for the two benchmark papers and closely related benchmark context. PolBench denotes two distinct 2026 benchmarks rather than a single canonical resource. In "GitSearch: Enhancing Community Notes Generation with Gap-Informed Targeted Search" (Singh et al., 9 Feb 2026), PolBench is a benchmark for community-notes generation and evaluation in U.S. political discourse, built from real X Community Notes and English-language U.S. political tweets spanning 2021–2025. In "PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Data" (Cheng et al., 3 Apr 2026), “PolBench” is an alias for PolyBench, a multimodal benchmark derived from Polymarket for evaluating LLM forecasting and trading under timestamp-locked market and news conditions. The shared label is therefore cross-domain: one benchmark studies community-based moderation and evidence-grounded note synthesis, while the other studies forecasting and execution under market microstructure constraints.

1. Terminological scope and disambiguation

The label “PolBench” is used in two non-equivalent senses in the 2026 literature. One refers to a Community Notes benchmark centered on U.S. political tweets; the other refers to PolyBench, a prediction-market benchmark whose executive summary explicitly states that “PolBench” is an alias while the preferred name in the paper is PolyBench (Singh et al., 9 Feb 2026, Cheng et al., 3 Apr 2026).

Label in use	Domain and task	Scale
PolBench	Community Notes generation and evaluation in U.S. political discourse	78,698 tweets; 169,992 notes
PolBench / PolyBench	LLM forecasting and trading on live prediction market data	38,666 binary markets; 4,997 events; 36,165 predictions

This naming overlap is methodologically consequential. In the first case, the benchmark is evaluation-first and anchored to platform-native moderation outcomes such as HELPFUL, NOT_HELPFUL, and NEEDS_MORE_RATINGS. In the second, the benchmark is contamination-proof and financially grounded, coupling historical CLOB state with aligned news and converting model outputs into simulated trades. A plausible implication is that citation-level disambiguation is necessary whenever “PolBench” is invoked in technical discussion.

2. PolBench as a Community Notes benchmark

In the GitSearch paper, PolBench was created to evaluate Community Notes generation at scale in fast-moving, contentious U.S. political discourse on X, precisely where community-based moderation is structurally challenged (Singh et al., 9 Feb 2026). The benchmark targets English-language U.S. political tweets from 2021-01-05 to 2025-09-06, with Community Notes from 2021-01-28 to 2025-09-06. It contains 78,698 unique English-language tweets and 169,992 attached notes. Community Notes are sourced from the X Community Notes public archive through 2025-09-07, while tweet text and metadata are retrieved via the X API for tweet IDs referenced by notes.

The benchmark is explicitly motivated by three conditions. First, it addresses cold-start scenarios: about 92% of notes are stuck in NEEDS_MORE_RATINGS, leaving both coverage and judgment sparse. Second, it centers missing context and information gaps, including timelines, baselines, definitions, and source verification. Third, it preserves community moderation realism by pairing tweets with observed Community Notes and metadata including status, votes, and timing. PolBench is therefore not synthetic; each entry is a real $(\text{tweet}, \text{note})$ association with time-ordered, user-generated explanations, corrections, or context.

Its inclusion pipeline uses a note-level U.S.-politics filter before tweet retrieval: a zero-shot MNLI classifier for U.S.-politics relevance, curated U.S. political keyword tagging, and retention of the union to improve recall without sacrificing precision. The reported hybrid method macro-F1 is 0.892, and U.S.-politics recall improves to 0.710 versus 0.428 for MNLI and 0.524 for keywords alone. Further filters exclude tweets whose associated notes indicate essential context depends on images or videos, exclude non-English tweets, and exclude pre-2021 tweets due to sparse Community Notes coverage.

Descriptively, the corpus is strongly status-imbalanced: Helpful 3.57%, Not Helpful 4.19%, and NMR 92.25%. Notes are short and unevenly sourced, with average length 28 words; 77.06% include external sources, implying 22.93% without citations. The benchmark includes 4,288 unique contributors, with median 1 note per contributor (IQR 1–3) and max 1,355. It includes 19,537 unique tweet authors, with median follower count approximately 22,423 and median engagement (likes+RTs) approximately 6,216. Multi-note coverage is common: 51% of tweets have multiple notes. Temporal dynamics are also explicit, with median lag tweet $\rightarrow$ first note approximately 5h 39m and note $\rightarrow$ status resolution approximately 5h 32m. Deleted tweets are retained if captured earlier; 12.10% of tweets in the corpus were later deleted.

The benchmark’s primary label is note status. CURRENTLY_RATED_HELPFUL notes serve as ground truth for many evaluations, while NOT_HELPFUL and NMR support analysis of partially aligned or incomplete community efforts. Helpfulness voting is available at the note level, and a derived signal used for summarization baselines is

$H_{\text{ratio}} = \frac{N_H}{N_H + N_{NH}},$

where $N_H$ and $N_{NH}$ are helpful and not-helpful votes, respectively.

PolBench also operationalizes a six-class gap taxonomy: unsubstantiated claim, contradiction, vague reference, missing context, source verification, and missing coverage. In the GitSearch framework, these gap categories are first-class signals for diagnosis, targeted retrieval, and downstream evaluation.

3. PolyBench as the prediction-market sense of “PolBench”

In the PolyBench paper, “PolBench” refers to a multimodal benchmark that evaluates LLM forecasting and trading performance on live prediction market data from Polymarket (Cheng et al., 3 Apr 2026). The benchmark consists of point-in-time cross-sections that synchronize a Central Limit Order Book snapshot with contemporaneous news and event metadata. The released dataset covers 38,666 binary markets across 4,997 events, yielding 36,165 model predictions from seven LLMs over snapshots collected February 6–12, 2026. The latest recorded market resolution date is February 21, 2026.

Each cross-section is a State Snapshot anchored to an exact historical timestamp. Inputs include event metadata from Polymarket’s Gamma API, such as titles, token identifiers, trading volumes, and official resolution criteria; precise CLOB state, including bid-ask spreads, midpoint, depth, and volume imbalances; and conditionally prefetched exogenous news aligned to the snapshot time. Coverage of aligned news is 91.6% of snapshots. The benchmark emphasizes strict temporal discipline: the LLM sees only information available at the snapshot timestamp, while outcomes are matched later at the market’s resolution time $\tau_i$ with $o_{ij} \in \{0,1\}$.

The task formulation requires each model to output a directional judgment and a confidence score $c \in [0,1]$. A BUY decision is permissible only when $c > 0.6$; otherwise the model should skip, and skips are allowed and not penalized. Prompts inject a timestamp-locked “Current Date,” event title and description, official resolution criteria, CLOB state, and aligned news if available. The paper summarizes system rules as Resolution Rule Primacy, Value Identification, Market Microstructure Analysis, Evidence-Backed Conviction, and Temporal Awareness.

The output schema is a structured JSON object with six required fields: decision, outcome, strategy, confidence, est_resolution_date, and reasoning. Strategy tags include examples such as value_bet, news_catalyst, arbitrage, and stable_yield. Confidence maps linearly to position size for base lot size $\rightarrow$0:

$\rightarrow$1

Execution is simulated as market-sweep BUY orders against the historical CLOB, absorbing liquidity from the best ask upward and explicitly capturing slippage. Invested capital is $\rightarrow$2, acquired shares are $\rightarrow$3, and top-of-book depth is recorded up to five levels. The simulator assumes zero latency at the snapshot timestamp, models no fees or rebates, and holds positions to market resolution with no early exit logic.

4. Evaluation frameworks and metrics

The two benchmarks differ sharply in what they regard as valid evidence of system quality. PolBench for Community Notes evaluates whether a system can generate a helpful, platform-compliant note for a given U.S. political tweet under cold-start and warm-start conditions, while aligning with Community Notes policy on neutrality, concision, and evidence (Singh et al., 9 Feb 2026). Its reported metrics include coverage,

$\rightarrow$4

pairwise win rate,

$\rightarrow$5

human helpfulness aggregated as a mean,

$\rightarrow$6

similarity metrics against human CURRENTLY_RATED_HELPFUL notes using ROUGE-L and BERTScore, URL Recall

$\rightarrow$7

LLM-as-a-Judge scores on Functional Errors, Claim Alignment, Fact Alignment, Completeness, and Helpfulness, and operational statistics such as number of URLs, character count under a 280-character constraint, and number of notes successfully generated.

Human judging in that benchmark uses a rubric spanning Factuality (F1–F4: accuracy, source existence, support, credibility), Completeness (C1–C2: relevance to the tweet, context coverage), and overall Helpfulness on a 1–5 scale. The benchmark’s evaluation set is the most recent 10% of tweets that have at least one CURRENTLY_RATED_HELPFUL note: 488 tweets from May 1 to Sept 6, 2025. Of these, only 265 have at least one NMR note, which directly exposes the cold-start limitation of summarization-only baselines.

PolyBench instead evaluates executed predictions under realistic order-book simulation and settlement (Cheng et al., 3 Apr 2026). For non-skipped executed predictions, directional accuracy is

$\rightarrow$8

Per-trade net profit is defined as

$\rightarrow$9

Confidence-Weighted Return is conceptually total net profit over total invested capital,

$\rightarrow$0

while APY uses simple linear annualization,

$\rightarrow$1

and Sharpe is defined as

$\rightarrow$2

The paper also describes Temporal Resolution Error as mean absolute error between est_resolution_date and the true resolution date. Proper scoring rules such as Brier are explicitly avoided because the benchmark does not assume reliable, unbiased continuous probability references in volatile decentralized markets.

5. Empirical findings and methodological lessons

On the Community Notes benchmark, GitSearch is evaluated against Supernotes-Lite and web-search-enabled LLM agents such as GPT-5-nano, Gemini-2.5-Flash, Grok-4, and Sonar-Deep-Research (Singh et al., 9 Feb 2026). Using the 488-tweet test set, GitSearch reaches 99% coverage (484/488), nearly doubling Supernotes-Lite coverage of 54% (265/488). Against the same model run as a web agent, GitSearch improves Claim Alignment from 3.315 to 3.744, Completeness from 2.388 to 2.744, and Helpfulness from 2.772 to 3.114, all with $\rightarrow$3. In human evaluation over $\rightarrow$4 cases with a Gemini-2.5-Flash backbone, GitSearch achieves a 69% win rate versus human helpful notes, Helpfulness 3.87 versus 3.36, Completeness (context coverage C2) 0.87 versus 0.69, and Factuality F1 0.99 versus 0.94. It also wins 59% versus a web agent using the same base model. The paper interprets these results as evidence that gap-driven retrieval balances scale and quality better than summarization alone or generic one-stage web search.

The benchmark’s error analysis is also informative. The easiest gap types are contradictions and unsubstantiated claims, with Claim Alignment often above 3.8 and Helpfulness approximately 3.2–3.5. The hardest are vague references, with Claim Alignment approximately 3.27, Fact Alignment approximately 1.91, and Completeness approximately 2.27, and missing context or missing coverage, with Fact Alignment approximately 2.2–2.3 and Completeness approximately 2.2–2.3. An ablation shows that adding existing NMR notes as context improves GitSearch for Gemini-2.5-Flash by CA +0.322, FA +0.275, C +0.255, and H +0.225, all with $\rightarrow$5.

On PolyBench, seven LLMs are evaluated under identical timestamp-locked market states: Gemini-3-Flash, MiMo-V2-Flash, Grok-4.1-Fast, GPT-OSS-120B, DeepSeek-V3.2, Trinity-Large, and MiniMax-M2.5 (Cheng et al., 3 Apr 2026). At base lot size %%%%26$\rightarrow$027%%%%L$\rightarrow$810 to \$\rightarrow$9500, while Gemini-3-Flash decays earlier.

Several methodological lessons follow directly from these findings. A common misconception in the Community Notes setting is that high ROUGE-L or BERTScore is sufficient; however, Supernotes-Lite achieves higher surface similarity only when NMR notes exist, which sharply limits coverage. A common misconception in the PolyBench setting is that high directional accuracy or high declared confidence is equivalent to trading viability; however, only two of seven models produce positive financial returns, and the paper reports uniformly high average declared confidence of approximately 0.81–0.87. In both cases, the benchmarked outcome is not mere textual plausibility but performance under an operational constraint: platform-native moderation for one, execution and settlement for the other.

6. Biases, limitations, ethics, and reproducibility

The Community Notes PolBench has several explicit limitations (Singh et al., 9 Feb 2026). It is topic-skewed toward U.S. political discourse and is not globally representative. It exhibits selection bias because only tweets that drew Community Notes attention are included. With 92% NMR notes, status labels reflect uncertain human consensus and therefore inject noise into ground truth. Source distribution includes popular outlets and platform links such as x.com, Wikipedia, AP News, and Reuters, which may encode source preferences and partisan perceptions. The authors mitigate some of these issues through high-recall hybrid filtering, quality checks, and analysis by gap type. The data are derived from public sources, the paper states exemption from institutional review, and the authors caution against fully autonomous deployment, advocating assistive, human-in-the-loop use. Reproducibility is supported through publicly available code and data, except that which contains PII.

PolyBench states a different limitation profile (Cheng et al., 3 Apr 2026). Order-book capture is limited to the top five levels, so deeper liquidity and hidden orders are not modeled. The simulator omits transaction fees, rebates, and on-chain execution frictions; it assumes zero latency; and it uses long-only BUY simulation held to resolution, without shorting, No-share execution, dynamic hedging, or early exit logic. News coverage, although high at 91.6%, is incomplete. APY is simple linear annualization per trade and can be extreme for short-dated resolutions. The paper positions the benchmark for evaluation only, not direct deployment, and frames strict timestamp locking and simulated execution as safeguards against leakage and overclaiming.

Taken together, the two uses of “PolBench” exemplify a broader methodological shift toward evaluation under endogenous constraints. One benchmark embeds models inside community-based fact-checking with vote histories, note statuses, and platform rules; the other embeds models inside a market simulator with CLOB depth, slippage, and realized returns. The shared name is accidental rather than conceptual, but the pair is informative precisely because both benchmarks reject decontextualized evaluation in favor of temporally and operationally grounded testing.