PolitiFact-Hidden Benchmark Overview

• PolitiFact-Hidden is a family of benchmarks derived from PolitiFact data that evaluates retrieval, ordinal truth prediction, and omission-aware fact verification.
• The benchmarks employ methodologies like evidence alignment, intent inference, and causal analysis to distinguish between presented and hidden evidence with notable performance gains.
• The evaluation setups highlight challenges such as paraphrase mismatches, leakage control, and domain-specific limitations, underscoring the need for refined verification systems.

PolitiFact-Hidden denotes more than one PolitiFact-derived evaluation setting in the literature. In the older retrieval literature, it can refer to a held-out PolitiFact test partition for detecting whether a new claim has already been fact-checked. In later omission-aware fact verification, it denotes a benchmark built from PolitiFact articles for determining whether a claim is literally correct yet misleading because critical context has been omitted. A related but distinct precursor is the coarse-to-fine truthfulness corpus derived from PolitiFact’s six-level “Truth-O-Meter,” where a hidden split was recommended but not released. As a result, the name designates a small family of held-out and omission-sensitive tasks rather than a single invariant dataset artifact (Shaar et al., 2020, Wu, 2021, Tang et al., 1 Aug 2025, Tang et al., 21 Apr 2026).

1. Terminological scope and lineage

The term acquired its present meaning through successive PolitiFact-based research programs. The retrieval-oriented work "That is a Known Lie: Detecting Previously Fact-Checked Claims" does not use the term “PolitiFact-Hidden,” but its closest analogue is a held-out PolitiFact test split of unseen Input–VerClaim pairs. "Rating Facts under Coarse-to-fine Regimes" likewise does not release a hidden test set, but it provides explicit guidance for constructing one from PolitiFact truthfulness labels. The benchmark named PolitiFact-Hidden is introduced later in "The Missing Parts: Augmenting Fact Verification with Half-Truth Detection," and "Debating the Unspoken: Role-Anchored Multi-Agent Reasoning for Half-Truth Detection" adopts that benchmark for omission-aware, retrieval-grounded evaluation (Shaar et al., 2020, Wu, 2021, Tang et al., 1 Aug 2025, Tang et al., 21 Apr 2026).

Work	Meaning of “PolitiFact-Hidden”	Scale
(Shaar et al., 2020)	held-out PolitiFact test split for Input–VerClaim retrieval	154 test pairs; 16,636 fact-check pages
(Wu, 2021)	recommended hidden split for coarse-to-fine truthfulness prediction	source corpus of approximately 24K manually rated statements
(Tang et al., 1 Aug 2025)	omission-aware benchmark with evidence alignment and inferred intent	14,994 claims; 11,994/1,000/2,000 train/dev/test
(Tang et al., 21 Apr 2026)	evaluation benchmark for RADAR under gold and retrieved evidence	~15k claims; 41,952-document corpus

This terminological dispersion matters methodologically. The retrieval split asks whether a new claim matches a previously checked canonical claim. The coarse-to-fine setup asks how well models discriminate ordered truthfulness categories. The omission-aware benchmark asks whether hidden context changes the plausibility of a claim’s implied intent. Treating these as interchangeable obscures major differences in supervision, evidence structure, and evaluation targets.

2. Label structures and task semantics

The coarse-to-fine truthfulness formulation begins from PolitiFact’s ordered six-class taxonomy: True means “The statement is accurate and there is nothing significant missing”; Mostly true means “The statement is accurate but needs clarification or additional information”; Half true means “The statement is partially accurate but leaves out important details or takes things out of context”; Mostly false means “The statement contains an element of truth but ignores critical facts that would give a different impression”; False means “The statement is not accurate”; Pants on fire means “The statement is not accurate and makes a ridiculous claim.” These labels are treated as ordinal, and the paper defines three regimes: fine-grained FPF with all six labels in natural order, coarse-grained CPF with $[True, Mostly\ true]$, $[Half\ true, Mostly\ false]$, and $[False, Pants\ on\ fire]$, and binary BPF with only $[True, Mostly\ true]$ versus $[False, Pants\ on\ fire]$ (Wu, 2021).

The omission-aware PolitiFact-Hidden benchmark collapses PolitiFact’s six verdicts into three mutually exclusive labels: True $\leftarrow$ True, Half-True $\leftarrow$ Mostly True and Half True, and False $\leftarrow$ Mostly False, False, and Pants on Fire. Its formalization is different from ordinal truthfulness prediction. Given a claim $C$ and retrieved evidence sentences $E=\{e_1,e_2,\dots,e_n\}$, the benchmark distinguishes Presented Evidence (PE), which is explicitly stated or clearly implied in $[Half\ true, Mostly\ false]$0, from Hidden Evidence (HE), which is relevant but not mentioned. It further defines Intent as the implied conclusion a claim aims to convey and Critical Hidden Evidence (CHE) as a subset of hidden evidence, formally $[Half\ true, Mostly\ false]$1, that would significantly affect the plausibility of that intent if revealed (Tang et al., 1 Aug 2025).

A common conflation is to treat the later three-way benchmark as merely a relabeling of the older ordinal dataset. The published formulations do not support that interpretation. The earlier work studies graded class similarity among truthfulness labels, whereas the later benchmark operationalizes omission-based misinformation through PE/HE alignment, intent inference, and CHE-oriented causal analysis. This suggests that the two lines share PolitiFact provenance but differ at the level of task ontology.

3. Benchmark construction and annotation

The omission-aware PolitiFact-Hidden benchmark is built from PolitiFact fact-checking articles, each of which provides a concise checked claim, evidence paragraphs containing factual context, and a narrative ruling. The dataset size is 14,994 claims. A temporally disjoint test set of 2,000 claims from 2020–2025 was collected, and claims overlapping in date with train were removed from the training pool. The split distribution is Train 11,994, Dev 1,000, and Test 2,000, with per-split class counts of Train: True 1,352, Half-True 4,564, False 6,078; Dev: True 64, Half-True 195, False 741; Test: True 93, Half-True 406, False 1,501 (Tang et al., 1 Aug 2025).

Split	True / Half-True / False	Total
Train	1,352 / 4,564 / 6,078	11,994
Dev	64 / 195 / 741	1,000
Test	93 / 406 / 1,501	2,000

Construction is explicitly leakage-aware. PolitiFact articles are separated into evidence paragraphs and ruling paragraphs, and ruling content is excluded from model inputs using structural cues such as “Our Ruling.” Evidence is segmented at the sentence level, after which each sentence is classified as PE or HE relative to the claim. The annotation pipeline is semi-automated: an LLM-based relevance check filters irrelevant sentences; a presentation check determines whether relevant content is explicitly or implicitly stated in the claim; and cosine similarity with XLM-RoBERTa embeddings refines borderline cases and mitigates hallucinations. Manual inspection on 50 samples showed 88% agreement between LLM predictions and human judgments (Tang et al., 1 Aug 2025).

Intent is also annotated. The process first performs “Ruling Enhancement,” then extracts the intended conclusion using instruction-tuned prompting, and finally applies quality filtering under four criteria: Plausibility 96.0%, Implicity 96.0%, Sufficiency 91.0%, and Readability 86.0% in LLM–human agreement. The resulting intent spans are designed to be plausible, implicit rather than paraphrastic, sufficient, and readable. A plausible implication is that the benchmark’s novelty lies not only in its three-way labels but in the availability of sentence-level evidence structure and an explicit semantic target for omission analysis.

4. Hidden splits, temporal holdout, and leakage control

Before the 2025 omission-aware benchmark, the 2021 coarse-to-fine study used balanced PolitiFact datasets but neither used nor released a hidden test set. Its proposed hidden construction is nevertheless specific: start from the full six-class set, stratify by the ordered labels, optionally map to 3-class and 2-class regimes after the split, reserve the most recent 20% as a temporal holdout, prevent near-duplicate or paraphrastic leakage with approximate text deduplication such as MinHash, token-level Jaccard, or SimHash at a threshold such as $[Half\ true, Mostly\ false]$2, and, where metadata are available, prevent exact repeated quotes from the same speaker or event from crossing boundaries. Suggested FPF sizes are Train $[Half\ true, Mostly\ false]$3, Public Dev $[Half\ true, Mostly\ false]$4, and Hidden Test $[Half\ true, Mostly\ false]$5, with analogous mappings for CPF and BPF (Wu, 2021).

The omission-aware PolitiFact-Hidden benchmark instantiates some of the same principles differently. Its test set is temporally disjoint, taken from 2020–2025, and ruling content is removed from model inputs to avoid direct verdict leakage. This convergence between the recommendation-oriented 2021 setup and the later 2025 benchmark indicates that temporal and structural separation became central design principles for PolitiFact-derived hidden evaluation (Wu, 2021, Tang et al., 1 Aug 2025).

The retrieval-oriented held-out split in the 2020 claim-matching literature follows another notion of hiddenness. It contains 154 unseen PolitiFact Input–VerClaim pairs drawn from 768 total pairs, with 614 pairs used for training. Evaluation is conducted by retrieving the correct match from the full PolitiFact database of 16,636 entries. There is no explicit timestamp-based split; “hidden” here means held-out claims unseen during training, not omission-bearing evidence withheld within an article (Shaar et al., 2020).

5. Modeling paradigms

Three modeling paradigms dominate the literature attached to PolitiFact-Hidden. The coarse-to-fine truthfulness work fine-tunes BERT-base end to end and compares a plain classification head over the final-layer $[Half\ true, Mostly\ false]$6 representation with two augmented heads: BERT-BiLSTM, which feeds the full final-layer token sequence into a BiLSTM of hidden size 768 followed by max-over-time pooling, and BERT-CNN, which applies a 1D CNN with filter region sizes $[Half\ true, Mostly\ false]$7, 768 feature maps in total, max-over-time pooling, dropout 0.5, and a classifier. Training uses Adam, learning rate $[Half\ true, Mostly\ false]$8, batch size 32, and 4 epochs; dropout is 0.1 in BERT and 0.5 in the BiLSTM/CNN heads. No custom ordinal loss is specified; standard softmax and cross-entropy are implied, although mean absolute error is recommended as an ordinally sensitive metric if labels are integer-mapped (Wu, 2021).

TRACER reframes the task as omission-aware re-assessment. Its pipeline has four stages: Evidence Alignment, Intent Generation, Causality Analysis, and Final Re-Assessment. Evidence Alignment takes pairs $[Half\ true, Mostly\ false]$9 and uses RoBERTa-large with a classification head to predict PE versus HE; training is 5 epochs, batch size 8, learning rate $[False, Pants\ on\ fire]$0, with context-aware batch sampling that groups sequential evidence sentences. Intent Generation uses a fine-tuned GPT-4o-mini via OpenAI API, trained for 3 epochs with batch size 4. Causality Analysis generates assumptions $[False, Pants\ on\ fire]$1 needed for intent $[False, Pants\ on\ fire]$2 to hold and evaluates them with counterfactual prompting, asking “given $[False, Pants\ on\ fire]$3, does the intent $[False, Pants\ on\ fire]$4 still hold?” and, more specifically, “Evaluate $[False, Pants\ on\ fire]$5.” Hidden sentences supporting or contradicting validated assumptions are then retrieved as CHE using semantic similarity and an NLI check. Final Re-Assessment preserves the original label if no CHE is found and otherwise relabels the claim as Half-True, or False if claim-level facts are directly contradicted (Tang et al., 1 Aug 2025).

RADAR extends omission-aware verification to realistic noisy retrieval. It retrieves the top-20 passages per claim from a 41,952-document corpus using the dense retriever bge-base-en-v1.5 and assigns three complementary roles over a shared evidence pool: a Politician that builds the strongest evidence-backed narrative supporting the claim, a Scientist that probes for missing or selectively presented evidence, and a Judge that moderates and issues the final verdict. At the end of round $[False, Pants\ on\ fire]$6, the Judge computes a stop margin $[False, Pants\ on\ fire]$7 and a veracity confidence $[False, Pants\ on\ fire]$8. Early termination occurs only when $[False, Pants\ on\ fire]$9 and $[True, Mostly\ true]$0. Debates run for at most three rounds, thresholds are tuned on development data via offline grid search over cached logits, and the system is unsupervised, with no training objective or fine-tuning loss (Tang et al., 21 Apr 2026).

The earlier held-out retrieval setting uses a multi-stage ranking pipeline rather than veracity classification. BM25 indexes are built over Title, VerClaim, and Body, and the top candidates are re-ranked with a pairwise RankSVM with RBF kernel using BM25 scores, Sentence-BERT similarities for claim–VerClaim and claim–Title, top-$[True, Mostly\ true]$1 claim–Body sentence similarities, and reciprocal-rank features. This design targets previously fact-checked claim detection rather than omission reasoning, even though both lines operate over PolitiFact-derived data (Shaar et al., 2020).

6. Empirical behavior, failure modes, and limitations

In the coarse-to-fine truthfulness setting, performance improves as the label space becomes coarser. Weighted-average F1 over five random seeds is $[True, Mostly\ true]$2 for BERT on FPF, $[True, Mostly\ true]$3 on CPF, and $[True, Mostly\ true]$4 on BPF; BERT-CNN yields $[True, Mostly\ true]$5, $[True, Mostly\ true]$6, and $[True, Mostly\ true]$7, respectively. The reported confusion structure is strongly ordinal: misclassifications concentrate on adjacent classes, the probability of a predicted label decreases as ordinal distance from the gold label increases, and polarized extremes are easier than middle or neutral labels. This supports the paper’s central claim that class similarity is itself a source of error (Wu, 2021).

On the omission-aware benchmark, TRACER’s Evidence Alignment module improves over a RoBERTa-large baseline from Accuracy 93.2 and F1 90.3 to Accuracy 94.0 and F1 91.6. Intent generation improves from ROUGE-L 37.7, BLEU 6.1, and BERTScore 91.2 with few-shot prompting to ROUGE-L 46.2, BLEU 8.0, and BERTScore 91.5 with fine-tuned GPT-4o-mini. In end-to-end fact verification, CoT on test moves from Accuracy 76.3 and macro-F1 64.3 to 78.5 and 68.0 with TRACER re-assessment; HiSS moves from 78.3 and 59.4 to 81.9 and 65.7. The benchmark’s central omission-sensitive metric, F1 on the Half-True class, rises from 52.8 to 60.2 for CoT and from 44.4 to 60.5 for HiSS, with the latter representing a +16.1 test gain. The ablation study attributes much of the gain to intent modeling, assumption inference, and causality filtering (Tang et al., 1 Aug 2025).

RADAR reports additional gains under both full-evidence and noisy-retrieval conditions. With GPT-4o-mini in the Full Evidence setting, RADAR_multi reaches Accuracy 83.6 and Macro-F1 70.8, compared with 78.5 and 68.0 for CoT + RA and 77.8 and 65.7 for HiSS + RA. Under Retrieved Evidence, RADAR_multi reaches 77.7 Accuracy and 63.3 Macro-F1, versus 60.3 and 46.9 for FIRE, 63.0 and 50.9 for D2D, and 58.4 and 51.0 for RADAR_single. On LLaMA3-8B-Instruct, early stopping preserves or slightly improves performance while reducing average tokens from 1834.7 for fixed three rounds to 1551.4; on Qwen2.5-7B-Instruct, early-stop RADAR reports 76.7 Accuracy, 63.1 Macro-F1, and 1723.7 average tokens (Tang et al., 21 Apr 2026).

The held-out retrieval split exhibits a different empirical profile. BM25 over Body is the strongest single-stage baseline with MRR 0.565 and MAP@1 0.484, while RankSVM re-ranking of the top-100 IR:Body candidates raises MRR to 0.608 and MAP@1 to 0.531. Error analysis emphasizes paraphrase and normalization mismatch: in a manual analysis of 100 PolitiFact Input–VerClaim pairs, 48% are Type-2 non-trivial paraphrastic mappings; only 27% exceed TF.IDF-weighted cosine similarity 0.25, and 8% exceed 0.75 (Shaar et al., 2020).

Limitations are correspondingly heterogeneous. The omission-aware benchmark is political in scope, and its generalization to health or finance is explicitly left open. It assumes that claims have coherent inferable intent, while real claims may be vague or multi-intent. LLM-assisted annotations may encode model biases despite agreement checks and similarity filtering. RADAR adds another layer of trade-off: multi-agent scrutiny can over-penalize fully true claims by nudging them toward Half-True, and performance remains dependent on retrieval quality. In RADAR’s manual review of 50 misclassified cases, 56% involve claim constraint violations, 30% semantic framing instability, 10% causal attribution errors, and 4% other failures. Taken together, these results suggest that PolitiFact-Hidden has become a focal testbed not merely for fact verification, but for evaluating whether systems can reason about completeness, implied intent, and the consequences of what is left unsaid (Tang et al., 1 Aug 2025, Tang et al., 21 Apr 2026).