WildClaims: Implicit Facts in LLM Conversations

• WildClaims is a dataset focused on implicit information access in LLM conversations, identifying check-worthy factual claims beyond explicit queries.
• It employs two LLM-based extractors (F_Huo and F_Song) with conversation history to decontextualize claims and assess verification needs based on varied recall and precision metrics.
• The findings imply that fact-checking workflows should be integrated into systems even during non-informational interactions like creative writing, editing, and role playing.

WildClaims is a dataset and problem formulation for studying implicit information access in LLM conversations. Derived from WildChat, it consists of 121,905 extracted factual claims from 7,587 system utterances in 3,000 conversations, each annotated for check-worthiness to estimate when conversational outputs should trigger retrieval, verification, or fact-checking workflows (Joko et al., 22 Sep 2025). Its central contribution is conceptual as much as empirical: it treats factual transfer in dialogue not as a phenomenon confined to explicit question answering, but as something that also occurs in creative writing, editing, planning, role playing, and other interactions whose overt intent is non-informational. In that sense, WildClaims sits at the intersection of conversational IR, factual claim extraction, check-worthiness detection, and conversational factuality analysis.

1. Conceptual scope and definition

WildClaims was introduced to answer the question “What do real-world information access conversations with LLMs actually look like?” (Joko et al., 22 Sep 2025). The work distinguishes two regimes. Explicit information access corresponds to the conventional IR setting in which a participant states a clear information need and the system responds with retrieved or generated content. Implicit information access arises when a system makes factual assertions during interactions whose primary purpose is not overtly informational, such as creative writing or drafting. The dataset operationalizes the latter regime.

Within this framing, a factual claim is a self-contained statement that asserts something about the world and is potentially verifiable against external sources. A check-worthy factual claim is a factual assertion that is worth checking and deemed necessary to be verified through external sources (Joko et al., 22 Sep 2025). The point is not merely that the statement has truth conditions, but that uncritical reliance on it could be consequential. The paper’s examples include a fabricated legal citation, claims about tool capabilities, and historically questionable statements such as a 14th-century “Bushido code” (Joko et al., 22 Sep 2025).

The authors also propose a clarified definition of an information access conversation (in the wild): a conversation in which information needs of participants are satisfied by exchanging verified information, using all necessary resources for reliable knowledge transfer, regardless of whether the user’s original goal was implicitly or explicitly information-seeking (Joko et al., 22 Sep 2025). This directly contests a common narrowing of conversational IR to overt search-like dialogues. In WildClaims, the relevant trigger for information access is not the user’s surface intent alone, but the system’s production of check-worthy assertions.

2. Corpus construction and annotation pipeline

WildClaims is a derived resource built from WildChat, described as a corpus of over 1M real user–ChatGPT logs (Joko et al., 22 Sep 2025). The construction pipeline first retains English conversations, yielding 478,498 English conversations, then filters out math/coding-focused interactions using GPT-4.1-mini with a prompt from Zhang et al. (2024); 79.6% of the English subset is non-math/non-coding (Joko et al., 22 Sep 2025). From this filtered pool, 3,000 conversations are sampled.

The resulting sample contains 15,174 utterances in total, including 7,587 system utterances, and conversations are mostly short with an average of 2.52 utterances per conversation (Joko et al., 22 Sep 2025). System outputs are comparatively long, with an average of 219 words, a property that matters because long-form responses often contain many atomic factual units rather than a single claim (Joko et al., 22 Sep 2025). User intents were categorized via GPT-4.1 following the WildBench taxonomy: information seeking 33.5%, creative writing 18.8%, editing 15.8%, and reasoning 9.9%, with role playing, planning, brainstorming, advice seeking, data analysis, and other categories comprising the remainder (Joko et al., 22 Sep 2025).

Claim extraction is performed only on system utterances. Two LLM-based extractors are used. $F_{Huo}$, adapted from Huo et al. (2023), extracts self-contained factual claims. $F_{Song}$, adapted from Song et al. (2024, VeriScore), targets verifiable claims (Joko et al., 22 Sep 2025). Both use GPT-4.1 and are modified to incorporate conversation history so that pronouns and ellipses can be resolved during decontextualization. Formally, given a system utterance $u_s$ and history $h$, extraction returns a set

$\mathbf{A} = \{a_1, a_2, \dots, a_n\} = F(u_s, h).$

Check-worthiness is then assigned using two classifiers: $CW_{Hassan}$, reflecting ClaimBuster-style prompting, and $CW_{Majer}$, based on optimized prompts designed to better capture annotation guidelines (Joko et al., 22 Sep 2025). Both are implemented with GPT-4.1 and use conversation context. The paper further defines: $$CW_{\text{Intersection}}(a,h) = CW_{Majer}(a,h) \land CW_{Hassan}(a,h),$$

$$CW_{\text{Union}}(a,h) = CW_{Majer}(a,h) \lor CW_{Hassan}(a,h).$$

Manual validation covers 200 claims: 100 sampled from $F_{Huo}$ and 100 from $F_{Song}$0, each from a distinct conversation (Joko et al., 22 Sep 2025). Two authors independently annotate the claims as CW/not CW, with a third author serving as tie-breaker. Reported Cohen’s $F_{Song}$1 is 0.672 for $F_{Song}$2 and 0.580 for $F_{Song}$3, indicating that the notion of conversational check-worthiness is non-trivial but sufficiently stable for empirical study (Joko et al., 22 Sep 2025).

3. Formalization and empirical properties

WildClaims formalizes claim-level check-worthiness as a binary decision over an extracted factual claim $F_{Song}$4 given discourse history $F_{Song}$5: $F_{Song}$6 An utterance is check-worthy if it contains at least one extracted claim with $F_{Song}$7, and a conversation is check-worthy if it contains at least one such utterance (Joko et al., 22 Sep 2025). This moves the unit of analysis from document-level or utterance-level heuristics to explicit claims embedded inside responses.

The two extractors differ sharply in aggressiveness. $F_{Song}$8 produces 31,108 claims, or 4.1 claims per utterance and 10.4 claims per conversation on average; 45.6% of utterances and 45.1% of conversations contain at least one extracted fact under this method (Joko et al., 22 Sep 2025). $F_{Song}$9 produces 90,797 claims, or 12.0 claims per utterance and 30.3 claims per conversation; 72.1% of utterances and 79.0% of conversations contain at least one extracted fact (Joko et al., 22 Sep 2025). The difference is not incidental: it demonstrates that conversational factuality estimates are heavily conditioned on the granularity of extraction.

Human validation confirms that a large fraction of extracted claims are actually check-worthy. The aggregated gold labels yield 40% CW claims for $u_s$0 and 64% for $u_s$1 (Joko et al., 22 Sep 2025). The paper derives a conservative lower-bound estimate of the proportion of conversations with at least one check-worthy claim by multiplying $u_s$2, the human-estimated proportion of extracted claims that are check-worthy, by $u_s$3, the proportion of conversations containing at least one extracted claim. This produces 18% for $u_s$4 and 51% for $u_s$5 (Joko et al., 22 Sep 2025). Because conversations often contain many claims rather than at most one, these are explicitly characterized as lower bounds.

Among the automatic check-worthiness methods, $u_s$6 obtains the best F1 and the highest $u_s$7 across both extractors (Joko et al., 22 Sep 2025). For $u_s$8, it reaches P = 0.613, R = 0.950, F = 0.745, and $u_s$9. For $h$0, it reaches P = 0.769, R = 0.938, F = 0.845, and $h$1 (Joko et al., 22 Sep 2025). Using the strongest pipeline reported, $h$2, 91.8% of extracted claims are marked CW, 70.2% of system utterances contain at least one CW claim, 76.4% of conversations contain at least one CW claim, and the average is 11 CW claims per conversation (Joko et al., 22 Sep 2025).

A key empirical implication is that check-worthy assertions are not confined to explicitly informational tasks. With $h$3, creative writing still shows 57.7% of system utterances as check-worthy; with $h$4 the corresponding figure is 41.3% (Joko et al., 22 Sep 2025). This is the central evidence for the paper’s broader thesis: real-world LLM conversations often function as information transfer even when neither side is conducting canonical search.

4. Relation to claim span identification and multilingual claim extraction

WildClaims extracts decontextualized claims from full system utterances, but adjacent work shows that claim detection can also be framed as span identification. The Claim Span Identification (CSI) task is defined as extracting the specific part or parts of a text that “minimally represent a claim-like statement”, operationalized as token-level binary labeling over a post $h$5 with labels $h$6 indicating whether token $h$7 belongs to a claim span (Poddar et al., 2024). CSI differs from binary claim classification by providing exact span boundaries rather than a post-level claim/non-claim label, and is motivated as more challenging and more useful for downstream fact-checking, retrieval, argument mining, and explanation (Poddar et al., 2024).

The multilingual literature is especially relevant. X-CLAIM introduces roughly 7K real-world claims from social media in English, Hindi, Punjabi, Tamil, Telugu, and Bengali, framing CSI as sequence labeling with token-level IO tags; the paper reports that IO outperformed BIO, BEO, and BEIO encodings in its experiments (Mittal et al., 2023). It also reports that multilingual training is best overall relative to monolingual training, zero-shot transfer from English, and training on translated data, while prompted GPT-series models underperform the smaller encoder-only LLMs for low-resource languages (Mittal et al., 2023). For GPT-4, 10-shot in-context prompting improves F1 by ~2.78 points on average over zero-shot under the LANGUAGE prompt (Mittal et al., 2023).

The ICPR 2024 Competition on Multilingual Claim-Span Identification extends the span-level paradigm with HECSI, containing about 8K posts in English and about 8K posts in Hindi with human-marked claim spans (Poddar et al., 2024). The competition paper explicitly states that, for a system like WildClaims, CSI is the component that highlights the “wild” bit in a post and enables targeted verification workflows (Poddar et al., 2024).

This suggests that WildClaims and CSI address adjacent granularities of the same problem. WildClaims identifies self-contained, decontextualized factual assertions in long conversational outputs; CSI localizes minimal claim-bearing segments within a source text. A plausible implication is that a span-localization layer of the CSI type could refine WildClaims-style pipelines by marking exact token spans inside long system utterances before decontextualization, especially if the framework is extended beyond English.

5. Downstream claim organization: comparison, clustering, and repeated assertions

WildClaims focuses on extracting and labeling factual claims in conversation, but downstream fact-checking requires organization of those claims by relation and redundancy. ClaimDiff addresses one part of this problem by modeling how pairs of claims on the same contentious issue can strengthen or weaken one another, while also extracting the textual rationale responsible for that relation (Ko et al., 2022). The relation is directional and explicitly distinguished from canonical fact verification: the task assumes that, among trusted sources, one argument is not necessarily more true than the other, so the goal is comparison rather than truth assessment (Ko et al., 2022). This is important for conversational outputs because two system-generated claims may both be factually consistent while still framing an issue differently.

A second downstream task is claim clustering. MultiClaimNet introduces three multilingual cluster datasets spanning 86 languages, with the largest component containing 85.3K fact-checked claims written in 78 languages (Panchendrarajan et al., 28 Mar 2025). Claims are clustered when they discuss the same underlying fact, with clusters formed as connected components in a similarity graph built from pairwise claim matches (Panchendrarajan et al., 28 Mar 2025). The resource is motivated by the observation that fact-checking pipelines face massive redundancy: the same claim reappears across platforms, languages, and paraphrases. The paper reports that 50% of claims in a cluster are repeated within 1.6 days of the first occurrence and 75% within 29.3 days (Panchendrarajan et al., 28 Mar 2025).

The connection to WildClaims is direct at the systems level. A plausible implication is that extracted conversational claims can be routed into ClaimDiff-style relational analysis when multiple claims on a contentious issue coexist, and into MultiClaimNet-style clustering when the goal is to detect repetition, reuse verdicts, or prioritize verification at the cluster level rather than the claim-instance level. This would align WildClaims with the broader fact-checking workflow in which claim detection precedes retrieval, triage, and verdict propagation.

6. Extensions, limitations, and adjacent infrastructures

WildClaims is deliberately scoped: it studies English, non-coding/non-math conversations sampled from WildChat, and its manual validation set contains 200 claims (Joko et al., 22 Sep 2025). The resource therefore characterizes a substantial but bounded portion of LLM-mediated discourse. Another limitation is methodological sensitivity: the difference between $h$8 and $h$9 shows that the observed prevalence of conversational factuality depends strongly on the extractor’s recall-granularity trade-off (Joko et al., 22 Sep 2025). The work’s own presentation acknowledges this by reporting both conservative lower bounds and less conservative estimates.

Adjacent research indicates how the WildClaims agenda may broaden. HintsOfTruth defines multimodal checkworthiness detection over image–claim pairs and introduces a dataset with 27K real-world and synthetic image/claim pairs (Meer et al., 17 Feb 2025). Its experiments show that well-configured lightweight text-based encoders can perform comparably to multimodal models, but primarily by focusing on non-claim-like content; multimodal LLMs can be more accurate yet incur substantial computational cost, while multimodal models are more robust on synthetic data (Meer et al., 17 Feb 2025). This suggests that if WildClaims is extended to multimodal conversational settings, check-worthiness will likely need to be treated as a joint function of textual assertion and visual grounding rather than text alone.

A different adjacent direction concerns evidence-bearing claims processes rather than open-domain chat. SoverClaim combines Hyperledger Indy for self-sovereign identity with Storj for decentralized document storage in document-based claims workflows such as proof of residence or medical-image verification, reporting response times of under 750 ms (Gomes et al., 2024). WildClaims is not a decentralized identity system, but the juxtaposition is instructive: once conversational claims move from detection to evidence exchange, provenance, selective disclosure, storage, and deletion become operational concerns rather than purely linguistic ones.

In the current literature, WildClaims therefore occupies a specific but consequential niche. It is not a truth-label corpus, not a span-level sequence-tagging benchmark, and not a multilingual claim cluster dataset. Instead, it is a resource for quantifying how often LLM conversations themselves become sites of factual transfer that should trigger verification. Its principal significance lies in reframing conversational factuality as an information access problem even when the conversation does not outwardly resemble search (Joko et al., 22 Sep 2025).