AI Fact-Checking in the Wild: A Field Evaluation of LLM-Written Community Notes on X

Abstract: LLMs show promising capabilities for contextual fact-checking on social media: they can verify contested claims through deep research, synthesize evidence from multiple sources, and draft explanations at scale. However, prior work evaluates LLM fact-checking only in controlled settings using benchmarks or crowdworker judgments, leaving open how these systems perform in authentic platform environments. We present the first field evaluation of LLM-based fact-checking deployed on a live social media platform, testing performance directly through X Community Notes' AI writer feature over a three-month period. Our LLM writer, a multi-step pipeline that handles multimodal content (text, images, and videos), conducts web and platform-native search, and writes contextual notes, was deployed to write 1,614 notes on 1,597 tweets and compared against 1,332 human-written notes on the same tweets using 108,169 ratings from 42,521 raters. Direct comparison of note-level platform outcomes is complicated by differences in submission timing and rating exposure between LLM and human notes; we therefore pursue two complementary strategies: a rating-level analysis modeling individual rater evaluations, and a note-level analysis that equalizes rater exposure across note types. Rating-level analysis shows that LLM notes receive more positive ratings than human notes across raters with different political viewpoints, suggesting the potential for LLM-written notes to achieve the cross-partisan consensus. Note-level analysis confirms this advantage: among raters who evaluated all notes on the same post, LLM notes achieve significantly higher helpfulness scores. Our findings demonstrate that LLMs can contribute high-quality, broadly helpful fact-checking at scale, while highlighting that real-world evaluation requires careful attention to platform dynamics absent from controlled settings.

• The paper presents a field evaluation comparing LLM-authored fact-check notes to human ones using mixed-effects modeling and exposure normalization.
• The results reveal that LLM-generated notes achieve up to a 10-point higher helpfulness rating among centrist and left-leaning users.
• LLM notes are longer and cite more authoritative sources, indicating trade-offs between standardized fact-checking and platform-specific discourse.

Field Evaluation of LLM-Written Fact-Checks in Social Media: An In-Depth Analysis

Introduction and Context

This study presents a rigorous field evaluation of LLM-authored Community Notes for fact-checking within X (formerly Twitter), benchmarking them directly against human-authored notes on the live platform (2604.02592). Prior research has established that LLMs can generate explanations that compare favorably to human writing in controlled environments. However, such studies do not account for the complex sociotechnical platform dynamics that impact real-world perceptions of helpfulness and consensus, including rater heterogeneity, exposure differentials due to note timing, and the bridging requirements unique to X's Community Notes. This research is the first to study LLM fact-checking in situ, utilizing platform-native feedback from hundreds of thousands of organic user ratings across diverse topical and modal content.

Experimental Design and Methodology

The authors engineered a multi-step LLM writer, leveraging web and platform-native retrieval, multimodal (text, image, video) conditioning, and both evidence-based triage and quality filtering before note posting. This pipeline generated 1,614 notes across 1,597 tweets over a three-month period, compared to 1,332 human-written notes on the same tweets, and evaluated by 108,169 ratings from 42,521 unique raters.

To isolate intrinsic note quality from platform-external confounds, two analytical strategies are employed:

• Rating-level analysis: Mixed-effects modeling of over 100,000 individual ratings, capturing user ideology (as inferred from historical bridging patterns) to assess cross-partisan reception and LLM–human deltas independent of exposure.
• Note-level, equal-exposure analysis: Recomputing helpfulness and CRH/CRNH metrics using only raters who evaluated all notes per tweet, thus neutralizing rating-count penalties and rater selection effects.

Results: Ideology, Topic, and Modality Effects

Ideological Bridging Performance

LLM-authored notes received significantly higher rates of "helpful" ratings and lower "unhelpful" ratings across rater ideology buckets (left, center, right), with the largest margin of improvement for centrists and left-leaning raters.

Figure 1: Mean % helpful and % unhelpful ratings per note for LLM and human notes, stratified by rater ideology group (left, neutral, right), with 95% CIs.

Linear mixed-effects modeling quantified the LLM advantage as approximately a 10-percentage-point increase in helpfulness ratings among centrist raters. The advantage persists but attenuates for ideologically extreme (especially right-leaning) raters. The results are robust to multiple analytical specifications, supporting that LLM notes are not only accurate but more likely than human notes to achieve the cross-partisan bridging that the Community Notes system targets.

Topical and Modal Heterogeneity

Analysis by tweet modality demonstrated a consistently positive LLM effect, strongest in text-only tweets relative to image and video contexts.

Figure 2: LLM vs. human note rating advantage (AI main-effect coefficient with 95% CI) by tweet modality (text, image, video).

When disaggregated by tweet topic, the LLM performance advantage is most pronounced in health/medicine and conspiracy/pseudoscience topics, domains where the availability and curation of authoritative external references seem to play a key role. In contrast, the LLM is no better than humans on AI-generated content, likely reflecting the limits of current automated detection capabilities and training data.

Figure 3: LLM vs. human note rating advantage (AI main-effect coefficient with 95% CI) by tweet topic category.

Platform Dynamics and Exposure

A structural timing disadvantage exists for LLM notes—they are submitted later due to platform policy (triggered after multiple user flags), thus accrue fewer ratings and face algorithmic penalties for rating count. Human notes are overrepresented in cases with higher exposure, giving them more opportunities to achieve "Currently Rated Helpful" (CRH) status. The note-level, equal-exposure analysis resolves this by restricting to raters who evaluated all notes per tweet, showing that when controlling for exposure, LLM notes score higher on helpfulness and reach CRH more often, though the differences in CRH and CRNH status are sometimes not significant due to reduced power in this subset.

Writing and Sourcing Differences

LLM notes tend to be longer (in word count) and cite more URLs on average than human notes. Citation behavior diverges: LLMs preferentially reference traditional and mainstream news outlets (e.g., reuters.com, en.wikipedia.org, bbc.com), whereas human writers substantially favor platform-native content (notably, x.com URLs) and social media posts as citations. This suggests that current LLM research configurations—by default—produce annotative behavior more aligned with academic or journalist fact-checking rather than the platform-local discourse preferred by some users.

Implications and Theoretical Perspectives

The fielded deployment and direct comparative analysis address a major gap in the literature, establishing that LLMs can match or surpass human crowd performance for stringent, cross-ideological fact-check annotation by platform-native users. The use of ideologically stratified naturalistic raters and exposure normalization sidesteps limitations of lab benchmarks and avoids demand characteristics present in crowdworker studies.

Practically, LLM scale and consistency enable higher annotation coverage without loss of user-perceived helpfulness—critical for addressing the volume at which misinformation is produced. The findings have strong implications for the complementary integration of LLMs and human contributors: LLMs excel at synthesizing well-documented, mainstream topics, while humans retain indispensable value for rapid adaptation to novel, underdocumented, or culturally nuanced events.

For downstream research, the topic- and modality-specific outcomes require extension—LLM pipelines may need augmentation with specialized modules or retrieval capabilities (e.g., AI-content detection, multi-hop reasoning) to close remaining gaps, especially for adversarial or rapidly-evolving topics. The observed sourcing bias in LLM contributions also raises questions about epistemic pluralism and representativeness in automated fact-checking. Robustness checks (e.g., at least 30 ratings per note, timing-matched analyses) reinforce the core findings.

Figure 4: Distributions of rater characteristics (helpfulness leniency and inferred political leaning) for all raters versus those rating both types of notes on a tweet, confirming representativeness and robustness of the equal-exposure analysis.

Conclusion

This work establishes that carefully engineered, prompt- and retrieval-augmented LLM writers can, in real-world deployment, outperform human authors in eliciting broad-based, cross-ideological perceptions of helpfulness in social media fact-checking. While significant advantages exist in key topical domains, relative performance is modulated by both platform-level constraints and the domain-specific reasoning/detection tasks required. Continued advances in modular pipeline engineering and the integration of non-LLM capabilities will likely further broaden the scope of LLM-augmented community moderation systems. These findings provide empirical evidence for both the promise and the limitations of deploying LLMs for trustworthy, scalable, and consensus-oriented fact-checking in high-stakes online environments.