Know Your Source: A Public Knowledge Store for Media Background Checks

Abstract: LLM-based retrieval-augmented generation (RAG) is increasingly used for automated fact-checking (AFC) and related tasks. By grounding LLM outputs in retrieved evidence, RAG-based systems provide transparent justifications while allowing external information to be updated independently of the underlying model. However, existing approaches often assume retrieved evidence is reliable, although real-world information may be conflicting, outdated, and can originate from unreliable or biased sources. Recent work on source-critical reasoning addresses this challenge through media background checks (MBCs) (Schlichtkrull, 2024), which assess the credibility of evidence sources to support downstream fact verification. However, generating MBCs relies on costly proprietary search APIs, limiting reproducibility. To mitigate this issue, we introduce MEDIAREF, a publicly available knowledge store of web-sourced documents that enables reproducible, low-cost evaluation of MBC generation across 200 media sources. We describe a reproducible methodology for constructing and updating the collection, assess widely used LLMs on the MBC generation task, and demonstrate that MEDIAREF supports higher-quality MBC generation through both automatic and qualitative evaluation.

• The paper presents MediaRef—a public repository of over 21,000 web-sourced documents—designed to decouple evidence retrieval from proprietary APIs for reproducible media background checks.
• It details a systematic methodology combining targeted query generation, web scraping, and iterative LLM updates to enhance source-critical reasoning in automated fact-checking.
• Experimental findings indicate that retrieval-augmented outputs boost fact recall and information density, though challenges remain in managing conflicting evidence and maintaining update integrity.

A Public Infrastructure for Source Credibility: MediaRef and Media Background Checks

Introduction and Motivation

The paper "Know Your Source: A Public Knowledge Store for Media Background Checks" (2607.02383) addresses a critical limitation in automated fact-checking (AFC) pipelines utilizing retrieval-augmented generation (RAG) with LLMs. While RAG architectures promise increased transparency by grounding model outputs in retrieved evidence, they are predicated on the assumption that this evidence is reliable. This is a problematic premise given the prevalence of conflicting, manipulated, or low-credibility information in real-world sources. Previous approaches to mitigating this issue—most notably through media background checks (MBCs) [Schlichtkrull-2024:MBCs]—have proven effective for source-critical reasoning but rely on proprietary APIs, hindering scalability, reproducibility, and research access.

This paper introduces MediaRef, a large, publicly available, regularly updatable collection of web-sourced documents for 200 diverse news sources. MediaRef decouples evidence retrieval from commercial web APIs, allowing reproducible, low-cost, and robust evaluation and development of MBC generation methods across LLMs. The authors establish a methodology for dataset construction and evaluate multiple baseline and state-of-the-art models with and without access to this resource. They further propose a qualitative analysis framework that probes model outputs for clarity, relevance, informativeness, and verifiability, supporting more granular assessment of source-critical generative models.

MediaRef Resource Construction and Methodology

MBCs are compact, human- and machine-readable textual summaries of characteristics that inform the credibility of media outlets. These include political bias, funding structure, history of factual reporting, engagements with fact-checking organizations, and ownership structures. The MediaRef pipeline is as follows:

1. Selection of Gold-Standard MBCs: The dataset anchors its ground truth in Media Bias/Fact-Check (MB/FC), an annotated corpus widely referenced in source credibility studies.
2. Targeted Query Generation: For each of 200 sampled news outlets, queries combining the outlet name and topical dimensions (ownership, funding, bias, etc.) are generated.
3. Web Document Retrieval and Filtering: Google Search API provides initial candidate URLs; these are augmented with citations from MB/FC and filtered via blacklist heuristics to remove circular or contaminated sources.
4. Web Scraping and Cleanup: The trafilatura tool extracts main content from collected URLs; deduplication and final quality control yield over 21,000 documents.

   Figure 1: The pipeline for generating an MBC via external evidence: targeted queries, support extraction, LLM generation, and iterative evidence incorporation.

This resource is periodically updatable and constructed with full methodological transparency, emphasizing extensibility for future research.

MBC Generation and Information Retrieval Pipeline

Experiments assess LLM performance on MBC generation both with and without explicit information retrieval (IR) from MediaRef. The GPT-3.5, GPT-4o-mini, GPT-5-mini, Qwen2, Qwen3, Llama-3.3, Mistral-7b, and Claude-3.5 models are benchmarked.

• Without IR ("zero-shot" prompt): The LLM generates an MBC based solely on its internal knowledge given an outlet name.
• With IR: The pipeline first generates an initial MBC, then iteratively refines it using evidence retrieved from MediaRef. Keyword queries and BM25 search identify relevant documents, from which passages are extracted via a DeBERTa-based QA model. The LLM then incrementally updates the MBC, minimizing loss of existing information.

  Figure 2: An example MBC for a specific outlet, illustrating the information granularity required and the provenance of background check statements.

Automatic and Human Evaluation

Automatic Metrics include ROUGE-L, METEOR, and a fine-grained FActScore (decomposing MBCs into atomic facts and evaluating their entailment in generated and reference outputs). Key findings:

• IR Increases Fact Recall: Across all models, access to MediaRef-derived evidence boosts fact recall (e.g., Llama-3.3 IR vs non-IR: 29.03% vs. 28.98%). Error rates are largely unaffected or increase only marginally.
• Model Competitiveness: Open-source models (Llama, Mistral) approximate or surpass proprietary systems (notably, GPT-5's lower error rate does not yield large gains in fact recall).
• Length and Information Content: IR-augmented outputs increase in length and informational density but do not always improve factual specificity proportionally.

Human Evaluation provides a qualitative assessment (Likert scales and correlation analysis) along four axes:

• Clarity: Consistently high across models; output is generally syntactically and semantically coherent.
• Relevance: Llama-3.3-70b-instruct exhibits the highest informativeness and verifiability, while GPT-4o-mini provides the most consistently on-topic outputs; GPT-5-mini is least effective in these dimensions.
• Informativeness and Verifiability: These criteria are more challenging. Verifiable MBCs tend to also be informative (Spearman's $\rho = 0.83$ correlation).

  Figure 3: Qualitative analysis (Clarity, Relevance, Informativeness, Verifiability) of MBCs across leading LLMs, average expert-annotator scores.

Analysis of Failure Modes and Model Behaviors

The study identifies several practical challenges:

1. Insufficient Information for Local/Obscure Sources: Outlets lacking external scrutiny rarely receive highly informative/verifiable MBCs; models fail to find or generate substantial credibility signals beyond generic statements.
2. Information Loss in Updates: Mistral-7b-instruct sometimes omits earlier points when updating MBCs. Prompt engineering modifications are suggested to enforce retention or context-aware regeneration.
3. Conflicting Evidence: LLMs occasionally create internal contradictions when iteratively adding information, especially if instructed never to delete prior statements. More sophisticated conflict resolution in prompt instructions is warranted.
4. Interpretive Commentary: GPT-5-mini demonstrates more evaluative reasoning on ambiguous evidence, indicating potential for customized or chain-of-thought prompting to support critical source assessment—but also risks increasing MBCs' length and diminishing readability.

   Figure 4: Spearman correlations between qualitative criteria, showing informativeness and verifiability are strongly linked.

Practical and Theoretical Implications

MediaRef as a Public Infrastructure: By open-sourcing MediaRef, the authors facilitate reproducibility, large-scale benchmarking, and accessibility for both academic and practitioner communities. The decoupling of evidence retrieval from proprietary web APIs removes a significant barrier faced by both debugging and real-world deployment of AFC systems.

Limitations and Ethical Considerations: While the resource's open nature and reproducibility are strengths, reliance on web search results and public sources can introduce systemic biases (e.g., topical, regional, linguistic, or reputational), and completeness is bounded by what is publicly accessible. There is still risk of low-quality or biased sources contaminating the knowledge store, despite blacklist-driven filtering.

Implications for Future Fact-Checking Systems: The results suggest that simply adding retrieval is not a panacea—model architectures and update strategies must be optimized for context-aware information synthesis, conflict resolution, and transparent evidence citation. Verifiability, critical for both layperson trust and professional use, should be a focus for future model development and evaluation.

Prospects for Future Research

This work lays the groundwork for several research directions:

• Scalable and Fine-tuned MBC Generation: Improved LLM instruction-following, conflict sensitivity, and evidence attribution in MBC updates.
• Cross-lingual and Regional Expansion: MediaRef's methods can be extended to less-resourced languages and to a broader spectrum of outlets, improving coverage and generalizability.
• Integrated End-to-End AFC Pipelines: Incorporating MBC outputs directly into fact verdict prediction, weighting evidence by MBC-informed source reliability.
• Active and Adversarial Testing: Exploring model robustness to adversarially constructed or ambiguous evidence, especially as LLMs are deployed in real-world, high-stakes information environments.

Conclusion

"Know Your Source: A Public Knowledge Store for Media Background Checks" presents a significant contribution to the science and practice of source-critical reasoning in fact-checking. MediaRef enables broad, reproducible, and affordable research on source credibility, while the empirical evaluation clarifies current LLMs' capabilities and limits on this task. Ensuring MBCs are clear, relevant, informative, and verifiable—especially in ambiguous or low-information scenarios—remains a key challenge for both automatic systems and the underlying resource curation process.