HerO: Open LLM Fact-Checking Pipeline

• HerO is an automated fact-checking system that employs a modular pipeline of evidence retrieval, question generation, and veracity prediction using open large language models.
• It transforms real-world claims into clarifying question-answer pairs through a two-stage BM25 and dense reranking process, enhancing transparency and accuracy.
• The system achieved 2nd place in the AVeriTeC shared task, demonstrating significant gains from hypothetical-document retrieval and strong question generation.

HerO, short for Herd of Open LLMs for verifying real-world claims, is an automated fact-checking system introduced for the AVeriTeC shared task hosted by FEVER-24. It is an end-to-end pipeline that uses only publicly available LLMs for evidence retrieval, question generation, and veracity prediction. In the reported shared-task evaluation, HerO achieved 2nd place on the leaderboard with an AVeriTeC score of 0.57, positioning it as a fully open-model alternative to systems that relied on proprietary models (Yoon et al., 2024).

1. Task setting and system identity

HerO was designed for the AVeriTeC shared task, whose dataset consists of real-world claims with crowdsourced, fine-grained evidence from web documents. The task uses four verdict labels: Supported, Refuted, Not Enough Evidence, and Conflicting Evidence. The reported evaluation metrics are Q score, defined as Hungarian-METEOR on questions; Q+A score, defined as Hungarian-METEOR on question-answer pairs; and AVeriTeC score, defined as overall accuracy of the final label (Yoon et al., 2024).

A defining characteristic of HerO is that it is not a monolithic model. The system is organized as a three-stage pipeline: Evidence Retrieval, Question Generation, and Veracity Prediction. The paper explicitly frames this design as a way to perform each fact-checking subproblem with open LLMs rather than mixing open and closed components. A common misunderstanding is to treat HerO as a single fact-checking model; in fact, the name denotes a coordinated pipeline of retrieval, generation, and classification modules.

The input to the system is a claim $c$. The output is a justification and one of the four AVeriTeC verdict labels. Between these endpoints, HerO first retrieves evidence, then generates clarifying questions for the retrieved evidence, and finally predicts veracity from the claim together with the resulting question-answer evidence.

2. End-to-end architecture

The pipeline schematic reported in the paper is as follows. Given a claim $c$, HerO first applies HyDE-FC to generate $N$ hypothetical fact-checking documents $D=\{d_1,\dots,d_N\}$ via an LLM. It then performs BM25 retrieval using a query built from the concatenation $[c;d_1;\dots;d_N]$, retrieving the top 10 000 sentences. A dense reranking stage computes

$$v_q \;=\;\frac{1}{N+1}\Bigl(g(c)\;+\;\sum_{k=1}^N g(d_k)\Bigr)$$

where $g(\cdot)$ is the embedding model SFR-embedding-2, and then selects the top 10 evidences. For question generation, an 8B-parameter Llama-3 model is prompted with 10 in-context (claim, evidence, question) examples for each evidence. For veracity prediction, the system feeds the claim plus the top 10 generated (question, answer) pairs into a fine-tuned Llama-3.1-70B (“—it”) model, which produces a justification and one of the four labels (Yoon et al., 2024).

This architecture has two notable properties. First, retrieval and reasoning are separated rather than collapsed into one generation step. Second, the interface between stages is explicitly linguistic: evidence is turned into questions, then into question-answer pairs, and only then into a verdict. This suggests a modular interpretation of fact-checking in which retrieval quality, question quality, and label prediction can be evaluated separately.

3. Evidence retrieval and HyDE-FC

HerO’s retrieval component is centered on HyDE-FC, a hypothetical-document method specialized for fact-checking. The model $f$ used for hypothetical document generation is Llama-3.1-70B (“—it”). The prompt reported in the paper is:

“Please write a fact-checking article passage to support, refute, indicate not enough evidence, or present conflicting evidence regarding the claim. Claim: <claim> Passage:”

The sampling hyperparameters are max_tokens=512, temperature=0.7, and top_p=1.0, and the system generates $N=8$ distinct passages $D=\{d_k\}_{k=1}^8$ (Yoon et al., 2024).

Retrieval then proceeds in two stages. First, sparse retrieval via BM25 is performed over web documents using the concatenated query

$c$0

which returns the top 10 000 candidate sentences. Second, dense reranking embeds the query representation and each candidate via SFR-embedding-2, computes cosine similarity to $c$1, and selects the top 10 evidences.

The paper evaluates retrieval with A score, defined as Hungarian-METEOR over retrieved versus gold evidence. In Table 4, HyDE-FC(Llama-3.1-70b) achieves A=0.2801 versus 0.26 for BM25+dense alone, a +0.02 absolute gain (Yoon et al., 2024). The result is important because it isolates a retrieval benefit from hypothetical fact-checking passages rather than from downstream classification alone. At the same time, the discussion notes that retrieval remained slightly weaker than the winner, and that this contributed more to the end-to-end gap in final score.

4. Question generation with in-context learning

HerO’s second stage converts retrieved evidence into clarifying questions. The primary model choice is Llama-3-8B (“—it”) (instruction-tuned). The experiments also tested Llama-3.1-8B, Llama-3.1-70B, and GPT-4o-mini, but the reported best trade-off was Llama-3-8B (Yoon et al., 2024).

For each retrieved evidence, the system selects 10 (claim, evidence, question) triples from the training set via BM25 on evidence only. The current claim is included in the prompt context. The prompt template begins with the instruction:

Your task is to generate a question based on the given claim and evidence. The question should clarify the relationship between the evidence and the claim.

It then provides Examples 1–10 and ends with:

Now, generate a question that links the following claim and evidence:

Claim: <current claim> Evidence: <current evidence> Question:

The output is a single clarifying question whose answer can be filled by the evidence.

The paper evaluates question generation with Q score, defined as Hungarian-METEOR on generated versus gold questions. In Table 5, Llama-3-8B with claim in context achieves Q=0.4938, compared with 0.4210 baseline without claim (Yoon et al., 2024). The dev-set summary also reports best Q=0.4938 versus a 0.2404 baseline. These numbers indicate that including the claim materially changes question quality, which is consistent with the system’s framing of question generation as relation clarification rather than generic evidence paraphrase.

A useful clarification is that this stage does not directly predict verdicts. Its role is to restructure retrieved evidence into question form so that downstream veracity prediction operates over a more explicit evidentiary interface.

5. Veracity prediction and training regime

The veracity module uses Llama-3.1-70B (“—it”) as its base model. The paper compares three approaches: in-context learning with 10 examples and no weight updates, LoRA fine-tuning, and full parameter fine-tuning. For LoRA fine-tuning, the reported settings are rank=128, $c$2, Adam lr=2e-5, batch=128, and 2 epochs. Full parameter fine-tuning uses the same optimizer settings (Yoon et al., 2024).

The prompt template starts with:

Your task is to predict the verdict of a claim based on the provided question-answer pair evidence. The possible labels are: Supported’,Refuted’, Not Enough Evidence’,Conflicting Evidence/Cherrypicking’. Justify your answer using the provided evidence and select the correct label.

It then presents the claim and Q1/A1 through Q10/A10, followed by Justification: and Verdict:. The decision mechanism is generative: the model first produces a free-form Justification, then emits one of the four labels. The paper states that there is no explicit threshold or auxiliary loss beyond standard cross-entropy on the four-way classification token.

On the dev set, Table 6 reports the following results:

Method	Accuracy	AVeriTeC
In-context Llama-3-70B	0.628	0.494
LoRA Llama-3-70B	0.724	0.556
Full-tune Llama-3.1-70B	0.752	0.578

The full-tune Llama-3.1-70B configuration was chosen for HerO (Yoon et al., 2024). The paper also notes a persistent difficulty: the veracity model still struggles with “Not Enough Evidence” vs. “Conflicting” distinctions. This is a concrete limitation of the classifier rather than a general weakness of open models as such.

6. Empirical results, strengths, and limitations

The dev-set results reported in the paper summarize the best stagewise performance as follows: A=0.2801 for retrieval, Q=0.4938 for question generation, and AVeriTeC=0.578 for veracity prediction. On the test set, Table 8 reports the following shared-task comparison (Yoon et al., 2024):

System	Q	Q+A	AVeriTeC
TUDA_MAI_0	0.45	0.34	0.63
HerO	0.48	0.35	0.57
CTU AIC	0.46	0.32	0.50
Baseline	0.24	0.20	0.11

These results support three factual conclusions emphasized in the discussion. First, HerO is a fully open-model solution that reached 2nd place behind a system using GPT-4 O. Second, it shows strong question generation, with the highest Q and Q+A among top systems. Third, HyDE-FC retrieval yielded clear gains over vanilla BM25.

The limitations are also explicit. Retrieval is described as slightly weaker than the winner, and this is said to contribute more to the gap in final score. The system’s end-to-end latency is approximately ~6.6 hr for 500 dev claims on two H100s. This suggests that HerO is competitive in quality but not lightweight in throughput. A plausible implication is that, within this architecture, retrieval quality and systems efficiency remain the main bottlenecks rather than question generation.

7. Reproducibility and future directions

The paper states that all code and model configurations for HerO are publicly released at https://github.com/ssu-humane/HerO, with the stated goal of enabling full reproducibility and further research on open-model fact checking (Yoon et al., 2024). This is consistent with the system’s broader positioning as an open alternative within a shared-task setting.

The future directions listed in the paper are concrete and pipeline-specific. For retrieval, the authors propose to improve generative retrieval, including more diverse HyDE samples or learning to rank documents. For question generation, they propose to explore multi-hop question generation to capture complex chains of reasoning. For system training, they propose to jointly fine-tune retrieval–generation–verification in an end-to-end fashion. For label uncertainty, they propose to incorporate uncertainty estimation to better handle “Not Enough Evidence.”

Taken together, these directions indicate that HerO should be understood less as a fixed recipe than as a modular research program for open-model fact-checking. The reported results suggest that open LLMs can be competitive in real-world claim verification, but they also show that competitiveness depends on careful orchestration of retrieval, in-context exemplars, and large-scale fine-tuning rather than on any single component alone.