---
title: 'HerO: Open LLM Fact-Checking Pipeline'
url: https://www.emergentmind.com/topics/hero
type: topic
---

# HerO: Open LLM Fact-Checking Pipeline

HerO, short for **Herd of Open LLMs for verifying real-world claims**, is an automated fact-checking system introduced for the AVeriTeC shared task hosted by FEVER-24. It is an end-to-end pipeline that uses only publicly available large language models for evidence retrieval, question generation, and veracity prediction. In the reported shared-task evaluation, HerO achieved **2nd place on the leaderboard** with an **AVeriTeC score of 0.57**, positioning it as a fully open-model alternative to systems that relied on proprietary models [2410.12377].

## 1. Task setting and system identity

HerO was designed for the **AVeriTeC shared task**, whose dataset consists of **real-world claims with crowdsourced, fine-grained evidence from web documents**. The task uses four verdict labels: **Supported**, **Refuted**, **Not Enough Evidence**, and **Conflicting Evidence**. The reported evaluation metrics are **Q score**, defined as Hungarian-METEOR on questions; **Q+A score**, defined as Hungarian-METEOR on question-answer pairs; and **AVeriTeC score**, defined as overall accuracy of the final label [2410.12377].

A defining characteristic of HerO is that it is not a monolithic model. The system is organized as a three-stage pipeline: **Evidence Retrieval**, **Question Generation**, and **Veracity Prediction**. The paper explicitly frames this design as a way to perform each fact-checking subproblem with open LLMs rather than mixing open and closed components. A common misunderstanding is to treat HerO as a single fact-checking model; in fact, the name denotes a coordinated pipeline of retrieval, generation, and classification modules.

The input to the system is a claim $c$. The output is a justification and one of the four AVeriTeC verdict labels. Between these endpoints, HerO first retrieves evidence, then generates clarifying questions for the retrieved evidence, and finally predicts veracity from the claim together with the resulting question-answer evidence.

## 2. End-to-end architecture

The pipeline schematic reported in the paper is as follows. Given a claim $c$, HerO first applies **HyDE-FC** to generate $N$ hypothetical fact-checking documents $D=\{d_1,\dots,d_N\}$ via an LLM. It then performs **BM25 retrieval** using a query built from the concatenation $[c;d_1;\dots;d_N]$, retrieving the **top 10 000 sentences**. A dense reranking stage computes

$$
v_q \;=\;\frac{1}{N+1}\Bigl(g(c)\;+\;\sum_{k=1}^N g(d_k)\Bigr)
$$

where $g(\cdot)$ is the embedding model **SFR-embedding-2**, and then selects the **top 10 evidences**. For question generation, an **8B-parameter Llama-3 model** is prompted with **10 in-context (claim, evidence, question) examples** for each evidence. For veracity prediction, the system feeds the claim plus the **top 10 generated (question, answer) pairs** into a fine-tuned **Llama-3.1-70B (“—it”)** model, which produces a justification and one of the four labels [2410.12377].

This architecture has two notable properties. First, retrieval and reasoning are separated rather than collapsed into one generation step. Second, the interface between stages is explicitly linguistic: evidence is turned into questions, then into question-answer pairs, and only then into a verdict. This suggests a modular interpretation of fact-checking in which retrieval quality, question quality, and label prediction can be evaluated separately.

## 3. Evidence retrieval and HyDE-FC

HerO’s retrieval component is centered on **HyDE-FC**, a hypothetical-document method specialized for fact-checking. The model $f$ used for hypothetical document generation is **Llama-3.1-70B (“—it”)**. The prompt reported in the paper is:

> “Please write a fact-checking article passage to support, refute, indicate not enough evidence, or present conflicting evidence regarding the claim.  
> Claim: <claim>  
> Passage:”

The sampling hyperparameters are **max_tokens=512**, **temperature=0.7**, and **top_p=1.0**, and the system generates **$N=8$ distinct passages** $D=\{d_k\}_{k=1}^8$ [2410.12377].

Retrieval then proceeds in two stages. First, sparse retrieval via BM25 is performed over web documents using the concatenated query

$$
q \;=\; [c;d_1;\dots;d_N],
$$

which returns the **top 10 000 candidate sentences**. Second, dense reranking embeds the query representation and each candidate via **SFR-embedding-2**, computes cosine similarity to $v_q$, and selects the **top 10** evidences.

The paper evaluates retrieval with **A score**, defined as Hungarian-METEOR over retrieved versus gold evidence. In Table 4, **HyDE-FC(Llama-3.1-70b)** achieves **A=0.2801** versus **0.26** for **BM25+dense alone**, a **+0.02 absolute gain** [2410.12377]. The result is important because it isolates a retrieval benefit from hypothetical fact-checking passages rather than from downstream classification alone. At the same time, the discussion notes that retrieval remained **slightly weaker than the winner**, and that this contributed more to the end-to-end gap in final score.

## 4. Question generation with in-context learning

HerO’s second stage converts retrieved evidence into clarifying questions. The primary model choice is **Llama-3-8B (“—it”) (instruction-tuned)**. The experiments also tested **Llama-3.1-8B**, **Llama-3.1-70B**, and **GPT-4o-mini**, but the reported **best trade-off** was **Llama-3-8B** [2410.12377].

For each retrieved evidence, the system selects **10 (claim, evidence, question) triples from the training set via BM25 on evidence only**. The current claim is included in the prompt context. The prompt template begins with the instruction:

> **Your task is to generate a question based on the given claim and evidence. The question should clarify the relationship between the evidence and the claim.**

It then provides **Examples 1–10** and ends with:

> **Now, generate a question that links the following claim and evidence:**  
> Claim: *<current claim>*  
> Evidence: *<current evidence>*  
> Question:

The output is **a single clarifying question whose answer can be filled by the evidence**.

The paper evaluates question generation with **Q score**, defined as Hungarian-METEOR on generated versus gold questions. In Table 5, **Llama-3-8B with claim in context** achieves **Q=0.4938**, compared with **0.4210 baseline without claim** [2410.12377]. The dev-set summary also reports **best Q=0.4938** versus a **0.2404 baseline**. These numbers indicate that including the claim materially changes question quality, which is consistent with the system’s framing of question generation as relation clarification rather than generic evidence paraphrase.

A useful clarification is that this stage does not directly predict verdicts. Its role is to restructure retrieved evidence into question form so that downstream veracity prediction operates over a more explicit evidentiary interface.

## 5. Veracity prediction and training regime

The veracity module uses **Llama-3.1-70B (“—it”)** as its base model. The paper compares three approaches: **in-context learning** with **10 examples** and no weight updates, **LoRA fine-tuning**, and **full parameter fine-tuning**. For LoRA fine-tuning, the reported settings are **rank=128**, **$\alpha=256$**, **Adam lr=2e-5**, **batch=128**, and **2 epochs**. Full parameter fine-tuning uses the **same optimizer settings** [2410.12377].

The prompt template starts with:

> **Your task is to predict the verdict of a claim based on the provided question-answer pair evidence. The possible labels are: `Supported’, `Refuted’, `Not Enough Evidence’, `Conflicting Evidence/Cherrypicking’. Justify your answer using the provided evidence and select the correct label.**

It then presents the claim and **Q1/A1** through **Q10/A10**, followed by **Justification:** and **Verdict:**. The decision mechanism is generative: the model first produces a free-form **Justification**, then emits one of the four labels. The paper states that there is **no explicit threshold or auxiliary loss beyond standard cross-entropy on the four-way classification token**.

On the dev set, Table 6 reports the following results:

| Method | Accuracy | AVeriTeC |
|---|---:|---:|
| In-context Llama-3-70B | 0.628 | 0.494 |
| LoRA Llama-3-70B | 0.724 | 0.556 |
| Full-tune Llama-3.1-70B | 0.752 | 0.578 |

The **full-tune Llama-3.1-70B** configuration was **chosen for HerO** [2410.12377]. The paper also notes a persistent difficulty: the veracity model **still struggles with “Not Enough Evidence” vs. “Conflicting” distinctions**. This is a concrete limitation of the classifier rather than a general weakness of open models as such.

## 6. Empirical results, strengths, and limitations

The dev-set results reported in the paper summarize the best stagewise performance as follows: **A=0.2801** for retrieval, **Q=0.4938** for question generation, and **AVeriTeC=0.578** for veracity prediction. On the test set, Table 8 reports the following shared-task comparison [2410.12377]:

| System | Q | Q+A | AVeriTeC |
|---|---:|---:|---:|
| TUDA_MAI_0 | 0.45 | 0.34 | 0.63 |
| HerO | 0.48 | 0.35 | 0.57 |
| CTU AIC | 0.46 | 0.32 | 0.50 |
| Baseline | 0.24 | 0.20 | 0.11 |

These results support three factual conclusions emphasized in the discussion. First, HerO is a **fully open-model solution** that reached **2nd place** behind a system using **GPT-4 O**. Second, it shows **strong question generation**, with the **highest Q and Q+A among top systems**. Third, **HyDE-FC retrieval yielded clear gains over vanilla BM25**.

The limitations are also explicit. Retrieval is described as **slightly weaker than the winner**, and this is said to contribute more to the gap in final score. The system’s end-to-end latency is approximately **~6.6 hr for 500 dev claims on two H100s**. This suggests that HerO is competitive in quality but not lightweight in throughput. A plausible implication is that, within this architecture, retrieval quality and systems efficiency remain the main bottlenecks rather than question generation.

## 7. Reproducibility and future directions

The paper states that **all code and model configurations for HerO are publicly released** at **https://github.com/ssu-humane/HerO**, with the stated goal of enabling **full reproducibility and further research on open-model fact checking** [2410.12377]. This is consistent with the system’s broader positioning as an open alternative within a shared-task setting.

The future directions listed in the paper are concrete and pipeline-specific. For retrieval, the authors propose to **improve generative retrieval**, including **more diverse HyDE samples or learning to rank documents**. For question generation, they propose to **explore multi-hop question generation to capture complex chains of reasoning**. For system training, they propose to **jointly fine-tune retrieval–generation–verification in an end-to-end fashion**. For label uncertainty, they propose to **incorporate uncertainty estimation to better handle “Not Enough Evidence.”**

Taken together, these directions indicate that HerO should be understood less as a fixed recipe than as a modular research program for open-model fact-checking. The reported results suggest that open LLMs can be competitive in real-world claim verification, but they also show that competitiveness depends on careful orchestration of retrieval, in-context exemplars, and large-scale fine-tuning rather than on any single component alone.

Source: https://www.emergentmind.com/topics/hero